2023-09-30T01:38:18.504 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-09-30T01:38:18.504 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --description rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados tasks/rados_api_tests validater/valgrind} --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7407087 --name yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi -- /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7407087/orig.config.yaml 2023-09-30T01:38:18.547 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-09-30T01:38:18.766 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7407087 branch: wip-yuri4-testing-2023-09-29-0928-quincy description: rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados tasks/rados_api_tests validater/valgrind} email: yweinste@redhat.com first_in_suite: false job_id: '7407087' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri4-testing-2023-09-29-0928-quincy ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 3 ms bind msgr1: false ms bind msgr2: true ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd memory target: 1610612736 osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: debuginfo: true flavor: default sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri4-testing-2023-09-29-0928-quincy sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 9810 sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri4-testing-2023-09-29-0928-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 targets: smithi059.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEIBRHJ0E9yId33pAQ6zaKUIboMamQ5eavNMIuT0xJxpdZT67asfI4+/iMT8fjQz+Bty8+qBIkRMYPZtedOmIbM= smithi110.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEoQZBS3c/f0LpnsOM7gYX1QUT6fO3qGKPBNFX4nLXfv7pJqFWKHqO/grsNaWh8iLnuAcpGMUTMEOtrac/4nX/Y= tasks: - install: null - ceph: null - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 54e62bcbac4e53d9685e08328b790d3b20d71cae timestamp: 2023-09-29_19:38:56 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-09-30T01:38:18.766 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa; will attempt to use it 2023-09-30T01:38:18.770 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks 2023-09-30T01:38:18.770 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-09-30T01:38:18.771 INFO:teuthology.task.internal:Checking packages... 2023-09-30T01:38:18.795 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '93dce5e17918f8aca65993a7a2dc40233a47b481' 2023-09-30T01:38:18.795 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-09-30T01:38:18.795 INFO:teuthology.packaging:ref: None 2023-09-30T01:38:18.795 INFO:teuthology.packaging:tag: None 2023-09-30T01:38:18.795 INFO:teuthology.packaging:branch: wip-yuri4-testing-2023-09-29-0928-quincy 2023-09-30T01:38:18.795 INFO:teuthology.packaging:sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-30T01:38:18.796 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri4-testing-2023-09-29-0928-quincy 2023-09-30T01:38:18.937 INFO:teuthology.task.internal:Found packages for ceph version 17.2.6-1165.g93dce5e1 2023-09-30T01:38:18.937 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-09-30T01:38:18.944 INFO:teuthology.task.internal:no buildpackages task found 2023-09-30T01:38:18.944 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-09-30T01:38:18.950 INFO:teuthology.task.internal:Saving configuration 2023-09-30T01:38:18.966 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-09-30T01:38:18.973 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-30T01:38:18.994 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi059.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7407087', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-30 01:33:11.163621', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEIBRHJ0E9yId33pAQ6zaKUIboMamQ5eavNMIuT0xJxpdZT67asfI4+/iMT8fjQz+Bty8+qBIkRMYPZtedOmIbM='} 2023-09-30T01:38:19.016 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi110.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7407087', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-30 01:33:11.165031', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEoQZBS3c/f0LpnsOM7gYX1QUT6fO3qGKPBNFX4nLXfv7pJqFWKHqO/grsNaWh8iLnuAcpGMUTMEOtrac/4nX/Y='} 2023-09-30T01:38:19.016 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-09-30T01:38:19.022 INFO:teuthology.task.internal:roles: ubuntu@smithi059.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2023-09-30T01:38:19.022 INFO:teuthology.task.internal:roles: ubuntu@smithi110.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2023-09-30T01:38:19.022 INFO:teuthology.run_tasks:Running task console_log... 2023-09-30T01:38:19.110 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f0d54dd2ee0>, signals=[15]) 2023-09-30T01:38:19.111 INFO:teuthology.run_tasks:Running task internal.connect... 2023-09-30T01:38:19.118 INFO:teuthology.task.internal:Opening connections... 2023-09-30T01:38:19.118 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi059.front.sepia.ceph.com 2023-09-30T01:38:19.121 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T01:38:19.193 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi110.front.sepia.ceph.com 2023-09-30T01:38:19.194 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T01:38:19.262 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-09-30T01:38:19.269 DEBUG:teuthology.orchestra.run.smithi059:> uname -m 2023-09-30T01:38:19.292 INFO:teuthology.orchestra.run.smithi059.stdout:x86_64 2023-09-30T01:38:19.292 DEBUG:teuthology.orchestra.run.smithi059:> cat /etc/os-release 2023-09-30T01:38:19.355 INFO:teuthology.orchestra.run.smithi059.stdout:NAME="CentOS Stream" 2023-09-30T01:38:19.355 INFO:teuthology.orchestra.run.smithi059.stdout:VERSION="8" 2023-09-30T01:38:19.355 INFO:teuthology.orchestra.run.smithi059.stdout:ID="centos" 2023-09-30T01:38:19.355 INFO:teuthology.orchestra.run.smithi059.stdout:ID_LIKE="rhel fedora" 2023-09-30T01:38:19.356 INFO:teuthology.orchestra.run.smithi059.stdout:VERSION_ID="8" 2023-09-30T01:38:19.356 INFO:teuthology.orchestra.run.smithi059.stdout:PLATFORM_ID="platform:el8" 2023-09-30T01:38:19.356 INFO:teuthology.orchestra.run.smithi059.stdout:PRETTY_NAME="CentOS Stream 8" 2023-09-30T01:38:19.356 INFO:teuthology.orchestra.run.smithi059.stdout:ANSI_COLOR="0;31" 2023-09-30T01:38:19.356 INFO:teuthology.orchestra.run.smithi059.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-09-30T01:38:19.356 INFO:teuthology.orchestra.run.smithi059.stdout:HOME_URL="https://centos.org/" 2023-09-30T01:38:19.356 INFO:teuthology.orchestra.run.smithi059.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-09-30T01:38:19.356 INFO:teuthology.orchestra.run.smithi059.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-09-30T01:38:19.356 INFO:teuthology.orchestra.run.smithi059.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-09-30T01:38:19.357 INFO:teuthology.lock.ops:Updating smithi059.front.sepia.ceph.com on lock server 2023-09-30T01:38:19.380 DEBUG:teuthology.orchestra.run.smithi110:> uname -m 2023-09-30T01:38:19.402 INFO:teuthology.orchestra.run.smithi110.stdout:x86_64 2023-09-30T01:38:19.403 DEBUG:teuthology.orchestra.run.smithi110:> cat /etc/os-release 2023-09-30T01:38:19.464 INFO:teuthology.orchestra.run.smithi110.stdout:NAME="CentOS Stream" 2023-09-30T01:38:19.464 INFO:teuthology.orchestra.run.smithi110.stdout:VERSION="8" 2023-09-30T01:38:19.464 INFO:teuthology.orchestra.run.smithi110.stdout:ID="centos" 2023-09-30T01:38:19.464 INFO:teuthology.orchestra.run.smithi110.stdout:ID_LIKE="rhel fedora" 2023-09-30T01:38:19.465 INFO:teuthology.orchestra.run.smithi110.stdout:VERSION_ID="8" 2023-09-30T01:38:19.465 INFO:teuthology.orchestra.run.smithi110.stdout:PLATFORM_ID="platform:el8" 2023-09-30T01:38:19.465 INFO:teuthology.orchestra.run.smithi110.stdout:PRETTY_NAME="CentOS Stream 8" 2023-09-30T01:38:19.465 INFO:teuthology.orchestra.run.smithi110.stdout:ANSI_COLOR="0;31" 2023-09-30T01:38:19.465 INFO:teuthology.orchestra.run.smithi110.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-09-30T01:38:19.465 INFO:teuthology.orchestra.run.smithi110.stdout:HOME_URL="https://centos.org/" 2023-09-30T01:38:19.465 INFO:teuthology.orchestra.run.smithi110.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-09-30T01:38:19.465 INFO:teuthology.orchestra.run.smithi110.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-09-30T01:38:19.465 INFO:teuthology.orchestra.run.smithi110.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-09-30T01:38:19.465 INFO:teuthology.lock.ops:Updating smithi110.front.sepia.ceph.com on lock server 2023-09-30T01:38:19.497 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-09-30T01:38:19.510 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-09-30T01:38:19.518 INFO:teuthology.task.internal:Checking for old test directory... 2023-09-30T01:38:19.518 DEBUG:teuthology.orchestra.run.smithi059:> test '!' -e /home/ubuntu/cephtest 2023-09-30T01:38:19.522 DEBUG:teuthology.orchestra.run.smithi110:> test '!' -e /home/ubuntu/cephtest 2023-09-30T01:38:19.543 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-09-30T01:38:19.553 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-09-30T01:38:19.553 DEBUG:teuthology.orchestra.run.smithi059:> test -z $(ls -A /var/lib/ceph) 2023-09-30T01:38:19.584 DEBUG:teuthology.orchestra.run.smithi110:> test -z $(ls -A /var/lib/ceph) 2023-09-30T01:38:19.611 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-09-30T01:38:19.688 INFO:teuthology.run_tasks:Running task kernel... 2023-09-30T01:38:19.705 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-09-30T01:38:19.705 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-09-30T01:38:19.706 DEBUG:teuthology.orchestra.run.smithi059:> test -f /run/.containerenv -o -f /.dockerenv 2023-09-30T01:38:19.706 DEBUG:teuthology.orchestra.run.smithi110:> test -f /run/.containerenv -o -f /.dockerenv 2023-09-30T01:38:19.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T01:38:19.727 DEBUG:teuthology.orchestra.run.smithi110:> uname -r 2023-09-30T01:38:19.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T01:38:19.728 DEBUG:teuthology.orchestra.run.smithi059:> uname -r 2023-09-30T01:38:19.789 INFO:teuthology.orchestra.run.smithi110.stdout:4.18.0-489.el8.x86_64 2023-09-30T01:38:19.790 INFO:teuthology.task.kernel:Running kernel on smithi110: 4.18.0-489.el8.x86_64 2023-09-30T01:38:19.790 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum install -y kernel 2023-09-30T01:38:19.791 INFO:teuthology.orchestra.run.smithi059.stdout:4.18.0-489.el8.x86_64 2023-09-30T01:38:19.791 INFO:teuthology.task.kernel:Running kernel on smithi059: 4.18.0-489.el8.x86_64 2023-09-30T01:38:19.791 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum install -y kernel 2023-09-30T01:38:23.295 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 8 - AppStream 31 MB/s | 33 MB 00:01 2023-09-30T01:38:25.074 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 8 - AppStream 12 MB/s | 33 MB 00:02 2023-09-30T01:38:30.945 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 8 - BaseOS 40 MB/s | 48 MB 00:01 2023-09-30T01:38:32.416 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 8 - BaseOS 36 MB/s | 48 MB 00:01 2023-09-30T01:38:38.495 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 8 - Extras 75 kB/s | 18 kB 00:00 2023-09-30T01:38:39.238 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 8 - Extras common packages 13 kB/s | 6.9 kB 00:00 2023-09-30T01:38:39.573 INFO:teuthology.orchestra.run.smithi059.stdout:Copr repo for python3-asyncssh owned by ceph 21 kB/s | 3.5 kB 00:00 2023-09-30T01:38:39.633 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 8 - Extras 44 kB/s | 18 kB 00:00 2023-09-30T01:38:40.358 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 8 - Extras common packages 13 kB/s | 6.9 kB 00:00 2023-09-30T01:38:40.652 INFO:teuthology.orchestra.run.smithi110.stdout:Copr repo for python3-asyncssh owned by ceph 24 kB/s | 3.5 kB 00:00 2023-09-30T01:38:40.867 INFO:teuthology.orchestra.run.smithi059.stdout:Extra Packages for Enterprise Linux 14 MB/s | 16 MB 00:01 2023-09-30T01:38:41.394 INFO:teuthology.orchestra.run.smithi110.stdout:Extra Packages for Enterprise Linux 27 MB/s | 16 MB 00:00 2023-09-30T01:38:44.890 INFO:teuthology.orchestra.run.smithi059.stdout:lab-extras 387 kB/s | 24 kB 00:00 2023-09-30T01:38:45.194 INFO:teuthology.orchestra.run.smithi110.stdout:lab-extras 579 kB/s | 24 kB 00:00 2023-09-30T01:38:48.564 INFO:teuthology.orchestra.run.smithi059.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-09-30T01:38:48.564 INFO:teuthology.orchestra.run.smithi059.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-09-30T01:38:48.693 INFO:teuthology.orchestra.run.smithi110.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-09-30T01:38:48.693 INFO:teuthology.orchestra.run.smithi110.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-09-30T01:38:48.829 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-09-30T01:38:48.830 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T01:38:48.830 INFO:teuthology.orchestra.run.smithi059.stdout: Package Architecture Version Repository Size 2023-09-30T01:38:48.830 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T01:38:48.830 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2023-09-30T01:38:48.830 INFO:teuthology.orchestra.run.smithi059.stdout: kernel x86_64 4.18.0-514.el8 baseos 10 M 2023-09-30T01:38:48.830 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2023-09-30T01:38:48.831 INFO:teuthology.orchestra.run.smithi059.stdout: kernel-core x86_64 4.18.0-514.el8 baseos 43 M 2023-09-30T01:38:48.831 INFO:teuthology.orchestra.run.smithi059.stdout: kernel-modules x86_64 4.18.0-514.el8 baseos 35 M 2023-09-30T01:38:48.831 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:38:48.831 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2023-09-30T01:38:48.831 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T01:38:48.831 INFO:teuthology.orchestra.run.smithi059.stdout:Install 3 Packages 2023-09-30T01:38:48.831 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:38:48.832 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 88 M 2023-09-30T01:38:48.832 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 96 M 2023-09-30T01:38:48.832 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2023-09-30T01:38:48.962 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2023-09-30T01:38:48.963 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T01:38:48.963 INFO:teuthology.orchestra.run.smithi110.stdout: Package Architecture Version Repository Size 2023-09-30T01:38:48.963 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T01:38:48.963 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2023-09-30T01:38:48.964 INFO:teuthology.orchestra.run.smithi110.stdout: kernel x86_64 4.18.0-514.el8 baseos 10 M 2023-09-30T01:38:48.964 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2023-09-30T01:38:48.964 INFO:teuthology.orchestra.run.smithi110.stdout: kernel-core x86_64 4.18.0-514.el8 baseos 43 M 2023-09-30T01:38:48.964 INFO:teuthology.orchestra.run.smithi110.stdout: kernel-modules x86_64 4.18.0-514.el8 baseos 35 M 2023-09-30T01:38:48.964 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:38:48.964 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2023-09-30T01:38:48.964 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T01:38:48.964 INFO:teuthology.orchestra.run.smithi110.stdout:Install 3 Packages 2023-09-30T01:38:48.964 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:38:48.965 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 88 M 2023-09-30T01:38:48.965 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 96 M 2023-09-30T01:38:48.965 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2023-09-30T01:38:49.541 INFO:teuthology.orchestra.run.smithi059.stdout:(1/3): kernel-4.18.0-514.el8.x86_64.rpm 15 MB/s | 10 MB 00:00 2023-09-30T01:38:49.662 INFO:teuthology.orchestra.run.smithi110.stdout:(1/3): kernel-4.18.0-514.el8.x86_64.rpm 15 MB/s | 10 MB 00:00 2023-09-30T01:38:50.341 INFO:teuthology.orchestra.run.smithi059.stdout:(2/3): kernel-modules-4.18.0-514.el8.x86_64.rpm 24 MB/s | 35 MB 00:01 2023-09-30T01:38:50.412 INFO:teuthology.orchestra.run.smithi110.stdout:(2/3): kernel-modules-4.18.0-514.el8.x86_64.rpm 25 MB/s | 35 MB 00:01 2023-09-30T01:38:50.800 INFO:teuthology.orchestra.run.smithi059.stdout:(3/3): kernel-core-4.18.0-514.el8.x86_64.rpm 22 MB/s | 43 MB 00:01 2023-09-30T01:38:50.800 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:38:50.800 INFO:teuthology.orchestra.run.smithi059.stdout:Total 45 MB/s | 88 MB 00:01 2023-09-30T01:38:50.905 INFO:teuthology.orchestra.run.smithi110.stdout:(3/3): kernel-core-4.18.0-514.el8.x86_64.rpm 22 MB/s | 43 MB 00:01 2023-09-30T01:38:50.905 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:38:50.905 INFO:teuthology.orchestra.run.smithi110.stdout:Total 45 MB/s | 88 MB 00:01 2023-09-30T01:38:51.503 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2023-09-30T01:38:51.574 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2023-09-30T01:38:51.945 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2023-09-30T01:38:51.946 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2023-09-30T01:38:51.991 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2023-09-30T01:38:51.992 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2023-09-30T01:38:53.224 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2023-09-30T01:38:53.230 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2023-09-30T01:38:53.249 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2023-09-30T01:38:53.252 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2023-09-30T01:38:56.052 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2023-09-30T01:38:56.139 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2023-09-30T01:38:56.626 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : kernel-core-4.18.0-514.el8.x86_64 1/3 2023-09-30T01:38:56.885 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : kernel-core-4.18.0-514.el8.x86_64 1/3 2023-09-30T01:39:00.221 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: kernel-core-4.18.0-514.el8.x86_64 1/3 2023-09-30T01:39:00.606 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: kernel-core-4.18.0-514.el8.x86_64 1/3 2023-09-30T01:39:00.625 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : kernel-modules-4.18.0-514.el8.x86_64 2/3 2023-09-30T01:39:01.002 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : kernel-modules-4.18.0-514.el8.x86_64 2/3 2023-09-30T01:39:04.009 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: kernel-modules-4.18.0-514.el8.x86_64 2/3 2023-09-30T01:39:04.313 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : kernel-4.18.0-514.el8.x86_64 3/3 2023-09-30T01:39:04.629 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: kernel-modules-4.18.0-514.el8.x86_64 2/3 2023-09-30T01:39:04.946 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : kernel-4.18.0-514.el8.x86_64 3/3 2023-09-30T01:39:25.924 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: kernel-core-4.18.0-514.el8.x86_64 3/3 2023-09-30T01:39:25.947 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: kernel-modules-4.18.0-514.el8.x86_64 3/3 2023-09-30T01:39:26.286 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: kernel-4.18.0-514.el8.x86_64 3/3 2023-09-30T01:39:26.286 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : kernel-4.18.0-514.el8.x86_64 1/3 2023-09-30T01:39:26.286 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : kernel-core-4.18.0-514.el8.x86_64 2/3 2023-09-30T01:39:27.009 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : kernel-modules-4.18.0-514.el8.x86_64 3/3 2023-09-30T01:39:27.009 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:39:27.009 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2023-09-30T01:39:27.009 INFO:teuthology.orchestra.run.smithi110.stdout: kernel-4.18.0-514.el8.x86_64 kernel-core-4.18.0-514.el8.x86_64 2023-09-30T01:39:27.010 INFO:teuthology.orchestra.run.smithi110.stdout: kernel-modules-4.18.0-514.el8.x86_64 2023-09-30T01:39:27.010 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:39:27.010 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2023-09-30T01:39:27.189 DEBUG:teuthology.orchestra.run.smithi110:> rpm -q kernel | sort -rV | head -n 1 2023-09-30T01:39:27.401 INFO:teuthology.orchestra.run.smithi110.stdout:kernel-4.18.0-514.el8.x86_64 2023-09-30T01:39:27.402 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-514.el8.x86_64 2023-09-30T01:39:27.402 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-514.el8.x86_64 2023-09-30T01:39:27.402 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-09-30T01:39:27.402 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-09-30T01:39:27.402 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2023-09-30T01:39:27.402 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi110.front.sepia.ceph.com, path=None, version=distro) 2023-09-30T01:39:27.402 DEBUG:teuthology.orchestra.run.smithi110:> rpm -q kernel | sort -rV | head -n 1 2023-09-30T01:39:27.610 INFO:teuthology.orchestra.run.smithi110.stdout:kernel-4.18.0-514.el8.x86_64 2023-09-30T01:39:27.610 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-514.el8.x86_64 2023-09-30T01:39:27.610 DEBUG:teuthology.orchestra.run.smithi110:> sudo rpm -qi grub2-tools 2023-09-30T01:39:27.643 INFO:teuthology.orchestra.run.smithi110.stdout:Name : grub2-tools 2023-09-30T01:39:27.643 INFO:teuthology.orchestra.run.smithi110.stdout:Epoch : 1 2023-09-30T01:39:27.643 INFO:teuthology.orchestra.run.smithi110.stdout:Version : 2.02 2023-09-30T01:39:27.643 INFO:teuthology.orchestra.run.smithi110.stdout:Release : 129.el8 2023-09-30T01:39:27.643 INFO:teuthology.orchestra.run.smithi110.stdout:Architecture: x86_64 2023-09-30T01:39:27.644 INFO:teuthology.orchestra.run.smithi110.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-09-30T01:39:27.644 INFO:teuthology.orchestra.run.smithi110.stdout:Group : System Environment/Base 2023-09-30T01:39:27.644 INFO:teuthology.orchestra.run.smithi110.stdout:Size : 9265586 2023-09-30T01:39:27.644 INFO:teuthology.orchestra.run.smithi110.stdout:License : GPLv3+ 2023-09-30T01:39:27.644 INFO:teuthology.orchestra.run.smithi110.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-09-30T01:39:27.644 INFO:teuthology.orchestra.run.smithi110.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-09-30T01:39:27.644 INFO:teuthology.orchestra.run.smithi110.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-09-30T01:39:27.644 INFO:teuthology.orchestra.run.smithi110.stdout:Build Host : kbuilder.bsys.centos.org 2023-09-30T01:39:27.644 INFO:teuthology.orchestra.run.smithi110.stdout:Relocations : (not relocatable) 2023-09-30T01:39:27.644 INFO:teuthology.orchestra.run.smithi110.stdout:Packager : CentOS BuildSystem 2023-09-30T01:39:27.644 INFO:teuthology.orchestra.run.smithi110.stdout:Vendor : CentOS 2023-09-30T01:39:27.645 INFO:teuthology.orchestra.run.smithi110.stdout:URL : http://www.gnu.org/software/grub/ 2023-09-30T01:39:27.645 INFO:teuthology.orchestra.run.smithi110.stdout:Summary : Support tools for GRUB. 2023-09-30T01:39:27.645 INFO:teuthology.orchestra.run.smithi110.stdout:Description : 2023-09-30T01:39:27.645 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:39:27.645 INFO:teuthology.orchestra.run.smithi110.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-09-30T01:39:27.645 INFO:teuthology.orchestra.run.smithi110.stdout:customizable bootloader with modular architecture. It supports a rich 2023-09-30T01:39:27.645 INFO:teuthology.orchestra.run.smithi110.stdout:variety of kernel formats, file systems, computer architectures and 2023-09-30T01:39:27.645 INFO:teuthology.orchestra.run.smithi110.stdout:hardware devices. 2023-09-30T01:39:27.645 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:39:27.645 INFO:teuthology.orchestra.run.smithi110.stdout:This subpackage provides tools for support of all platforms. 2023-09-30T01:39:27.646 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-09-30T01:39:27.646 INFO:teuthology.task.kernel:Updating grub on smithi110 to boot 4.18.0-514.el8.x86_64 2023-09-30T01:39:27.646 DEBUG:teuthology.orchestra.run.smithi110:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-09-30T01:39:28.097 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: kernel-core-4.18.0-514.el8.x86_64 3/3 2023-09-30T01:39:28.122 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: kernel-modules-4.18.0-514.el8.x86_64 3/3 2023-09-30T01:39:28.470 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: kernel-4.18.0-514.el8.x86_64 3/3 2023-09-30T01:39:28.470 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : kernel-4.18.0-514.el8.x86_64 1/3 2023-09-30T01:39:28.470 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : kernel-core-4.18.0-514.el8.x86_64 2/3 2023-09-30T01:39:29.436 INFO:teuthology.orchestra.run.smithi110.stderr:Generating grub configuration file ... 2023-09-30T01:39:29.619 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : kernel-modules-4.18.0-514.el8.x86_64 3/3 2023-09-30T01:39:29.619 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:39:29.619 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2023-09-30T01:39:29.619 INFO:teuthology.orchestra.run.smithi059.stdout: kernel-4.18.0-514.el8.x86_64 kernel-core-4.18.0-514.el8.x86_64 2023-09-30T01:39:29.620 INFO:teuthology.orchestra.run.smithi059.stdout: kernel-modules-4.18.0-514.el8.x86_64 2023-09-30T01:39:29.620 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:39:29.620 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-09-30T01:39:29.762 DEBUG:teuthology.orchestra.run.smithi059:> rpm -q kernel | sort -rV | head -n 1 2023-09-30T01:39:29.986 INFO:teuthology.orchestra.run.smithi059.stdout:kernel-4.18.0-514.el8.x86_64 2023-09-30T01:39:29.986 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-514.el8.x86_64 2023-09-30T01:39:29.986 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-514.el8.x86_64 2023-09-30T01:39:29.986 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-09-30T01:39:29.986 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-09-30T01:39:29.986 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-09-30T01:39:29.987 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi059.front.sepia.ceph.com, path=None, version=distro) 2023-09-30T01:39:29.987 DEBUG:teuthology.orchestra.run.smithi059:> rpm -q kernel | sort -rV | head -n 1 2023-09-30T01:39:30.203 INFO:teuthology.orchestra.run.smithi059.stdout:kernel-4.18.0-514.el8.x86_64 2023-09-30T01:39:30.204 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-514.el8.x86_64 2023-09-30T01:39:30.204 DEBUG:teuthology.orchestra.run.smithi059:> sudo rpm -qi grub2-tools 2023-09-30T01:39:30.238 INFO:teuthology.orchestra.run.smithi059.stdout:Name : grub2-tools 2023-09-30T01:39:30.238 INFO:teuthology.orchestra.run.smithi059.stdout:Epoch : 1 2023-09-30T01:39:30.238 INFO:teuthology.orchestra.run.smithi059.stdout:Version : 2.02 2023-09-30T01:39:30.238 INFO:teuthology.orchestra.run.smithi059.stdout:Release : 129.el8 2023-09-30T01:39:30.238 INFO:teuthology.orchestra.run.smithi059.stdout:Architecture: x86_64 2023-09-30T01:39:30.239 INFO:teuthology.orchestra.run.smithi059.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-09-30T01:39:30.239 INFO:teuthology.orchestra.run.smithi059.stdout:Group : System Environment/Base 2023-09-30T01:39:30.239 INFO:teuthology.orchestra.run.smithi059.stdout:Size : 9265586 2023-09-30T01:39:30.239 INFO:teuthology.orchestra.run.smithi059.stdout:License : GPLv3+ 2023-09-30T01:39:30.239 INFO:teuthology.orchestra.run.smithi059.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-09-30T01:39:30.239 INFO:teuthology.orchestra.run.smithi059.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-09-30T01:39:30.239 INFO:teuthology.orchestra.run.smithi059.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-09-30T01:39:30.239 INFO:teuthology.orchestra.run.smithi059.stdout:Build Host : kbuilder.bsys.centos.org 2023-09-30T01:39:30.239 INFO:teuthology.orchestra.run.smithi059.stdout:Relocations : (not relocatable) 2023-09-30T01:39:30.239 INFO:teuthology.orchestra.run.smithi059.stdout:Packager : CentOS BuildSystem 2023-09-30T01:39:30.239 INFO:teuthology.orchestra.run.smithi059.stdout:Vendor : CentOS 2023-09-30T01:39:30.239 INFO:teuthology.orchestra.run.smithi059.stdout:URL : http://www.gnu.org/software/grub/ 2023-09-30T01:39:30.240 INFO:teuthology.orchestra.run.smithi059.stdout:Summary : Support tools for GRUB. 2023-09-30T01:39:30.240 INFO:teuthology.orchestra.run.smithi059.stdout:Description : 2023-09-30T01:39:30.240 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:39:30.240 INFO:teuthology.orchestra.run.smithi059.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-09-30T01:39:30.240 INFO:teuthology.orchestra.run.smithi059.stdout:customizable bootloader with modular architecture. It supports a rich 2023-09-30T01:39:30.240 INFO:teuthology.orchestra.run.smithi059.stdout:variety of kernel formats, file systems, computer architectures and 2023-09-30T01:39:30.240 INFO:teuthology.orchestra.run.smithi059.stdout:hardware devices. 2023-09-30T01:39:30.240 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:39:30.240 INFO:teuthology.orchestra.run.smithi059.stdout:This subpackage provides tools for support of all platforms. 2023-09-30T01:39:30.241 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-09-30T01:39:30.241 INFO:teuthology.task.kernel:Updating grub on smithi059 to boot 4.18.0-514.el8.x86_64 2023-09-30T01:39:30.241 DEBUG:teuthology.orchestra.run.smithi059:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-09-30T01:39:31.364 INFO:teuthology.orchestra.run.smithi110.stderr:done 2023-09-30T01:39:31.365 DEBUG:teuthology.orchestra.run.smithi110:> mktemp 2023-09-30T01:39:31.400 INFO:teuthology.orchestra.run.smithi110.stdout:/tmp/tmp.c9tadsbULd 2023-09-30T01:39:31.401 DEBUG:teuthology.orchestra.run.smithi110:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.c9tadsbULd 2023-09-30T01:39:31.412 INFO:teuthology.orchestra.run.smithi059.stderr:Generating grub configuration file ... 2023-09-30T01:39:31.474 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 0666 /tmp/tmp.c9tadsbULd 2023-09-30T01:39:31.611 DEBUG:teuthology.orchestra.remote:smithi110:/tmp/tmp.c9tadsbULd is 6KB 2023-09-30T01:39:31.659 DEBUG:teuthology.orchestra.run.smithi110:> rm -fr /tmp/tmp.c9tadsbULd 2023-09-30T01:39:31.677 DEBUG:teuthology.orchestra.run.smithi110:> sudo /bin/ls /boot/loader/entries || true 2023-09-30T01:39:31.753 INFO:teuthology.orchestra.run.smithi110.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-09-30T01:39:31.753 INFO:teuthology.orchestra.run.smithi110.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-09-30T01:39:31.753 INFO:teuthology.orchestra.run.smithi110.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-09-30T01:39:31.753 INFO:teuthology.orchestra.run.smithi110.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-514.el8.x86_64.conf 2023-09-30T01:39:31.754 DEBUG:teuthology.orchestra.run.smithi110:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-514.el8.x86_64 2023-09-30T01:39:31.918 DEBUG:teuthology.orchestra.run.smithi110:> sudo shutdown -r now 2023-09-30T01:39:33.089 INFO:teuthology.orchestra.run.smithi059.stderr:done 2023-09-30T01:39:33.090 DEBUG:teuthology.orchestra.run.smithi059:> mktemp 2023-09-30T01:39:33.119 INFO:teuthology.orchestra.run.smithi059.stdout:/tmp/tmp.9flPSfGuSw 2023-09-30T01:39:33.119 DEBUG:teuthology.orchestra.run.smithi059:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.9flPSfGuSw 2023-09-30T01:39:33.192 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 0666 /tmp/tmp.9flPSfGuSw 2023-09-30T01:39:33.348 DEBUG:teuthology.orchestra.remote:smithi059:/tmp/tmp.9flPSfGuSw is 6KB 2023-09-30T01:39:33.371 DEBUG:teuthology.orchestra.run.smithi059:> rm -fr /tmp/tmp.9flPSfGuSw 2023-09-30T01:39:33.389 DEBUG:teuthology.orchestra.run.smithi059:> sudo /bin/ls /boot/loader/entries || true 2023-09-30T01:39:33.456 INFO:teuthology.orchestra.run.smithi059.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-09-30T01:39:33.456 INFO:teuthology.orchestra.run.smithi059.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-09-30T01:39:33.456 INFO:teuthology.orchestra.run.smithi059.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-09-30T01:39:33.456 INFO:teuthology.orchestra.run.smithi059.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-514.el8.x86_64.conf 2023-09-30T01:39:33.457 DEBUG:teuthology.orchestra.run.smithi059:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-514.el8.x86_64 2023-09-30T01:39:33.629 DEBUG:teuthology.orchestra.run.smithi059:> sudo shutdown -r now 2023-09-30T01:40:01.948 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2023-09-30T01:40:01.949 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com' 2023-09-30T01:40:01.950 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T01:40:03.633 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-09-30T01:40:03.633 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi059.front.sepia.ceph.com' 2023-09-30T01:40:03.634 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T01:40:20.478 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.110 2023-09-30T01:40:29.486 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com' 2023-09-30T01:40:29.487 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T01:40:32.542 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.110 2023-09-30T01:40:44.555 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com' 2023-09-30T01:40:44.555 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T01:41:03.654 DEBUG:teuthology.orchestra.remote:timed out 2023-09-30T01:41:12.662 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi059.front.sepia.ceph.com' 2023-09-30T01:41:12.663 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T01:41:12.933 DEBUG:teuthology.orchestra.run.smithi059:> true 2023-09-30T01:41:13.383 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi059.front.sepia.ceph.com' 2023-09-30T01:41:13.383 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-514.el8.x86_64"... 2023-09-30T01:41:13.383 DEBUG:teuthology.orchestra.run.smithi059:> uname -r 2023-09-30T01:41:13.406 INFO:teuthology.orchestra.run.smithi059.stdout:4.18.0-514.el8.x86_64 2023-09-30T01:41:13.407 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-514.el8.x86_64 vs 4.18.0-514.el8.x86_64 2023-09-30T01:41:13.407 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-09-30T01:41:13.407 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-09-30T01:41:14.408 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-09-30T01:41:14.408 DEBUG:teuthology.orchestra.run.smithi059:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-09-30T01:41:14.490 INFO:teuthology.orchestra.run.smithi059.stdout:ttyS1 2023-09-30T01:41:14.507 DEBUG:teuthology.parallel:result is None 2023-09-30T01:41:16.734 DEBUG:teuthology.orchestra.run.smithi110:> true 2023-09-30T01:41:17.040 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi110.front.sepia.ceph.com' 2023-09-30T01:41:17.040 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "4.18.0-514.el8.x86_64"... 2023-09-30T01:41:17.041 DEBUG:teuthology.orchestra.run.smithi110:> uname -r 2023-09-30T01:41:17.058 INFO:teuthology.orchestra.run.smithi110.stdout:4.18.0-514.el8.x86_64 2023-09-30T01:41:17.058 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-514.el8.x86_64 vs 4.18.0-514.el8.x86_64 2023-09-30T01:41:17.058 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-09-30T01:41:17.058 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-09-30T01:41:18.059 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-09-30T01:41:18.059 DEBUG:teuthology.orchestra.run.smithi110:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-09-30T01:41:18.129 INFO:teuthology.orchestra.run.smithi110.stdout:ttyS1 2023-09-30T01:41:18.146 DEBUG:teuthology.parallel:result is None 2023-09-30T01:41:18.147 INFO:teuthology.run_tasks:Running task internal.base... 2023-09-30T01:41:18.159 INFO:teuthology.task.internal:Creating test directory... 2023-09-30T01:41:18.159 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-09-30T01:41:18.162 DEBUG:teuthology.orchestra.run.smithi110:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-09-30T01:41:18.182 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-09-30T01:41:18.189 INFO:teuthology.run_tasks:Running task internal.archive... 2023-09-30T01:41:18.197 INFO:teuthology.task.internal:Creating archive directory... 2023-09-30T01:41:18.198 DEBUG:teuthology.orchestra.run.smithi059:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-09-30T01:41:18.228 DEBUG:teuthology.orchestra.run.smithi110:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-09-30T01:41:18.268 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-09-30T01:41:18.275 INFO:teuthology.task.internal:Enabling coredump saving... 2023-09-30T01:41:18.276 DEBUG:teuthology.orchestra.run.smithi059:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-09-30T01:41:18.294 DEBUG:teuthology.orchestra.run.smithi110:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-09-30T01:41:18.330 INFO:teuthology.orchestra.run.smithi059.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:41:18.343 INFO:teuthology.orchestra.run.smithi059.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:41:18.344 INFO:teuthology.orchestra.run.smithi110.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:41:18.355 INFO:teuthology.orchestra.run.smithi110.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T01:41:18.356 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-09-30T01:41:18.364 INFO:teuthology.task.internal:Configuring sudo... 2023-09-30T01:41:18.364 DEBUG:teuthology.orchestra.run.smithi059:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-09-30T01:41:18.391 DEBUG:teuthology.orchestra.run.smithi110:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-09-30T01:41:18.427 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-09-30T01:41:18.437 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-09-30T01:41:18.437 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-09-30T01:41:18.467 DEBUG:teuthology.orchestra.run.smithi110:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-09-30T01:41:18.486 DEBUG:teuthology.orchestra.run.smithi059:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-30T01:41:18.578 DEBUG:teuthology.orchestra.run.smithi059:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-30T01:41:18.641 DEBUG:teuthology.orchestra.run.smithi059:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-30T01:41:18.730 DEBUG:teuthology.orchestra.run.smithi059:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-30T01:41:18.814 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2023-09-30T01:41:18.814 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-09-30T01:41:18.847 DEBUG:teuthology.orchestra.run.smithi110:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-30T01:41:18.890 DEBUG:teuthology.orchestra.run.smithi110:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-30T01:41:18.954 DEBUG:teuthology.orchestra.run.smithi110:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-30T01:41:18.993 DEBUG:teuthology.orchestra.run.smithi110:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-30T01:41:19.045 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-09-30T01:41:19.045 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-09-30T01:41:19.114 DEBUG:teuthology.orchestra.run.smithi059:> sudo service rsyslog restart 2023-09-30T01:41:19.116 DEBUG:teuthology.orchestra.run.smithi110:> sudo service rsyslog restart 2023-09-30T01:41:19.223 INFO:teuthology.orchestra.run.smithi110.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-09-30T01:41:19.237 INFO:teuthology.orchestra.run.smithi059.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-09-30T01:41:19.427 INFO:teuthology.run_tasks:Running task internal.timer... 2023-09-30T01:41:19.435 INFO:teuthology.task.internal:Starting timer... 2023-09-30T01:41:19.435 INFO:teuthology.run_tasks:Running task pcp... 2023-09-30T01:41:19.453 INFO:teuthology.run_tasks:Running task selinux... 2023-09-30T01:41:19.465 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-09-30T01:41:19.465 DEBUG:teuthology.orchestra.run.smithi059:> sudo service auditd rotate 2023-09-30T01:41:19.658 INFO:teuthology.orchestra.run.smithi059.stdout:Rotating logs: 2023-09-30T01:41:19.660 DEBUG:teuthology.orchestra.run.smithi110:> sudo service auditd rotate 2023-09-30T01:41:19.751 INFO:teuthology.orchestra.run.smithi110.stdout:Rotating logs: 2023-09-30T01:41:19.753 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-09-30T01:41:19.753 DEBUG:teuthology.orchestra.run.smithi059:> /usr/sbin/getenforce 2023-09-30T01:41:19.794 INFO:teuthology.orchestra.run.smithi059.stdout:Permissive 2023-09-30T01:41:19.794 DEBUG:teuthology.orchestra.run.smithi110:> /usr/sbin/getenforce 2023-09-30T01:41:19.823 INFO:teuthology.orchestra.run.smithi110.stdout:Permissive 2023-09-30T01:41:19.823 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi059.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi110.front.sepia.ceph.com': 'permissive'} 2023-09-30T01:41:19.823 DEBUG:teuthology.orchestra.run.smithi059:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-09-30T01:41:20.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T01:41:20.338 DEBUG:teuthology.orchestra.run.smithi110:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-09-30T01:41:20.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T01:41:20.366 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-09-30T01:41:20.366 DEBUG:teuthology.orchestra.run.smithi059:> sudo /usr/sbin/setenforce permissive 2023-09-30T01:41:20.917 DEBUG:teuthology.orchestra.run.smithi110:> sudo /usr/sbin/setenforce permissive 2023-09-30T01:41:20.949 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-09-30T01:41:20.959 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-09-30T01:41:20.960 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-09-30T01:41:20.975 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-09-30T01:41:20.977 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi059.front.sepia.ceph.com,smithi110.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-09-30T01:49:21.034 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi059.front.sepia.ceph.com'), Remote(name='ubuntu@smithi110.front.sepia.ceph.com')] 2023-09-30T01:49:21.035 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi059.front.sepia.ceph.com' 2023-09-30T01:49:21.036 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T01:49:21.109 DEBUG:teuthology.orchestra.run.smithi059:> true 2023-09-30T01:49:21.186 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi059.front.sepia.ceph.com' 2023-09-30T01:49:21.186 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi110.front.sepia.ceph.com' 2023-09-30T01:49:21.187 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi110.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T01:49:21.256 DEBUG:teuthology.orchestra.run.smithi110:> true 2023-09-30T01:49:21.283 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi110.front.sepia.ceph.com' 2023-09-30T01:49:21.283 INFO:teuthology.run_tasks:Running task clock... 2023-09-30T01:49:21.295 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-09-30T01:49:21.296 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-09-30T01:49:21.296 DEBUG:teuthology.orchestra.run.smithi059:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-30T01:49:21.299 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-09-30T01:49:21.299 DEBUG:teuthology.orchestra.run.smithi110:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-30T01:49:21.338 INFO:teuthology.orchestra.run.smithi059.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-09-30T01:49:21.354 INFO:teuthology.orchestra.run.smithi110.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-09-30T01:49:21.356 INFO:teuthology.orchestra.run.smithi059.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-09-30T01:49:21.368 INFO:teuthology.orchestra.run.smithi110.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-09-30T01:49:21.429 INFO:teuthology.orchestra.run.smithi059.stderr:sudo: ntpd: command not found 2023-09-30T01:49:21.438 INFO:teuthology.orchestra.run.smithi110.stderr:sudo: ntpd: command not found 2023-09-30T01:49:21.442 INFO:teuthology.orchestra.run.smithi059.stdout:506 Cannot talk to daemon 2023-09-30T01:49:21.449 INFO:teuthology.orchestra.run.smithi110.stdout:506 Cannot talk to daemon 2023-09-30T01:49:21.458 INFO:teuthology.orchestra.run.smithi059.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-09-30T01:49:21.464 INFO:teuthology.orchestra.run.smithi110.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-09-30T01:49:21.474 INFO:teuthology.orchestra.run.smithi059.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-09-30T01:49:21.477 INFO:teuthology.orchestra.run.smithi110.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-09-30T01:49:21.547 INFO:teuthology.orchestra.run.smithi059.stderr:bash: ntpq: command not found 2023-09-30T01:49:21.549 INFO:teuthology.orchestra.run.smithi110.stderr:bash: ntpq: command not found 2023-09-30T01:49:21.551 INFO:teuthology.orchestra.run.smithi059.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-09-30T01:49:21.551 INFO:teuthology.orchestra.run.smithi059.stdout:=============================================================================== 2023-09-30T01:49:21.551 INFO:teuthology.orchestra.run.smithi059.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T01:49:21.551 INFO:teuthology.orchestra.run.smithi059.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T01:49:21.551 INFO:teuthology.orchestra.run.smithi059.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T01:49:21.551 INFO:teuthology.orchestra.run.smithi059.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T01:49:21.553 INFO:teuthology.orchestra.run.smithi110.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-09-30T01:49:21.553 INFO:teuthology.orchestra.run.smithi110.stdout:=============================================================================== 2023-09-30T01:49:21.553 INFO:teuthology.orchestra.run.smithi110.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T01:49:21.554 INFO:teuthology.orchestra.run.smithi110.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T01:49:21.554 INFO:teuthology.orchestra.run.smithi110.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T01:49:21.554 INFO:teuthology.orchestra.run.smithi110.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T01:49:21.554 INFO:teuthology.run_tasks:Running task install... 2023-09-30T01:49:21.563 DEBUG:teuthology.task.install:project ceph 2023-09-30T01:49:21.563 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'}} 2023-09-30T01:49:21.563 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'} 2023-09-30T01:49:21.563 INFO:teuthology.task.install:Using flavor: default 2023-09-30T01:49:21.569 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2023-09-30T01:49:21.569 INFO:teuthology.task.install:extra packages: [] 2023-09-30T01:49:21.570 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481', 'tag': None, 'wait_for_package': False} 2023-09-30T01:49:21.570 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-30T01:49:21.572 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481', 'tag': None, 'wait_for_package': False} 2023-09-30T01:49:21.572 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-30T01:49:21.716 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/ 2023-09-30T01:49:21.716 INFO:teuthology.task.install.rpm:Package version is 17.2.6-1165.g93dce5e1 2023-09-30T01:49:21.786 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/ 2023-09-30T01:49:21.786 INFO:teuthology.task.install.rpm:Package version is 17.2.6-1165.g93dce5e1 2023-09-30T01:49:21.828 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-09-30T01:49:21.829 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2023-09-30T01:49:21.829 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-09-30T01:49:21.864 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2023-09-30T01:49:21.865 DEBUG:teuthology.orchestra.run.smithi059:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/93dce5e17918f8aca65993a7a2dc40233a47b481/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-09-30T01:49:21.894 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-09-30T01:49:21.894 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-09-30T01:49:21.894 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-09-30T01:49:21.923 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2023-09-30T01:49:21.923 DEBUG:teuthology.orchestra.run.smithi110:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/93dce5e17918f8aca65993a7a2dc40233a47b481/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-09-30T01:49:21.940 DEBUG:teuthology.orchestra.run.smithi059:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-09-30T01:49:21.993 DEBUG:teuthology.orchestra.run.smithi110:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-09-30T01:49:22.029 DEBUG:teuthology.orchestra.run.smithi059:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-09-30T01:49:22.062 INFO:teuthology.orchestra.run.smithi059.stdout:check_obsoletes = 1 2023-09-30T01:49:22.063 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum clean all 2023-09-30T01:49:22.071 DEBUG:teuthology.orchestra.run.smithi110:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-09-30T01:49:22.137 INFO:teuthology.orchestra.run.smithi110.stdout:check_obsoletes = 1 2023-09-30T01:49:22.138 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum clean all 2023-09-30T01:49:22.370 INFO:teuthology.orchestra.run.smithi059.stdout:76 files removed 2023-09-30T01:49:22.405 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-radosgw 2023-09-30T01:49:22.412 INFO:teuthology.orchestra.run.smithi110.stdout:76 files removed 2023-09-30T01:49:22.441 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-radosgw 2023-09-30T01:49:23.187 INFO:teuthology.orchestra.run.smithi110.stdout:ceph packages for x86_64 228 kB/s | 78 kB 00:00 2023-09-30T01:49:23.268 INFO:teuthology.orchestra.run.smithi059.stdout:ceph packages for x86_64 198 kB/s | 78 kB 00:00 2023-09-30T01:49:23.555 INFO:teuthology.orchestra.run.smithi110.stdout:ceph noarch packages 28 kB/s | 9.2 kB 00:00 2023-09-30T01:49:23.610 INFO:teuthology.orchestra.run.smithi059.stdout:ceph noarch packages 31 kB/s | 9.2 kB 00:00 2023-09-30T01:49:23.930 INFO:teuthology.orchestra.run.smithi110.stdout:ceph source packages 4.2 kB/s | 1.5 kB 00:00 2023-09-30T01:49:24.002 INFO:teuthology.orchestra.run.smithi059.stdout:ceph source packages 4.1 kB/s | 1.5 kB 00:00 2023-09-30T01:49:25.214 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 8 - AppStream 27 MB/s | 33 MB 00:01 2023-09-30T01:49:25.386 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 8 - AppStream 25 MB/s | 33 MB 00:01 2023-09-30T01:49:32.631 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 8 - BaseOS 36 MB/s | 48 MB 00:01 2023-09-30T01:49:33.627 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 8 - BaseOS 34 MB/s | 48 MB 00:01 2023-09-30T01:49:39.807 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 8 - Extras 68 kB/s | 18 kB 00:00 2023-09-30T01:49:40.400 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 8 - Extras common packages 17 kB/s | 6.9 kB 00:00 2023-09-30T01:49:41.495 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 8 - Extras 68 kB/s | 18 kB 00:00 2023-09-30T01:49:42.096 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 8 - Extras common packages 17 kB/s | 6.9 kB 00:00 2023-09-30T01:49:42.534 INFO:teuthology.orchestra.run.smithi110.stdout:CentOS Stream 8 - PowerTools 3.3 MB/s | 6.5 MB 00:01 2023-09-30T01:49:43.969 INFO:teuthology.orchestra.run.smithi110.stdout:Copr repo for python3-asyncssh owned by ceph 22 kB/s | 3.5 kB 00:00 2023-09-30T01:49:44.355 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 8 - PowerTools 3.1 MB/s | 6.5 MB 00:02 2023-09-30T01:49:45.907 INFO:teuthology.orchestra.run.smithi059.stdout:Copr repo for python3-asyncssh owned by ceph 27 kB/s | 3.5 kB 00:00 2023-09-30T01:49:46.154 INFO:teuthology.orchestra.run.smithi110.stdout:Extra Packages for Enterprise Linux 7.8 MB/s | 16 MB 00:02 2023-09-30T01:49:47.674 INFO:teuthology.orchestra.run.smithi059.stdout:Extra Packages for Enterprise Linux 10 MB/s | 16 MB 00:01 2023-09-30T01:49:50.086 INFO:teuthology.orchestra.run.smithi110.stdout:lab-extras 318 kB/s | 24 kB 00:00 2023-09-30T01:49:52.074 INFO:teuthology.orchestra.run.smithi059.stdout:lab-extras 357 kB/s | 24 kB 00:00 2023-09-30T01:49:54.469 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2023-09-30T01:49:54.472 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T01:49:54.472 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2023-09-30T01:49:54.472 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T01:49:54.472 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2023-09-30T01:49:54.472 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-radosgw x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 17 M 2023-09-30T01:49:54.472 INFO:teuthology.orchestra.run.smithi110.stdout:Upgrading: 2023-09-30T01:49:54.472 INFO:teuthology.orchestra.run.smithi110.stdout: librados2 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.5 M 2023-09-30T01:49:54.473 INFO:teuthology.orchestra.run.smithi110.stdout: librbd1 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.6 M 2023-09-30T01:49:54.473 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2023-09-30T01:49:54.473 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-base x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 5.2 M 2023-09-30T01:49:54.473 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-common x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 23 M 2023-09-30T01:49:54.473 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-selinux x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 24 k 2023-09-30T01:49:54.473 INFO:teuthology.orchestra.run.smithi110.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-09-30T01:49:54.473 INFO:teuthology.orchestra.run.smithi110.stdout: libcephfs2 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 796 k 2023-09-30T01:49:54.473 INFO:teuthology.orchestra.run.smithi110.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-09-30T01:49:54.473 INFO:teuthology.orchestra.run.smithi110.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-09-30T01:49:54.473 INFO:teuthology.orchestra.run.smithi110.stdout: libradosstriper1 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 506 k 2023-09-30T01:49:54.474 INFO:teuthology.orchestra.run.smithi110.stdout: librdkafka x86_64 0.11.4-3.el8 appstream 354 k 2023-09-30T01:49:54.474 INFO:teuthology.orchestra.run.smithi110.stdout: librgw2 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 5.6 M 2023-09-30T01:49:54.474 INFO:teuthology.orchestra.run.smithi110.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-09-30T01:49:54.474 INFO:teuthology.orchestra.run.smithi110.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-09-30T01:49:54.474 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ceph-argparse x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 45 k 2023-09-30T01:49:54.474 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ceph-common x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 115 k 2023-09-30T01:49:54.474 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cephfs x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 190 k 2023-09-30T01:49:54.474 INFO:teuthology.orchestra.run.smithi110.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-09-30T01:49:54.474 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rados x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 370 k 2023-09-30T01:49:54.474 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rbd x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 358 k 2023-09-30T01:49:54.474 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rgw x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 107 k 2023-09-30T01:49:54.475 INFO:teuthology.orchestra.run.smithi110.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-09-30T01:49:54.475 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:49:54.475 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2023-09-30T01:49:54.475 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T01:49:54.475 INFO:teuthology.orchestra.run.smithi110.stdout:Install 21 Packages 2023-09-30T01:49:54.475 INFO:teuthology.orchestra.run.smithi110.stdout:Upgrade 2 Packages 2023-09-30T01:49:54.475 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:49:54.477 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 63 M 2023-09-30T01:49:54.477 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2023-09-30T01:49:56.939 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-09-30T01:49:56.942 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T01:49:56.942 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2023-09-30T01:49:56.942 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T01:49:56.942 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2023-09-30T01:49:56.942 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-radosgw x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 17 M 2023-09-30T01:49:56.942 INFO:teuthology.orchestra.run.smithi059.stdout:Upgrading: 2023-09-30T01:49:56.942 INFO:teuthology.orchestra.run.smithi059.stdout: librados2 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.5 M 2023-09-30T01:49:56.942 INFO:teuthology.orchestra.run.smithi059.stdout: librbd1 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.6 M 2023-09-30T01:49:56.943 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2023-09-30T01:49:56.943 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-base x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 5.2 M 2023-09-30T01:49:56.943 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-common x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 23 M 2023-09-30T01:49:56.943 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-selinux x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 24 k 2023-09-30T01:49:56.943 INFO:teuthology.orchestra.run.smithi059.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-09-30T01:49:56.943 INFO:teuthology.orchestra.run.smithi059.stdout: libcephfs2 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 796 k 2023-09-30T01:49:56.943 INFO:teuthology.orchestra.run.smithi059.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-09-30T01:49:56.943 INFO:teuthology.orchestra.run.smithi059.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-09-30T01:49:56.943 INFO:teuthology.orchestra.run.smithi059.stdout: libradosstriper1 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 506 k 2023-09-30T01:49:56.943 INFO:teuthology.orchestra.run.smithi059.stdout: librdkafka x86_64 0.11.4-3.el8 appstream 354 k 2023-09-30T01:49:56.943 INFO:teuthology.orchestra.run.smithi059.stdout: librgw2 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 5.6 M 2023-09-30T01:49:56.944 INFO:teuthology.orchestra.run.smithi059.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-09-30T01:49:56.944 INFO:teuthology.orchestra.run.smithi059.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-09-30T01:49:56.944 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ceph-argparse x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 45 k 2023-09-30T01:49:56.944 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ceph-common x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 115 k 2023-09-30T01:49:56.944 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cephfs x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 190 k 2023-09-30T01:49:56.944 INFO:teuthology.orchestra.run.smithi059.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-09-30T01:49:56.944 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rados x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 370 k 2023-09-30T01:49:56.944 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rbd x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 358 k 2023-09-30T01:49:56.944 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rgw x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 107 k 2023-09-30T01:49:56.944 INFO:teuthology.orchestra.run.smithi059.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-09-30T01:49:56.944 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:49:56.944 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2023-09-30T01:49:56.945 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T01:49:56.945 INFO:teuthology.orchestra.run.smithi059.stdout:Install 21 Packages 2023-09-30T01:49:56.945 INFO:teuthology.orchestra.run.smithi059.stdout:Upgrade 2 Packages 2023-09-30T01:49:56.945 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:49:56.945 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 63 M 2023-09-30T01:49:56.946 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2023-09-30T01:49:59.067 INFO:teuthology.orchestra.run.smithi059.stdout:(1/23): ceph-radosgw-17.2.6-1165.g93dce5e1.el8. 9.2 MB/s | 17 MB 00:01 2023-09-30T01:49:59.109 INFO:teuthology.orchestra.run.smithi059.stdout:(2/23): ceph-selinux-17.2.6-1165.g93dce5e1.el8. 591 kB/s | 24 kB 00:00 2023-09-30T01:49:59.193 INFO:teuthology.orchestra.run.smithi059.stdout:(3/23): libcephfs2-17.2.6-1165.g93dce5e1.el8.x8 9.3 MB/s | 796 kB 00:00 2023-09-30T01:49:59.268 INFO:teuthology.orchestra.run.smithi059.stdout:(4/23): libradosstriper1-17.2.6-1165.g93dce5e1. 6.6 MB/s | 506 kB 00:00 2023-09-30T01:49:59.593 INFO:teuthology.orchestra.run.smithi059.stdout:(5/23): ceph-base-17.2.6-1165.g93dce5e1.el8.x86 2.2 MB/s | 5.2 MB 00:02 2023-09-30T01:49:59.685 INFO:teuthology.orchestra.run.smithi059.stdout:(6/23): librgw2-17.2.6-1165.g93dce5e1.el8.x86_6 13 MB/s | 5.6 MB 00:00 2023-09-30T01:49:59.741 INFO:teuthology.orchestra.run.smithi110.stdout:(1/23): ceph-base-17.2.6-1165.g93dce5e1.el8.x86 1.1 MB/s | 5.2 MB 00:04 2023-09-30T01:49:59.742 INFO:teuthology.orchestra.run.smithi059.stdout:(7/23): python3-ceph-argparse-17.2.6-1165.g93dc 300 kB/s | 45 kB 00:00 2023-09-30T01:49:59.778 INFO:teuthology.orchestra.run.smithi059.stdout:(8/23): python3-ceph-common-17.2.6-1165.g93dce5 1.2 MB/s | 115 kB 00:00 2023-09-30T01:49:59.797 INFO:teuthology.orchestra.run.smithi110.stdout:(2/23): ceph-selinux-17.2.6-1165.g93dce5e1.el8. 419 kB/s | 24 kB 00:00 2023-09-30T01:49:59.819 INFO:teuthology.orchestra.run.smithi059.stdout:(9/23): python3-cephfs-17.2.6-1165.g93dce5e1.el 2.4 MB/s | 190 kB 00:00 2023-09-30T01:49:59.845 INFO:teuthology.orchestra.run.smithi059.stdout:(10/23): python3-rados-17.2.6-1165.g93dce5e1.el 5.4 MB/s | 370 kB 00:00 2023-09-30T01:49:59.903 INFO:teuthology.orchestra.run.smithi059.stdout:(11/23): python3-rgw-17.2.6-1165.g93dce5e1.el8. 1.8 MB/s | 107 kB 00:00 2023-09-30T01:49:59.928 INFO:teuthology.orchestra.run.smithi059.stdout:(12/23): python3-rbd-17.2.6-1165.g93dce5e1.el8. 3.2 MB/s | 358 kB 00:00 2023-09-30T01:50:00.154 INFO:teuthology.orchestra.run.smithi059.stdout:(13/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.1 MB/s | 259 kB 00:00 2023-09-30T01:50:00.179 INFO:teuthology.orchestra.run.smithi059.stdout:(14/23): librdkafka-0.11.4-3.el8.x86_64.rpm 1.3 MB/s | 354 kB 00:00 2023-09-30T01:50:00.188 INFO:teuthology.orchestra.run.smithi110.stdout:(3/23): libcephfs2-17.2.6-1165.g93dce5e1.el8.x8 2.0 MB/s | 796 kB 00:00 2023-09-30T01:50:00.238 INFO:teuthology.orchestra.run.smithi059.stdout:(15/23): python3-prettytable-0.7.2-14.el8.noarc 526 kB/s | 44 kB 00:00 2023-09-30T01:50:00.371 INFO:teuthology.orchestra.run.smithi059.stdout:(16/23): librabbitmq-0.9.0-4.el8.x86_64.rpm 245 kB/s | 47 kB 00:00 2023-09-30T01:50:00.664 INFO:teuthology.orchestra.run.smithi110.stdout:(4/23): libradosstriper1-17.2.6-1165.g93dce5e1. 1.0 MB/s | 506 kB 00:00 2023-09-30T01:50:00.713 INFO:teuthology.orchestra.run.smithi059.stdout:(17/23): liboath-2.6.2-3.el8.x86_64.rpm 172 kB/s | 59 kB 00:00 2023-09-30T01:50:00.747 INFO:teuthology.orchestra.run.smithi059.stdout:(18/23): gperftools-libs-2.7-9.el8.x86_64.rpm 602 kB/s | 306 kB 00:00 2023-09-30T01:50:00.889 INFO:teuthology.orchestra.run.smithi059.stdout:(19/23): libunwind-1.3.1-3.el8.x86_64.rpm 429 kB/s | 75 kB 00:00 2023-09-30T01:50:01.131 INFO:teuthology.orchestra.run.smithi059.stdout:(20/23): ceph-common-17.2.6-1165.g93dce5e1.el8. 6.0 MB/s | 23 MB 00:03 2023-09-30T01:50:01.198 INFO:teuthology.orchestra.run.smithi059.stdout:(21/23): thrift-0.13.0-2.el8.x86_64.rpm 3.9 MB/s | 1.7 MB 00:00 2023-09-30T01:50:01.223 INFO:teuthology.orchestra.run.smithi110.stdout:(5/23): ceph-radosgw-17.2.6-1165.g93dce5e1.el8. 2.7 MB/s | 17 MB 00:06 2023-09-30T01:50:01.281 INFO:teuthology.orchestra.run.smithi110.stdout:(6/23): python3-ceph-argparse-17.2.6-1165.g93dc 768 kB/s | 45 kB 00:00 2023-09-30T01:50:01.348 INFO:teuthology.orchestra.run.smithi110.stdout:(7/23): python3-ceph-common-17.2.6-1165.g93dce5 1.7 MB/s | 115 kB 00:00 2023-09-30T01:50:01.440 INFO:teuthology.orchestra.run.smithi110.stdout:(8/23): python3-cephfs-17.2.6-1165.g93dce5e1.el 2.0 MB/s | 190 kB 00:00 2023-09-30T01:50:01.473 INFO:teuthology.orchestra.run.smithi059.stdout:(22/23): librbd1-17.2.6-1165.g93dce5e1.el8.x86_ 10 MB/s | 3.6 MB 00:00 2023-09-30T01:50:01.549 INFO:teuthology.orchestra.run.smithi110.stdout:(9/23): python3-rados-17.2.6-1165.g93dce5e1.el8 3.3 MB/s | 370 kB 00:00 2023-09-30T01:50:01.649 INFO:teuthology.orchestra.run.smithi110.stdout:(10/23): python3-rbd-17.2.6-1165.g93dce5e1.el8. 3.5 MB/s | 358 kB 00:00 2023-09-30T01:50:01.691 INFO:teuthology.orchestra.run.smithi110.stdout:(11/23): python3-rgw-17.2.6-1165.g93dce5e1.el8. 2.5 MB/s | 107 kB 00:00 2023-09-30T01:50:01.748 INFO:teuthology.orchestra.run.smithi059.stdout:(23/23): librados2-17.2.6-1165.g93dce5e1.el8.x8 4.0 MB/s | 3.5 MB 00:00 2023-09-30T01:50:01.751 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:50:01.751 INFO:teuthology.orchestra.run.smithi059.stdout:Total 13 MB/s | 63 MB 00:04 2023-09-30T01:50:01.796 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2023-09-30T01:50:01.848 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2023-09-30T01:50:01.848 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2023-09-30T01:50:01.916 INFO:teuthology.orchestra.run.smithi110.stdout:(12/23): librdkafka-0.11.4-3.el8.x86_64.rpm 1.5 MB/s | 354 kB 00:00 2023-09-30T01:50:01.975 INFO:teuthology.orchestra.run.smithi110.stdout:(13/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 4.4 MB/s | 259 kB 00:00 2023-09-30T01:50:02.033 INFO:teuthology.orchestra.run.smithi110.stdout:(14/23): python3-prettytable-0.7.2-14.el8.noarc 753 kB/s | 44 kB 00:00 2023-09-30T01:50:02.202 INFO:teuthology.orchestra.run.smithi110.stdout:(15/23): librabbitmq-0.9.0-4.el8.x86_64.rpm 281 kB/s | 47 kB 00:00 2023-09-30T01:50:02.257 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2023-09-30T01:50:02.261 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2023-09-30T01:50:02.534 INFO:teuthology.orchestra.run.smithi110.stdout:(16/23): gperftools-libs-2.7-9.el8.x86_64.rpm 919 kB/s | 306 kB 00:00 2023-09-30T01:50:02.626 INFO:teuthology.orchestra.run.smithi110.stdout:(17/23): liboath-2.6.2-3.el8.x86_64.rpm 643 kB/s | 59 kB 00:00 2023-09-30T01:50:02.726 INFO:teuthology.orchestra.run.smithi110.stdout:(18/23): libunwind-1.3.1-3.el8.x86_64.rpm 751 kB/s | 75 kB 00:00 2023-09-30T01:50:02.943 INFO:teuthology.orchestra.run.smithi110.stdout:(19/23): librgw2-17.2.6-1165.g93dce5e1.el8.x86_ 2.4 MB/s | 5.6 MB 00:02 2023-09-30T01:50:02.985 INFO:teuthology.orchestra.run.smithi110.stdout:(20/23): thrift-0.13.0-2.el8.x86_64.rpm 6.7 MB/s | 1.7 MB 00:00 2023-09-30T01:50:03.261 INFO:teuthology.orchestra.run.smithi110.stdout:(21/23): ceph-common-17.2.6-1165.g93dce5e1.el8. 2.8 MB/s | 23 MB 00:08 2023-09-30T01:50:03.398 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2023-09-30T01:50:03.603 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2023-09-30T01:50:03.669 INFO:teuthology.orchestra.run.smithi110.stdout:(22/23): librbd1-17.2.6-1165.g93dce5e1.el8.x86_ 5.2 MB/s | 3.6 MB 00:00 2023-09-30T01:50:03.790 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-09-30T01:50:04.053 INFO:teuthology.orchestra.run.smithi110.stdout:(23/23): librados2-17.2.6-1165.g93dce5e1.el8.x8 3.1 MB/s | 3.5 MB 00:01 2023-09-30T01:50:04.055 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:50:04.055 INFO:teuthology.orchestra.run.smithi110.stdout:Total 6.6 MB/s | 63 MB 00:09 2023-09-30T01:50:04.098 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2023-09-30T01:50:04.144 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2023-09-30T01:50:04.145 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2023-09-30T01:50:04.208 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-09-30T01:50:04.399 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/25 2023-09-30T01:50:04.486 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/25 2023-09-30T01:50:04.510 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2023-09-30T01:50:04.513 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2023-09-30T01:50:04.770 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 4/25 2023-09-30T01:50:04.932 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/25 2023-09-30T01:50:05.377 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/25 2023-09-30T01:50:05.565 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/25 2023-09-30T01:50:05.575 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2023-09-30T01:50:05.805 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/25 2023-09-30T01:50:05.834 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2023-09-30T01:50:05.979 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : thrift-0.13.0-2.el8.x86_64 7/25 2023-09-30T01:50:06.020 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-09-30T01:50:06.192 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 8/25 2023-09-30T01:50:06.348 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 9/25 2023-09-30T01:50:06.452 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-09-30T01:50:06.637 INFO:teuthology.orchestra.run.smithi110.stdout: Upgrading : librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/25 2023-09-30T01:50:06.737 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/25 2023-09-30T01:50:06.958 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 9/25 2023-09-30T01:50:07.036 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 4/25 2023-09-30T01:50:07.106 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 10/25 2023-09-30T01:50:07.244 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 10/25 2023-09-30T01:50:07.253 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/25 2023-09-30T01:50:07.394 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-ceph-argparse-2:17.2.6-1165.g93dce5e1.el8. 11/25 2023-09-30T01:50:07.586 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 12/25 2023-09-30T01:50:07.652 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/25 2023-09-30T01:50:07.779 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 13/25 2023-09-30T01:50:07.862 INFO:teuthology.orchestra.run.smithi110.stdout: Upgrading : librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/25 2023-09-30T01:50:07.986 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 14/25 2023-09-30T01:50:08.081 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/25 2023-09-30T01:50:08.140 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_6 15/25 2023-09-30T01:50:08.202 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_6 15/25 2023-09-30T01:50:08.257 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : thrift-0.13.0-2.el8.x86_64 7/25 2023-09-30T01:50:08.395 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2023-09-30T01:50:08.459 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 8/25 2023-09-30T01:50:08.608 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2023-09-30T01:50:08.628 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 9/25 2023-09-30T01:50:08.798 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2023-09-30T01:50:08.990 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-ceph-common-2:17.2.6-1165.g93dce5e1.el8.x8 19/25 2023-09-30T01:50:09.167 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 9/25 2023-09-30T01:50:09.336 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 10/25 2023-09-30T01:50:09.390 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 10/25 2023-09-30T01:50:09.555 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-ceph-argparse-2:17.2.6-1165.g93dce5e1.el8. 11/25 2023-09-30T01:50:09.789 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 12/25 2023-09-30T01:50:10.005 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 13/25 2023-09-30T01:50:10.248 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 14/25 2023-09-30T01:50:10.403 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_6 15/25 2023-09-30T01:50:10.472 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_6 15/25 2023-09-30T01:50:10.695 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2023-09-30T01:50:10.886 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2023-09-30T01:50:11.095 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2023-09-30T01:50:11.628 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-ceph-common-2:17.2.6-1165.g93dce5e1.el8.x8 19/25 2023-09-30T01:50:11.928 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 20/25 2023-09-30T01:50:12.123 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 20/25 2023-09-30T01:50:13.058 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 20/25 2023-09-30T01:50:13.298 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 21/25 2023-09-30T01:50:13.446 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 21/25 2023-09-30T01:50:13.581 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 22/25 2023-09-30T01:50:14.322 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 20/25 2023-09-30T01:50:14.553 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 20/25 2023-09-30T01:50:15.132 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 20/25 2023-09-30T01:50:15.388 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 21/25 2023-09-30T01:50:15.491 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 21/25 2023-09-30T01:50:15.661 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 22/25 2023-09-30T01:50:34.025 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 22/25 2023-09-30T01:50:34.025 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /sys 2023-09-30T01:50:34.025 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /proc 2023-09-30T01:50:34.025 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /dev 2023-09-30T01:50:34.025 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /run 2023-09-30T01:50:34.025 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /mnt 2023-09-30T01:50:34.025 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /var/tmp 2023-09-30T01:50:34.025 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /home 2023-09-30T01:50:34.025 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /tmp 2023-09-30T01:50:34.026 INFO:teuthology.orchestra.run.smithi110.stdout:skipping the directory /dev 2023-09-30T01:50:34.026 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:50:34.874 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 22/25 2023-09-30T01:50:34.875 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /sys 2023-09-30T01:50:34.875 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /proc 2023-09-30T01:50:34.875 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /dev 2023-09-30T01:50:34.875 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /run 2023-09-30T01:50:34.875 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /mnt 2023-09-30T01:50:34.875 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /var/tmp 2023-09-30T01:50:34.875 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /home 2023-09-30T01:50:34.875 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /tmp 2023-09-30T01:50:34.875 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /dev 2023-09-30T01:50:34.875 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:50:35.744 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 23/25 2023-09-30T01:50:35.821 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 23/25 2023-09-30T01:50:35.822 INFO:teuthology.orchestra.run.smithi110.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-09-30T01:50:36.035 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-09-30T01:50:36.035 INFO:teuthology.orchestra.run.smithi110.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-09-30T01:50:36.708 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-09-30T01:50:36.708 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/25 2023-09-30T01:50:36.708 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/25 2023-09-30T01:50:36.708 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/25 2023-09-30T01:50:36.708 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 4/25 2023-09-30T01:50:36.708 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/25 2023-09-30T01:50:36.708 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_6 6/25 2023-09-30T01:50:36.708 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 7/25 2023-09-30T01:50:36.708 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-ceph-argparse-2:17.2.6-1165.g93dce5e1.el8. 8/25 2023-09-30T01:50:36.708 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-ceph-common-2:17.2.6-1165.g93dce5e1.el8.x8 9/25 2023-09-30T01:50:36.709 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 10/25 2023-09-30T01:50:36.709 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 11/25 2023-09-30T01:50:36.709 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 12/25 2023-09-30T01:50:36.709 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 13/25 2023-09-30T01:50:36.709 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 14/25 2023-09-30T01:50:36.709 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2023-09-30T01:50:36.709 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2023-09-30T01:50:36.709 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 17/25 2023-09-30T01:50:36.709 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2023-09-30T01:50:36.709 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2023-09-30T01:50:36.709 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2023-09-30T01:50:36.710 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2023-09-30T01:50:36.710 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 22/25 2023-09-30T01:50:36.710 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-09-30T01:50:36.710 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 24/25 2023-09-30T01:50:36.721 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 23/25 2023-09-30T01:50:36.806 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 23/25 2023-09-30T01:50:36.806 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-09-30T01:50:37.014 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-09-30T01:50:37.014 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-09-30T01:50:37.733 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-09-30T01:50:37.734 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:50:37.734 INFO:teuthology.orchestra.run.smithi110.stdout:Upgraded: 2023-09-30T01:50:37.734 INFO:teuthology.orchestra.run.smithi110.stdout: librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:50:37.734 INFO:teuthology.orchestra.run.smithi110.stdout: librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:50:37.734 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2023-09-30T01:50:37.734 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:50:37.734 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:50:37.734 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:50:37.734 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:50:37.734 INFO:teuthology.orchestra.run.smithi110.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-09-30T01:50:37.734 INFO:teuthology.orchestra.run.smithi110.stdout: libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:50:37.735 INFO:teuthology.orchestra.run.smithi110.stdout: liboath-2.6.2-3.el8.x86_64 2023-09-30T01:50:37.735 INFO:teuthology.orchestra.run.smithi110.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-09-30T01:50:37.735 INFO:teuthology.orchestra.run.smithi110.stdout: libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:50:37.735 INFO:teuthology.orchestra.run.smithi110.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-09-30T01:50:37.735 INFO:teuthology.orchestra.run.smithi110.stdout: librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:50:37.735 INFO:teuthology.orchestra.run.smithi110.stdout: libunwind-1.3.1-3.el8.x86_64 2023-09-30T01:50:37.735 INFO:teuthology.orchestra.run.smithi110.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-09-30T01:50:37.735 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ceph-argparse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:50:37.735 INFO:teuthology.orchestra.run.smithi110.stdout: python3-ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:50:37.735 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:50:37.735 INFO:teuthology.orchestra.run.smithi110.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-09-30T01:50:37.735 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:50:37.736 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:50:37.736 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:50:37.736 INFO:teuthology.orchestra.run.smithi110.stdout: thrift-0.13.0-2.el8.x86_64 2023-09-30T01:50:37.736 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:50:37.736 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2023-09-30T01:50:37.937 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-09-30T01:50:37.937 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/25 2023-09-30T01:50:37.937 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/25 2023-09-30T01:50:37.937 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/25 2023-09-30T01:50:37.937 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 4/25 2023-09-30T01:50:37.937 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/25 2023-09-30T01:50:37.937 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_6 6/25 2023-09-30T01:50:37.937 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 7/25 2023-09-30T01:50:37.937 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-ceph-argparse-2:17.2.6-1165.g93dce5e1.el8. 8/25 2023-09-30T01:50:37.938 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-ceph-common-2:17.2.6-1165.g93dce5e1.el8.x8 9/25 2023-09-30T01:50:37.938 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 10/25 2023-09-30T01:50:37.938 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 11/25 2023-09-30T01:50:37.938 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 12/25 2023-09-30T01:50:37.938 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 13/25 2023-09-30T01:50:37.938 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 14/25 2023-09-30T01:50:37.938 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2023-09-30T01:50:37.938 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2023-09-30T01:50:37.938 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 17/25 2023-09-30T01:50:37.938 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2023-09-30T01:50:37.938 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2023-09-30T01:50:37.939 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2023-09-30T01:50:37.939 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2023-09-30T01:50:37.939 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 22/25 2023-09-30T01:50:37.939 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-09-30T01:50:37.939 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 24/25 2023-09-30T01:50:38.033 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-test 2023-09-30T01:50:38.881 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:00:48 ago on Sat 30 Sep 2023 01:49:50 AM UTC. 2023-09-30T01:50:39.134 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-09-30T01:50:39.135 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:50:39.135 INFO:teuthology.orchestra.run.smithi059.stdout:Upgraded: 2023-09-30T01:50:39.135 INFO:teuthology.orchestra.run.smithi059.stdout: librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:50:39.135 INFO:teuthology.orchestra.run.smithi059.stdout: librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:50:39.136 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2023-09-30T01:50:39.136 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:50:39.136 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:50:39.136 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:50:39.136 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:50:39.137 INFO:teuthology.orchestra.run.smithi059.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-09-30T01:50:39.137 INFO:teuthology.orchestra.run.smithi059.stdout: libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:50:39.137 INFO:teuthology.orchestra.run.smithi059.stdout: liboath-2.6.2-3.el8.x86_64 2023-09-30T01:50:39.137 INFO:teuthology.orchestra.run.smithi059.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-09-30T01:50:39.138 INFO:teuthology.orchestra.run.smithi059.stdout: libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:50:39.138 INFO:teuthology.orchestra.run.smithi059.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-09-30T01:50:39.139 INFO:teuthology.orchestra.run.smithi059.stdout: librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:50:39.139 INFO:teuthology.orchestra.run.smithi059.stdout: libunwind-1.3.1-3.el8.x86_64 2023-09-30T01:50:39.139 INFO:teuthology.orchestra.run.smithi059.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-09-30T01:50:39.139 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ceph-argparse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:50:39.139 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:50:39.139 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:50:39.139 INFO:teuthology.orchestra.run.smithi059.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-09-30T01:50:39.139 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:50:39.139 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:50:39.140 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:50:39.140 INFO:teuthology.orchestra.run.smithi059.stdout: thrift-0.13.0-2.el8.x86_64 2023-09-30T01:50:39.140 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:50:39.140 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-09-30T01:50:39.437 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-test 2023-09-30T01:50:40.415 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:00:48 ago on Sat 30 Sep 2023 01:49:52 AM UTC. 2023-09-30T01:50:40.440 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2023-09-30T01:50:40.441 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T01:50:40.441 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2023-09-30T01:50:40.441 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T01:50:40.441 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2023-09-30T01:50:40.441 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-test x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 52 M 2023-09-30T01:50:40.441 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2023-09-30T01:50:40.441 INFO:teuthology.orchestra.run.smithi110.stdout: jq x86_64 1.6-7.el8 appstream 202 k 2023-09-30T01:50:40.441 INFO:teuthology.orchestra.run.smithi110.stdout: libcephsqlite x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 175 k 2023-09-30T01:50:40.442 INFO:teuthology.orchestra.run.smithi110.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-09-30T01:50:40.442 INFO:teuthology.orchestra.run.smithi110.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-09-30T01:50:40.442 INFO:teuthology.orchestra.run.smithi110.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-09-30T01:50:40.442 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:50:40.442 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2023-09-30T01:50:40.442 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T01:50:40.442 INFO:teuthology.orchestra.run.smithi110.stdout:Install 6 Packages 2023-09-30T01:50:40.442 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:50:40.443 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 53 M 2023-09-30T01:50:40.443 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 233 M 2023-09-30T01:50:40.443 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2023-09-30T01:50:40.702 INFO:teuthology.orchestra.run.smithi110.stdout:(1/6): jq-1.6-7.el8.x86_64.rpm 929 kB/s | 202 kB 00:00 2023-09-30T01:50:40.763 INFO:teuthology.orchestra.run.smithi110.stdout:(2/6): oniguruma-6.8.2-2.el8.x86_64.rpm 3.0 MB/s | 187 kB 00:00 2023-09-30T01:50:40.822 INFO:teuthology.orchestra.run.smithi110.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 5.4 MB/s | 323 kB 00:00 2023-09-30T01:50:40.880 INFO:teuthology.orchestra.run.smithi110.stdout:(4/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.2 MB/s | 71 kB 00:00 2023-09-30T01:50:41.106 INFO:teuthology.orchestra.run.smithi110.stdout:(5/6): libcephsqlite-17.2.6-1165.g93dce5e1.el8. 281 kB/s | 175 kB 00:00 2023-09-30T01:50:42.167 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-09-30T01:50:42.169 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T01:50:42.169 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2023-09-30T01:50:42.169 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T01:50:42.169 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2023-09-30T01:50:42.169 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-test x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 52 M 2023-09-30T01:50:42.169 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2023-09-30T01:50:42.169 INFO:teuthology.orchestra.run.smithi059.stdout: jq x86_64 1.6-7.el8 appstream 202 k 2023-09-30T01:50:42.169 INFO:teuthology.orchestra.run.smithi059.stdout: libcephsqlite x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 175 k 2023-09-30T01:50:42.169 INFO:teuthology.orchestra.run.smithi059.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-09-30T01:50:42.170 INFO:teuthology.orchestra.run.smithi059.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-09-30T01:50:42.170 INFO:teuthology.orchestra.run.smithi059.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-09-30T01:50:42.170 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:50:42.170 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2023-09-30T01:50:42.170 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T01:50:42.170 INFO:teuthology.orchestra.run.smithi059.stdout:Install 6 Packages 2023-09-30T01:50:42.170 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:50:42.171 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 53 M 2023-09-30T01:50:42.171 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 233 M 2023-09-30T01:50:42.171 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2023-09-30T01:50:42.551 INFO:teuthology.orchestra.run.smithi059.stdout:(1/6): jq-1.6-7.el8.x86_64.rpm 1.0 MB/s | 202 kB 00:00 2023-09-30T01:50:42.609 INFO:teuthology.orchestra.run.smithi059.stdout:(2/6): oniguruma-6.8.2-2.el8.x86_64.rpm 3.2 MB/s | 187 kB 00:00 2023-09-30T01:50:42.643 INFO:teuthology.orchestra.run.smithi059.stdout:(3/6): libcephsqlite-17.2.6-1165.g93dce5e1.el8. 605 kB/s | 175 kB 00:00 2023-09-30T01:50:42.668 INFO:teuthology.orchestra.run.smithi059.stdout:(4/6): socat-1.7.4.1-1.el8.x86_64.rpm 5.4 MB/s | 323 kB 00:00 2023-09-30T01:50:42.793 INFO:teuthology.orchestra.run.smithi059.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 470 kB/s | 71 kB 00:00 2023-09-30T01:50:45.102 INFO:teuthology.orchestra.run.smithi059.stdout:(6/6): ceph-test-17.2.6-1165.g93dce5e1.el8.x86_ 19 MB/s | 52 MB 00:02 2023-09-30T01:50:45.104 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:50:45.104 INFO:teuthology.orchestra.run.smithi059.stdout:Total 18 MB/s | 53 MB 00:02 2023-09-30T01:50:45.151 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2023-09-30T01:50:45.174 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2023-09-30T01:50:45.174 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2023-09-30T01:50:45.518 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2023-09-30T01:50:45.522 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2023-09-30T01:50:46.023 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2023-09-30T01:50:46.225 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-09-30T01:50:46.432 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-09-30T01:50:46.543 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-09-30T01:50:46.626 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-09-30T01:50:46.794 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : jq-1.6-7.el8.x86_64 4/6 2023-09-30T01:50:46.960 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/6 2023-09-30T01:50:51.530 INFO:teuthology.orchestra.run.smithi110.stdout:(6/6): ceph-test-17.2.6-1165.g93dce5e1.el8.x86_ 4.7 MB/s | 52 MB 00:11 2023-09-30T01:50:51.532 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:50:51.532 INFO:teuthology.orchestra.run.smithi110.stdout:Total 4.7 MB/s | 53 MB 00:11 2023-09-30T01:50:51.574 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2023-09-30T01:50:51.596 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2023-09-30T01:50:51.596 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2023-09-30T01:50:51.899 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2023-09-30T01:50:51.902 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2023-09-30T01:50:52.383 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2023-09-30T01:50:52.591 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/6 2023-09-30T01:50:52.624 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-09-30T01:50:52.826 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/6 2023-09-30T01:50:52.842 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-09-30T01:50:53.006 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-09-30T01:50:53.067 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-09-30T01:50:53.133 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/6 2023-09-30T01:50:53.134 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/6 2023-09-30T01:50:53.134 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/6 2023-09-30T01:50:53.134 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : jq-1.6-7.el8.x86_64 3/6 2023-09-30T01:50:53.134 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-09-30T01:50:53.134 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-09-30T01:50:53.272 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : jq-1.6-7.el8.x86_64 4/6 2023-09-30T01:50:53.456 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/6 2023-09-30T01:50:53.889 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-09-30T01:50:53.889 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:50:53.889 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2023-09-30T01:50:53.889 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:50:53.889 INFO:teuthology.orchestra.run.smithi059.stdout: jq-1.6-7.el8.x86_64 2023-09-30T01:50:53.889 INFO:teuthology.orchestra.run.smithi059.stdout: libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:50:53.889 INFO:teuthology.orchestra.run.smithi059.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-09-30T01:50:53.889 INFO:teuthology.orchestra.run.smithi059.stdout: socat-1.7.4.1-1.el8.x86_64 2023-09-30T01:50:53.889 INFO:teuthology.orchestra.run.smithi059.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-09-30T01:50:53.890 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:50:53.890 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-09-30T01:50:54.158 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph 2023-09-30T01:50:55.133 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:03 ago on Sat 30 Sep 2023 01:49:52 AM UTC. 2023-09-30T01:50:56.905 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-09-30T01:50:56.909 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T01:50:56.909 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2023-09-30T01:50:56.909 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T01:50:56.909 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2023-09-30T01:50:56.909 INFO:teuthology.orchestra.run.smithi059.stdout: ceph x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 6.4 k 2023-09-30T01:50:56.909 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2023-09-30T01:50:56.909 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mds x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 2.3 M 2023-09-30T01:50:56.909 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 1.6 M 2023-09-30T01:50:56.909 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-modules-core noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 238 k 2023-09-30T01:50:56.910 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mon x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.9 M 2023-09-30T01:50:56.910 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-osd x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 17 M 2023-09-30T01:50:56.910 INFO:teuthology.orchestra.run.smithi059.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-09-30T01:50:56.910 INFO:teuthology.orchestra.run.smithi059.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-09-30T01:50:56.910 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-09-30T01:50:56.910 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-09-30T01:50:56.910 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-09-30T01:50:56.910 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-09-30T01:50:56.910 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-09-30T01:50:56.910 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-09-30T01:50:56.910 INFO:teuthology.orchestra.run.smithi059.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-09-30T01:50:56.911 INFO:teuthology.orchestra.run.smithi059.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-09-30T01:50:56.911 INFO:teuthology.orchestra.run.smithi059.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-09-30T01:50:56.911 INFO:teuthology.orchestra.run.smithi059.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-09-30T01:50:56.911 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-09-30T01:50:56.911 INFO:teuthology.orchestra.run.smithi059.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-09-30T01:50:56.911 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-09-30T01:50:56.911 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-09-30T01:50:56.911 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-09-30T01:50:56.911 INFO:teuthology.orchestra.run.smithi059.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-09-30T01:50:56.911 INFO:teuthology.orchestra.run.smithi059.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-09-30T01:50:56.911 INFO:teuthology.orchestra.run.smithi059.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-09-30T01:50:56.912 INFO:teuthology.orchestra.run.smithi059.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-09-30T01:50:56.912 INFO:teuthology.orchestra.run.smithi059.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-09-30T01:50:56.912 INFO:teuthology.orchestra.run.smithi059.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-09-30T01:50:56.912 INFO:teuthology.orchestra.run.smithi059.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-09-30T01:50:56.912 INFO:teuthology.orchestra.run.smithi059.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-09-30T01:50:56.912 INFO:teuthology.orchestra.run.smithi059.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-09-30T01:50:56.912 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:50:56.912 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2023-09-30T01:50:56.912 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T01:50:56.912 INFO:teuthology.orchestra.run.smithi059.stdout:Install 32 Packages 2023-09-30T01:50:56.912 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:50:56.913 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 28 M 2023-09-30T01:50:56.913 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 110 M 2023-09-30T01:50:56.913 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2023-09-30T01:50:57.327 INFO:teuthology.orchestra.run.smithi059.stdout:(1/32): ceph-17.2.6-1165.g93dce5e1.el8.x86_64.r 48 kB/s | 6.4 kB 00:00 2023-09-30T01:50:57.544 INFO:teuthology.orchestra.run.smithi059.stdout:(2/32): ceph-mgr-17.2.6-1165.g93dce5e1.el8.x86_ 4.5 MB/s | 1.6 MB 00:00 2023-09-30T01:50:57.945 INFO:teuthology.orchestra.run.smithi059.stdout:(3/32): ceph-mon-17.2.6-1165.g93dce5e1.el8.x86_ 6.4 MB/s | 3.9 MB 00:00 2023-09-30T01:50:58.003 INFO:teuthology.orchestra.run.smithi059.stdout:(4/32): ceph-mgr-modules-core-17.2.6-1165.g93dc 4.0 MB/s | 238 kB 00:00 2023-09-30T01:50:58.212 INFO:teuthology.orchestra.run.smithi059.stdout:(5/32): python3-mako-1.0.6-14.el8.noarch.rpm 753 kB/s | 157 kB 00:00 2023-09-30T01:50:58.271 INFO:teuthology.orchestra.run.smithi059.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 666 kB/s | 39 kB 00:00 2023-09-30T01:50:58.313 INFO:teuthology.orchestra.run.smithi059.stdout:(7/32): ceph-mds-17.2.6-1165.g93dce5e1.el8.x86_ 2.1 MB/s | 2.3 MB 00:01 2023-09-30T01:50:58.338 INFO:teuthology.orchestra.run.smithi059.stdout:(8/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 1.5 MB/s | 103 kB 00:00 2023-09-30T01:50:58.401 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/6 2023-09-30T01:50:58.455 INFO:teuthology.orchestra.run.smithi059.stdout:(9/32): python3-werkzeug-0.12.2-4.el8.noarch.rp 3.8 MB/s | 457 kB 00:00 2023-09-30T01:50:58.480 INFO:teuthology.orchestra.run.smithi059.stdout:(10/32): python3-pytz-2017.2-11.el8.noarch.rpm 341 kB/s | 57 kB 00:00 2023-09-30T01:50:58.628 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/6 2023-09-30T01:50:58.707 INFO:teuthology.orchestra.run.smithi059.stdout:(11/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 994 kB/s | 249 kB 00:00 2023-09-30T01:50:58.764 INFO:teuthology.orchestra.run.smithi059.stdout:(12/32): python3-cryptography-3.2.1-6.el8.x86_6 2.2 MB/s | 641 kB 00:00 2023-09-30T01:50:58.789 INFO:teuthology.orchestra.run.smithi059.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 1.3 MB/s | 109 kB 00:00 2023-09-30T01:50:58.899 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/6 2023-09-30T01:50:58.899 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/6 2023-09-30T01:50:58.899 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/6 2023-09-30T01:50:58.900 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : jq-1.6-7.el8.x86_64 3/6 2023-09-30T01:50:58.900 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-09-30T01:50:58.900 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-09-30T01:50:59.023 INFO:teuthology.orchestra.run.smithi059.stdout:(14/32): ceph-osd-17.2.6-1165.g93dce5e1.el8.x86 11 MB/s | 17 MB 00:01 2023-09-30T01:50:59.579 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-09-30T01:50:59.579 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:50:59.579 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2023-09-30T01:50:59.579 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:50:59.579 INFO:teuthology.orchestra.run.smithi110.stdout: jq-1.6-7.el8.x86_64 2023-09-30T01:50:59.579 INFO:teuthology.orchestra.run.smithi110.stdout: libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:50:59.579 INFO:teuthology.orchestra.run.smithi110.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-09-30T01:50:59.579 INFO:teuthology.orchestra.run.smithi110.stdout: socat-1.7.4.1-1.el8.x86_64 2023-09-30T01:50:59.579 INFO:teuthology.orchestra.run.smithi110.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-09-30T01:50:59.579 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:50:59.580 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2023-09-30T01:50:59.784 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph 2023-09-30T01:51:00.007 INFO:teuthology.orchestra.run.smithi059.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 35 kB/s | 44 kB 00:01 2023-09-30T01:51:00.215 INFO:teuthology.orchestra.run.smithi059.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 130 kB/s | 185 kB 00:01 2023-09-30T01:51:00.390 INFO:teuthology.orchestra.run.smithi059.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 127 kB/s | 173 kB 00:01 2023-09-30T01:51:00.449 INFO:teuthology.orchestra.run.smithi059.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 48 kB/s | 11 kB 00:00 2023-09-30T01:51:00.633 INFO:teuthology.orchestra.run.smithi059.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 73 kB/s | 18 kB 00:00 2023-09-30T01:51:00.636 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:10 ago on Sat 30 Sep 2023 01:49:50 AM UTC. 2023-09-30T01:51:00.708 INFO:teuthology.orchestra.run.smithi059.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 188 kB/s | 49 kB 00:00 2023-09-30T01:51:00.925 INFO:teuthology.orchestra.run.smithi059.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 201 kB/s | 59 kB 00:00 2023-09-30T01:51:00.950 INFO:teuthology.orchestra.run.smithi059.stdout:(22/32): python3-cherrypy-18.4.0-1.el8.noarch.r 408 kB/s | 384 kB 00:00 2023-09-30T01:51:01.150 INFO:teuthology.orchestra.run.smithi059.stdout:(23/32): python3-pecan-1.3.2-9.el8.noarch.rpm 640 kB/s | 283 kB 00:00 2023-09-30T01:51:01.176 INFO:teuthology.orchestra.run.smithi059.stdout:(24/32): python3-portend-2.6-1.el8.noarch.rpm 62 kB/s | 16 kB 00:00 2023-09-30T01:51:01.201 INFO:teuthology.orchestra.run.smithi059.stdout:(25/32): python3-simplegeneric-0.8.1-17.el8.noa 75 kB/s | 19 kB 00:00 2023-09-30T01:51:01.401 INFO:teuthology.orchestra.run.smithi059.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 95 kB/s | 24 kB 00:00 2023-09-30T01:51:01.426 INFO:teuthology.orchestra.run.smithi059.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 117 kB/s | 29 kB 00:00 2023-09-30T01:51:01.452 INFO:teuthology.orchestra.run.smithi059.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 110 kB/s | 27 kB 00:00 2023-09-30T01:51:01.702 INFO:teuthology.orchestra.run.smithi059.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 850 kB/s | 255 kB 00:00 2023-09-30T01:51:01.727 INFO:teuthology.orchestra.run.smithi059.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 310 kB/s | 85 kB 00:00 2023-09-30T01:51:01.877 INFO:teuthology.orchestra.run.smithi059.stdout:(31/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 557 kB/s | 251 kB 00:00 2023-09-30T01:51:01.936 INFO:teuthology.orchestra.run.smithi059.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 98 kB/s | 23 kB 00:00 2023-09-30T01:51:01.941 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:51:01.941 INFO:teuthology.orchestra.run.smithi059.stdout:Total 5.7 MB/s | 28 MB 00:05 2023-09-30T01:51:02.032 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2023-09-30T01:51:02.074 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2023-09-30T01:51:02.074 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2023-09-30T01:51:02.226 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2023-09-30T01:51:02.229 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T01:51:02.229 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2023-09-30T01:51:02.229 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T01:51:02.230 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2023-09-30T01:51:02.230 INFO:teuthology.orchestra.run.smithi110.stdout: ceph x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 6.4 k 2023-09-30T01:51:02.230 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2023-09-30T01:51:02.230 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mds x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 2.3 M 2023-09-30T01:51:02.230 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 1.6 M 2023-09-30T01:51:02.230 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-modules-core noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 238 k 2023-09-30T01:51:02.230 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mon x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.9 M 2023-09-30T01:51:02.230 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-osd x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 17 M 2023-09-30T01:51:02.230 INFO:teuthology.orchestra.run.smithi110.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-09-30T01:51:02.230 INFO:teuthology.orchestra.run.smithi110.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-09-30T01:51:02.230 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-09-30T01:51:02.231 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-09-30T01:51:02.231 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-09-30T01:51:02.231 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-09-30T01:51:02.231 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-09-30T01:51:02.231 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-09-30T01:51:02.231 INFO:teuthology.orchestra.run.smithi110.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-09-30T01:51:02.231 INFO:teuthology.orchestra.run.smithi110.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-09-30T01:51:02.231 INFO:teuthology.orchestra.run.smithi110.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-09-30T01:51:02.231 INFO:teuthology.orchestra.run.smithi110.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-09-30T01:51:02.231 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-09-30T01:51:02.231 INFO:teuthology.orchestra.run.smithi110.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-09-30T01:51:02.232 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-09-30T01:51:02.232 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-09-30T01:51:02.232 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-09-30T01:51:02.232 INFO:teuthology.orchestra.run.smithi110.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-09-30T01:51:02.232 INFO:teuthology.orchestra.run.smithi110.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-09-30T01:51:02.232 INFO:teuthology.orchestra.run.smithi110.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-09-30T01:51:02.232 INFO:teuthology.orchestra.run.smithi110.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-09-30T01:51:02.232 INFO:teuthology.orchestra.run.smithi110.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-09-30T01:51:02.232 INFO:teuthology.orchestra.run.smithi110.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-09-30T01:51:02.232 INFO:teuthology.orchestra.run.smithi110.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-09-30T01:51:02.232 INFO:teuthology.orchestra.run.smithi110.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-09-30T01:51:02.232 INFO:teuthology.orchestra.run.smithi110.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-09-30T01:51:02.233 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:51:02.233 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2023-09-30T01:51:02.233 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T01:51:02.233 INFO:teuthology.orchestra.run.smithi110.stdout:Install 32 Packages 2023-09-30T01:51:02.233 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:51:02.233 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 28 M 2023-09-30T01:51:02.233 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 110 M 2023-09-30T01:51:02.234 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2023-09-30T01:51:02.406 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2023-09-30T01:51:02.409 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2023-09-30T01:51:03.449 INFO:teuthology.orchestra.run.smithi110.stdout:(1/32): ceph-17.2.6-1165.g93dce5e1.el8.x86_64.r 51 kB/s | 6.4 kB 00:00 2023-09-30T01:51:03.535 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2023-09-30T01:51:03.608 INFO:teuthology.orchestra.run.smithi110.stdout:(2/32): ceph-mgr-17.2.6-1165.g93dce5e1.el8.x86_ 5.5 MB/s | 1.6 MB 00:00 2023-09-30T01:51:03.749 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-09-30T01:51:04.000 INFO:teuthology.orchestra.run.smithi110.stdout:(3/32): ceph-mds-17.2.6-1165.g93dce5e1.el8.x86_ 3.4 MB/s | 2.3 MB 00:00 2023-09-30T01:51:04.085 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-09-30T01:51:04.125 INFO:teuthology.orchestra.run.smithi110.stdout:(4/32): ceph-mon-17.2.6-1165.g93dce5e1.el8.x86_ 5.8 MB/s | 3.9 MB 00:00 2023-09-30T01:51:04.151 INFO:teuthology.orchestra.run.smithi110.stdout:(5/32): ceph-mgr-modules-core-17.2.6-1165.g93dc 1.5 MB/s | 238 kB 00:00 2023-09-30T01:51:04.268 INFO:teuthology.orchestra.run.smithi110.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 332 kB/s | 39 kB 00:00 2023-09-30T01:51:04.304 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-09-30T01:51:04.310 INFO:teuthology.orchestra.run.smithi110.stdout:(7/32): python3-mako-1.0.6-14.el8.noarch.rpm 855 kB/s | 157 kB 00:00 2023-09-30T01:51:04.360 INFO:teuthology.orchestra.run.smithi110.stdout:(8/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 1.1 MB/s | 103 kB 00:00 2023-09-30T01:51:04.385 INFO:teuthology.orchestra.run.smithi110.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 757 kB/s | 57 kB 00:00 2023-09-30T01:51:04.499 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-09-30T01:51:04.569 INFO:teuthology.orchestra.run.smithi110.stdout:(10/32): ceph-osd-17.2.6-1165.g93dce5e1.el8.x86 17 MB/s | 17 MB 00:00 2023-09-30T01:51:04.594 INFO:teuthology.orchestra.run.smithi110.stdout:(11/32): python3-werkzeug-0.12.2-4.el8.noarch.r 1.9 MB/s | 457 kB 00:00 2023-09-30T01:51:04.696 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-09-30T01:51:04.786 INFO:teuthology.orchestra.run.smithi110.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 621 kB/s | 249 kB 00:00 2023-09-30T01:51:04.812 INFO:teuthology.orchestra.run.smithi110.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 502 kB/s | 109 kB 00:00 2023-09-30T01:51:04.929 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-09-30T01:51:04.937 INFO:teuthology.orchestra.run.smithi110.stdout:(14/32): python3-cryptography-3.2.1-6.el8.x86_6 1.7 MB/s | 641 kB 00:00 2023-09-30T01:51:05.054 INFO:teuthology.orchestra.run.smithi110.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 163 kB/s | 44 kB 00:00 2023-09-30T01:51:05.088 INFO:teuthology.orchestra.run.smithi110.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 672 kB/s | 185 kB 00:00 2023-09-30T01:51:05.146 INFO:teuthology.orchestra.run.smithi110.stdout:(17/32): python3-jaraco-6.2-6.el8.noarch.rpm 191 kB/s | 11 kB 00:00 2023-09-30T01:51:05.205 INFO:teuthology.orchestra.run.smithi110.stdout:(18/32): python3-jaraco-functools-2.0-4.el8.noa 302 kB/s | 18 kB 00:00 2023-09-30T01:51:05.230 INFO:teuthology.orchestra.run.smithi110.stdout:(19/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 590 kB/s | 173 kB 00:00 2023-09-30T01:51:05.281 INFO:teuthology.orchestra.run.smithi110.stdout:(20/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.7 MB/s | 384 kB 00:00 2023-09-30T01:51:05.306 INFO:teuthology.orchestra.run.smithi110.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 485 kB/s | 49 kB 00:00 2023-09-30T01:51:05.331 INFO:teuthology.orchestra.run.smithi110.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 583 kB/s | 59 kB 00:00 2023-09-30T01:51:05.356 INFO:teuthology.orchestra.run.smithi110.stdout:(23/32): python3-pecan-1.3.2-9.el8.noarch.rpm 3.7 MB/s | 283 kB 00:00 2023-09-30T01:51:05.369 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-09-30T01:51:05.390 INFO:teuthology.orchestra.run.smithi110.stdout:(24/32): python3-portend-2.6-1.el8.noarch.rpm 267 kB/s | 16 kB 00:00 2023-09-30T01:51:05.415 INFO:teuthology.orchestra.run.smithi110.stdout:(25/32): python3-simplegeneric-0.8.1-17.el8.noa 224 kB/s | 19 kB 00:00 2023-09-30T01:51:05.440 INFO:teuthology.orchestra.run.smithi110.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 285 kB/s | 24 kB 00:00 2023-09-30T01:51:05.474 INFO:teuthology.orchestra.run.smithi110.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 347 kB/s | 29 kB 00:00 2023-09-30T01:51:05.500 INFO:teuthology.orchestra.run.smithi110.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 327 kB/s | 27 kB 00:00 2023-09-30T01:51:05.525 INFO:teuthology.orchestra.run.smithi110.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 3.0 MB/s | 255 kB 00:00 2023-09-30T01:51:05.550 INFO:teuthology.orchestra.run.smithi110.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.7 MB/s | 85 kB 00:00 2023-09-30T01:51:05.575 INFO:teuthology.orchestra.run.smithi110.stdout:(31/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.4 MB/s | 251 kB 00:00 2023-09-30T01:51:05.589 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-09-30T01:51:05.601 INFO:teuthology.orchestra.run.smithi110.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 304 kB/s | 23 kB 00:00 2023-09-30T01:51:05.605 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:51:05.605 INFO:teuthology.orchestra.run.smithi110.stdout:Total 8.4 MB/s | 28 MB 00:03 2023-09-30T01:51:05.690 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2023-09-30T01:51:05.726 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2023-09-30T01:51:05.726 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2023-09-30T01:51:05.852 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-09-30T01:51:06.022 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2023-09-30T01:51:06.025 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2023-09-30T01:51:06.079 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-09-30T01:51:06.363 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-09-30T01:51:06.610 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-09-30T01:51:07.037 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2023-09-30T01:51:07.084 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2023-09-30T01:51:07.290 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 14/32 2023-09-30T01:51:07.325 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-09-30T01:51:07.516 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-09-30T01:51:07.545 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-09-30T01:51:07.762 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-09-30T01:51:07.809 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-09-30T01:51:07.960 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-09-30T01:51:08.035 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-09-30T01:51:08.158 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-09-30T01:51:08.388 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-09-30T01:51:08.406 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-09-30T01:51:08.565 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-09-30T01:51:08.608 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-09-30T01:51:08.801 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-09-30T01:51:08.854 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2023-09-30T01:51:09.028 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-09-30T01:51:09.046 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-09-30T01:51:09.344 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-09-30T01:51:09.555 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-09-30T01:51:09.845 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-09-30T01:51:09.881 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-09-30T01:51:10.116 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-09-30T01:51:10.181 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-09-30T01:51:10.325 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-09-30T01:51:10.546 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-09-30T01:51:10.707 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2023-09-30T01:51:10.973 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 14/32 2023-09-30T01:51:11.180 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-09-30T01:51:11.337 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-modules-core-2:17.2.6-1165.g93dce5e1.el8. 27/32 2023-09-30T01:51:11.393 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-09-30T01:51:11.524 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 28/32 2023-09-30T01:51:11.645 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-09-30T01:51:11.962 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-09-30T01:51:12.142 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-09-30T01:51:12.330 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2023-09-30T01:51:12.516 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-09-30T01:51:12.774 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-09-30T01:51:13.032 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-09-30T01:51:13.179 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 28/32 2023-09-30T01:51:13.273 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-09-30T01:51:13.374 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 29/32 2023-09-30T01:51:13.589 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-09-30T01:51:13.814 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 29/32 2023-09-30T01:51:13.899 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-09-30T01:51:14.015 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 30/32 2023-09-30T01:51:14.252 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-modules-core-2:17.2.6-1165.g93dce5e1.el8. 27/32 2023-09-30T01:51:14.257 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 30/32 2023-09-30T01:51:14.415 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 31/32 2023-09-30T01:51:14.437 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 31/32 2023-09-30T01:51:14.481 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 28/32 2023-09-30T01:51:14.566 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 32/32 2023-09-30T01:51:15.029 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 32/32 2023-09-30T01:51:15.029 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/32 2023-09-30T01:51:15.030 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/32 2023-09-30T01:51:15.030 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/32 2023-09-30T01:51:15.030 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 4/32 2023-09-30T01:51:15.030 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/32 2023-09-30T01:51:15.030 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-modules-core-2:17.2.6-1165.g93dce5e1.el8. 6/32 2023-09-30T01:51:15.030 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-09-30T01:51:15.030 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-09-30T01:51:15.030 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-09-30T01:51:15.030 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2023-09-30T01:51:15.030 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-09-30T01:51:15.030 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2023-09-30T01:51:15.030 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 13/32 2023-09-30T01:51:15.031 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-09-30T01:51:15.031 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-09-30T01:51:15.031 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-09-30T01:51:15.031 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-09-30T01:51:15.031 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-09-30T01:51:15.031 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-09-30T01:51:15.031 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-09-30T01:51:15.031 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-09-30T01:51:15.031 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-09-30T01:51:15.031 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-09-30T01:51:15.031 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-09-30T01:51:15.031 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-09-30T01:51:15.032 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-09-30T01:51:15.032 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-09-30T01:51:15.032 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-09-30T01:51:15.032 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-09-30T01:51:15.033 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-09-30T01:51:15.033 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-09-30T01:51:15.963 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 28/32 2023-09-30T01:51:16.164 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 29/32 2023-09-30T01:51:16.190 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-09-30T01:51:16.190 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:51:16.190 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2023-09-30T01:51:16.190 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:51:16.190 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:51:16.190 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:51:16.191 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-modules-core-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T01:51:16.191 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:51:16.191 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:51:16.191 INFO:teuthology.orchestra.run.smithi059.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-09-30T01:51:16.191 INFO:teuthology.orchestra.run.smithi059.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-09-30T01:51:16.191 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-09-30T01:51:16.191 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-09-30T01:51:16.191 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-09-30T01:51:16.191 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-09-30T01:51:16.191 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-6.2-6.el8.noarch 2023-09-30T01:51:16.192 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-09-30T01:51:16.192 INFO:teuthology.orchestra.run.smithi059.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-09-30T01:51:16.192 INFO:teuthology.orchestra.run.smithi059.stdout: python3-mako-1.0.6-14.el8.noarch 2023-09-30T01:51:16.192 INFO:teuthology.orchestra.run.smithi059.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-09-30T01:51:16.192 INFO:teuthology.orchestra.run.smithi059.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-09-30T01:51:16.192 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-09-30T01:51:16.192 INFO:teuthology.orchestra.run.smithi059.stdout: python3-portend-2.6-1.el8.noarch 2023-09-30T01:51:16.192 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-09-30T01:51:16.192 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pycparser-2.14-14.el8.noarch 2023-09-30T01:51:16.192 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pytz-2017.2-11.el8.noarch 2023-09-30T01:51:16.192 INFO:teuthology.orchestra.run.smithi059.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-09-30T01:51:16.192 INFO:teuthology.orchestra.run.smithi059.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-09-30T01:51:16.193 INFO:teuthology.orchestra.run.smithi059.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-09-30T01:51:16.193 INFO:teuthology.orchestra.run.smithi059.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-09-30T01:51:16.193 INFO:teuthology.orchestra.run.smithi059.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-09-30T01:51:16.193 INFO:teuthology.orchestra.run.smithi059.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-09-30T01:51:16.193 INFO:teuthology.orchestra.run.smithi059.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-09-30T01:51:16.193 INFO:teuthology.orchestra.run.smithi059.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-09-30T01:51:16.193 INFO:teuthology.orchestra.run.smithi059.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-09-30T01:51:16.193 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:51:16.193 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-09-30T01:51:16.556 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 29/32 2023-09-30T01:51:16.761 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 30/32 2023-09-30T01:51:16.978 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 30/32 2023-09-30T01:51:16.995 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-base 2023-09-30T01:51:17.136 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 31/32 2023-09-30T01:51:17.154 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 31/32 2023-09-30T01:51:17.261 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 32/32 2023-09-30T01:51:17.656 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 32/32 2023-09-30T01:51:17.657 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/32 2023-09-30T01:51:17.657 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/32 2023-09-30T01:51:17.657 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/32 2023-09-30T01:51:17.657 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 4/32 2023-09-30T01:51:17.657 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/32 2023-09-30T01:51:17.657 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-modules-core-2:17.2.6-1165.g93dce5e1.el8. 6/32 2023-09-30T01:51:17.657 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-09-30T01:51:17.657 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-09-30T01:51:17.657 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-09-30T01:51:17.658 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2023-09-30T01:51:17.658 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-09-30T01:51:17.658 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2023-09-30T01:51:17.658 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 13/32 2023-09-30T01:51:17.658 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-09-30T01:51:17.658 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-09-30T01:51:17.658 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-09-30T01:51:17.658 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-09-30T01:51:17.658 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-09-30T01:51:17.658 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-09-30T01:51:17.658 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-09-30T01:51:17.658 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-09-30T01:51:17.659 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-09-30T01:51:17.659 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-09-30T01:51:17.659 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-09-30T01:51:17.659 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-09-30T01:51:17.659 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-09-30T01:51:17.659 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-09-30T01:51:17.660 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-09-30T01:51:17.660 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-09-30T01:51:17.660 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-09-30T01:51:17.660 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-09-30T01:51:17.987 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:25 ago on Sat 30 Sep 2023 01:49:52 AM UTC. 2023-09-30T01:51:18.755 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-09-30T01:51:18.755 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:51:18.755 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2023-09-30T01:51:18.755 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:51:18.755 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:51:18.755 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:51:18.755 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-modules-core-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T01:51:18.756 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:51:18.756 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:51:18.756 INFO:teuthology.orchestra.run.smithi110.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-09-30T01:51:18.756 INFO:teuthology.orchestra.run.smithi110.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-09-30T01:51:18.756 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-09-30T01:51:18.756 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-09-30T01:51:18.756 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-09-30T01:51:18.756 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-09-30T01:51:18.756 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-6.2-6.el8.noarch 2023-09-30T01:51:18.756 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-09-30T01:51:18.756 INFO:teuthology.orchestra.run.smithi110.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-09-30T01:51:18.757 INFO:teuthology.orchestra.run.smithi110.stdout: python3-mako-1.0.6-14.el8.noarch 2023-09-30T01:51:18.757 INFO:teuthology.orchestra.run.smithi110.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-09-30T01:51:18.757 INFO:teuthology.orchestra.run.smithi110.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-09-30T01:51:18.757 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-09-30T01:51:18.757 INFO:teuthology.orchestra.run.smithi110.stdout: python3-portend-2.6-1.el8.noarch 2023-09-30T01:51:18.757 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-09-30T01:51:18.757 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pycparser-2.14-14.el8.noarch 2023-09-30T01:51:18.757 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pytz-2017.2-11.el8.noarch 2023-09-30T01:51:18.757 INFO:teuthology.orchestra.run.smithi110.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-09-30T01:51:18.757 INFO:teuthology.orchestra.run.smithi110.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-09-30T01:51:18.757 INFO:teuthology.orchestra.run.smithi110.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-09-30T01:51:18.757 INFO:teuthology.orchestra.run.smithi110.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-09-30T01:51:18.757 INFO:teuthology.orchestra.run.smithi110.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-09-30T01:51:18.758 INFO:teuthology.orchestra.run.smithi110.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-09-30T01:51:18.758 INFO:teuthology.orchestra.run.smithi110.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-09-30T01:51:18.758 INFO:teuthology.orchestra.run.smithi110.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-09-30T01:51:18.758 INFO:teuthology.orchestra.run.smithi110.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-09-30T01:51:18.758 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:51:18.758 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2023-09-30T01:51:18.988 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-base 2023-09-30T01:51:19.537 INFO:teuthology.orchestra.run.smithi059.stdout:Package ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T01:51:19.754 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-09-30T01:51:19.756 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2023-09-30T01:51:19.756 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-09-30T01:51:19.833 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:29 ago on Sat 30 Sep 2023 01:49:50 AM UTC. 2023-09-30T01:51:19.857 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install cephadm 2023-09-30T01:51:20.846 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:28 ago on Sat 30 Sep 2023 01:49:52 AM UTC. 2023-09-30T01:51:21.242 INFO:teuthology.orchestra.run.smithi110.stdout:Package ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T01:51:21.431 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2023-09-30T01:51:21.432 INFO:teuthology.orchestra.run.smithi110.stdout:Nothing to do. 2023-09-30T01:51:21.432 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2023-09-30T01:51:21.534 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install cephadm 2023-09-30T01:51:22.378 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:32 ago on Sat 30 Sep 2023 01:49:50 AM UTC. 2023-09-30T01:51:22.630 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-09-30T01:51:22.630 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T01:51:22.630 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2023-09-30T01:51:22.631 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T01:51:22.631 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2023-09-30T01:51:22.631 INFO:teuthology.orchestra.run.smithi059.stdout: cephadm noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 80 k 2023-09-30T01:51:22.631 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:51:22.631 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2023-09-30T01:51:22.631 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T01:51:22.631 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2023-09-30T01:51:22.631 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:51:22.632 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 80 k 2023-09-30T01:51:22.632 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 363 k 2023-09-30T01:51:22.632 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2023-09-30T01:51:23.159 INFO:teuthology.orchestra.run.smithi059.stdout:cephadm-17.2.6-1165.g93dce5e1.el8.noarch.rpm 152 kB/s | 80 kB 00:00 2023-09-30T01:51:23.160 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:51:23.160 INFO:teuthology.orchestra.run.smithi059.stdout:Total 151 kB/s | 80 kB 00:00 2023-09-30T01:51:23.160 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2023-09-30T01:51:23.170 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2023-09-30T01:51:23.170 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2023-09-30T01:51:23.173 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2023-09-30T01:51:23.176 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2023-09-30T01:51:23.682 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2023-09-30T01:51:23.959 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2023-09-30T01:51:23.959 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T01:51:23.959 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2023-09-30T01:51:23.960 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T01:51:23.960 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2023-09-30T01:51:23.960 INFO:teuthology.orchestra.run.smithi110.stdout: cephadm noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 80 k 2023-09-30T01:51:23.960 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:51:23.960 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2023-09-30T01:51:23.960 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T01:51:23.960 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2023-09-30T01:51:23.960 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:51:23.961 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 80 k 2023-09-30T01:51:23.961 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 363 k 2023-09-30T01:51:23.961 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2023-09-30T01:51:24.220 INFO:teuthology.orchestra.run.smithi110.stdout:cephadm-17.2.6-1165.g93dce5e1.el8.noarch.rpm 309 kB/s | 80 kB 00:00 2023-09-30T01:51:24.220 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:51:24.221 INFO:teuthology.orchestra.run.smithi110.stdout:Total 308 kB/s | 80 kB 00:00 2023-09-30T01:51:24.221 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2023-09-30T01:51:24.229 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2023-09-30T01:51:24.229 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2023-09-30T01:51:24.231 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2023-09-30T01:51:24.234 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2023-09-30T01:51:24.278 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2023-09-30T01:51:24.353 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T01:51:24.494 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T01:51:24.756 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T01:51:24.765 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T01:51:25.103 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T01:51:25.338 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T01:51:25.444 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T01:51:25.444 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:51:25.444 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2023-09-30T01:51:25.444 INFO:teuthology.orchestra.run.smithi059.stdout: cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T01:51:25.444 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:51:25.444 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-09-30T01:51:25.645 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-immutable-object-cache 2023-09-30T01:51:26.466 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T01:51:26.466 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:51:26.466 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2023-09-30T01:51:26.466 INFO:teuthology.orchestra.run.smithi110.stdout: cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T01:51:26.466 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:51:26.466 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2023-09-30T01:51:26.630 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:34 ago on Sat 30 Sep 2023 01:49:52 AM UTC. 2023-09-30T01:51:26.689 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-immutable-object-cache 2023-09-30T01:51:27.548 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:37 ago on Sat 30 Sep 2023 01:49:50 AM UTC. 2023-09-30T01:51:28.420 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-09-30T01:51:28.420 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T01:51:28.420 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repo Size 2023-09-30T01:51:28.420 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T01:51:28.421 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2023-09-30T01:51:28.421 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-immutable-object-cache x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 161 k 2023-09-30T01:51:28.421 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:51:28.421 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2023-09-30T01:51:28.421 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T01:51:28.421 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2023-09-30T01:51:28.421 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:51:28.422 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 161 k 2023-09-30T01:51:28.422 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 490 k 2023-09-30T01:51:28.422 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2023-09-30T01:51:28.597 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-immutable-object-cache-17.2.6-1165.g93dce5 926 kB/s | 161 kB 00:00 2023-09-30T01:51:28.597 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:51:28.597 INFO:teuthology.orchestra.run.smithi059.stdout:Total 918 kB/s | 161 kB 00:00 2023-09-30T01:51:28.598 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2023-09-30T01:51:28.611 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2023-09-30T01:51:28.611 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2023-09-30T01:51:28.667 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2023-09-30T01:51:28.670 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2023-09-30T01:51:28.793 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2023-09-30T01:51:28.971 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1. 1/1 2023-09-30T01:51:29.126 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2023-09-30T01:51:29.126 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T01:51:29.126 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2023-09-30T01:51:29.126 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T01:51:29.126 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2023-09-30T01:51:29.126 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-immutable-object-cache x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 161 k 2023-09-30T01:51:29.126 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:51:29.127 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2023-09-30T01:51:29.127 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T01:51:29.127 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2023-09-30T01:51:29.127 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:51:29.127 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 161 k 2023-09-30T01:51:29.127 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 490 k 2023-09-30T01:51:29.127 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2023-09-30T01:51:29.308 INFO:teuthology.orchestra.run.smithi110.stdout:ceph-immutable-object-cache-17.2.6-1165.g93dce5 879 kB/s | 161 kB 00:00 2023-09-30T01:51:29.309 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:51:29.309 INFO:teuthology.orchestra.run.smithi110.stdout:Total 872 kB/s | 161 kB 00:00 2023-09-30T01:51:29.309 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2023-09-30T01:51:29.322 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2023-09-30T01:51:29.323 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2023-09-30T01:51:29.368 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2023-09-30T01:51:29.370 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2023-09-30T01:51:29.437 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1. 1/1 2023-09-30T01:51:29.475 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2023-09-30T01:51:29.669 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1. 1/1 2023-09-30T01:51:30.087 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1. 1/1 2023-09-30T01:51:30.121 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1. 1/1 2023-09-30T01:51:30.121 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:51:30.121 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2023-09-30T01:51:30.121 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:51:30.122 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:51:30.122 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-09-30T01:51:30.340 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-mgr 2023-09-30T01:51:30.726 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1. 1/1 2023-09-30T01:51:30.726 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:51:30.726 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2023-09-30T01:51:30.726 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:51:30.726 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:51:30.726 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2023-09-30T01:51:30.943 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-mgr 2023-09-30T01:51:31.326 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:39 ago on Sat 30 Sep 2023 01:49:52 AM UTC. 2023-09-30T01:51:31.848 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:41 ago on Sat 30 Sep 2023 01:49:50 AM UTC. 2023-09-30T01:51:32.873 INFO:teuthology.orchestra.run.smithi059.stdout:Package ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T01:51:33.084 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-09-30T01:51:33.085 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2023-09-30T01:51:33.086 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-09-30T01:51:33.187 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-mgr-dashboard 2023-09-30T01:51:33.255 INFO:teuthology.orchestra.run.smithi110.stdout:Package ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T01:51:33.446 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2023-09-30T01:51:33.446 INFO:teuthology.orchestra.run.smithi110.stdout:Nothing to do. 2023-09-30T01:51:33.446 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2023-09-30T01:51:33.538 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-mgr-dashboard 2023-09-30T01:51:34.167 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:42 ago on Sat 30 Sep 2023 01:49:52 AM UTC. 2023-09-30T01:51:34.384 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:44 ago on Sat 30 Sep 2023 01:49:50 AM UTC. 2023-09-30T01:51:35.919 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-09-30T01:51:35.920 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T01:51:35.920 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2023-09-30T01:51:35.920 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T01:51:35.921 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2023-09-30T01:51:35.921 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-dashboard noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 1.7 M 2023-09-30T01:51:35.921 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2023-09-30T01:51:35.921 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-grafana-dashboards noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 23 k 2023-09-30T01:51:35.921 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-prometheus-alerts noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 14 k 2023-09-30T01:51:35.921 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-09-30T01:51:35.921 INFO:teuthology.orchestra.run.smithi059.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-09-30T01:51:35.921 INFO:teuthology.orchestra.run.smithi059.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-09-30T01:51:35.921 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:51:35.921 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2023-09-30T01:51:35.921 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T01:51:35.922 INFO:teuthology.orchestra.run.smithi059.stdout:Install 6 Packages 2023-09-30T01:51:35.922 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:51:35.922 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 2.0 M 2023-09-30T01:51:35.922 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 8.0 M 2023-09-30T01:51:35.922 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2023-09-30T01:51:35.966 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2023-09-30T01:51:35.967 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T01:51:35.967 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2023-09-30T01:51:35.967 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T01:51:35.967 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2023-09-30T01:51:35.967 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-dashboard noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 1.7 M 2023-09-30T01:51:35.968 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2023-09-30T01:51:35.968 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-grafana-dashboards noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 23 k 2023-09-30T01:51:35.968 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-prometheus-alerts noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 14 k 2023-09-30T01:51:35.968 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-09-30T01:51:35.968 INFO:teuthology.orchestra.run.smithi110.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-09-30T01:51:35.968 INFO:teuthology.orchestra.run.smithi110.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-09-30T01:51:35.968 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:51:35.968 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2023-09-30T01:51:35.968 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T01:51:35.968 INFO:teuthology.orchestra.run.smithi110.stdout:Install 6 Packages 2023-09-30T01:51:35.968 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:51:35.969 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 2.0 M 2023-09-30T01:51:35.969 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 8.0 M 2023-09-30T01:51:35.969 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2023-09-30T01:51:36.283 INFO:teuthology.orchestra.run.smithi059.stdout:(1/6): ceph-grafana-dashboards-17.2.6-1165.g93d 159 kB/s | 23 kB 00:00 2023-09-30T01:51:36.314 INFO:teuthology.orchestra.run.smithi110.stdout:(1/6): ceph-grafana-dashboards-17.2.6-1165.g93d 100 kB/s | 23 kB 00:00 2023-09-30T01:51:36.332 INFO:teuthology.orchestra.run.smithi059.stdout:(2/6): ceph-prometheus-alerts-17.2.6-1165.g93dc 75 kB/s | 14 kB 00:00 2023-09-30T01:51:36.337 INFO:teuthology.orchestra.run.smithi110.stdout:(2/6): ceph-prometheus-alerts-17.2.6-1165.g93dc 57 kB/s | 14 kB 00:00 2023-09-30T01:51:36.396 INFO:teuthology.orchestra.run.smithi110.stdout:(3/6): ceph-mgr-dashboard-17.2.6-1165.g93dce5e1 5.4 MB/s | 1.7 MB 00:00 2023-09-30T01:51:36.421 INFO:teuthology.orchestra.run.smithi110.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 401 kB/s | 34 kB 00:00 2023-09-30T01:51:36.467 INFO:teuthology.orchestra.run.smithi059.stdout:(3/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 251 kB/s | 34 kB 00:00 2023-09-30T01:51:36.488 INFO:teuthology.orchestra.run.smithi110.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 2.1 MB/s | 196 kB 00:00 2023-09-30T01:51:36.492 INFO:teuthology.orchestra.run.smithi059.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 204 kB/s | 43 kB 00:00 2023-09-30T01:51:36.522 INFO:teuthology.orchestra.run.smithi110.stdout:(6/6): python3-jwt-1.6.1-2.el8.noarch.rpm 203 kB/s | 43 kB 00:00 2023-09-30T01:51:36.523 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:51:36.523 INFO:teuthology.orchestra.run.smithi110.stdout:Total 3.6 MB/s | 2.0 MB 00:00 2023-09-30T01:51:36.534 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2023-09-30T01:51:36.543 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2023-09-30T01:51:36.543 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2023-09-30T01:51:36.617 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2023-09-30T01:51:36.618 INFO:teuthology.orchestra.run.smithi059.stdout:(5/6): ceph-mgr-dashboard-17.2.6-1165.g93dce5e1 3.5 MB/s | 1.7 MB 00:00 2023-09-30T01:51:36.620 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2023-09-30T01:51:36.642 INFO:teuthology.orchestra.run.smithi059.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.1 MB/s | 196 kB 00:00 2023-09-30T01:51:36.645 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:51:36.646 INFO:teuthology.orchestra.run.smithi059.stdout:Total 2.8 MB/s | 2.0 MB 00:00 2023-09-30T01:51:36.658 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2023-09-30T01:51:36.669 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2023-09-30T01:51:36.669 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2023-09-30T01:51:36.758 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2023-09-30T01:51:36.761 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2023-09-30T01:51:36.876 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2023-09-30T01:51:37.059 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-09-30T01:51:37.059 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2023-09-30T01:51:37.247 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-09-30T01:51:37.286 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-09-30T01:51:37.461 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-09-30T01:51:37.576 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-09-30T01:51:37.589 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-prometheus-alerts-2:17.2.6-1165.g93dce5e1.el8.n 4/6 2023-09-30T01:51:37.764 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-09-30T01:51:37.911 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-prometheus-alerts-2:17.2.6-1165.g93dce5e1.el8.n 4/6 2023-09-30T01:51:37.921 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-grafana-dashboards-2:17.2.6-1165.g93dce5e1.el8. 5/6 2023-09-30T01:51:38.119 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarc 6/6 2023-09-30T01:51:38.250 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-grafana-dashboards-2:17.2.6-1165.g93dce5e1.el8. 5/6 2023-09-30T01:51:38.403 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarc 6/6 2023-09-30T01:51:38.403 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-grafana-dashboards-2:17.2.6-1165.g93dce5e1.el8. 1/6 2023-09-30T01:51:38.403 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarc 2/6 2023-09-30T01:51:38.403 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-prometheus-alerts-2:17.2.6-1165.g93dce5e1.el8.n 3/6 2023-09-30T01:51:38.403 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-09-30T01:51:38.404 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-09-30T01:51:38.462 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarc 6/6 2023-09-30T01:51:38.783 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarc 6/6 2023-09-30T01:51:38.783 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-grafana-dashboards-2:17.2.6-1165.g93dce5e1.el8. 1/6 2023-09-30T01:51:38.783 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarc 2/6 2023-09-30T01:51:38.783 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-prometheus-alerts-2:17.2.6-1165.g93dce5e1.el8.n 3/6 2023-09-30T01:51:38.784 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-09-30T01:51:38.784 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-09-30T01:51:39.018 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-09-30T01:51:39.019 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:51:39.019 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2023-09-30T01:51:39.019 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-grafana-dashboards-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T01:51:39.019 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T01:51:39.019 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-prometheus-alerts-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T01:51:39.020 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-09-30T01:51:39.020 INFO:teuthology.orchestra.run.smithi110.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-09-30T01:51:39.020 INFO:teuthology.orchestra.run.smithi110.stdout: python3-routes-2.4.1-12.el8.noarch 2023-09-30T01:51:39.020 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:51:39.020 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2023-09-30T01:51:39.130 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-mgr-diskprediction-local 2023-09-30T01:51:39.501 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-09-30T01:51:39.502 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:51:39.502 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2023-09-30T01:51:39.502 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-grafana-dashboards-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T01:51:39.502 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T01:51:39.502 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-prometheus-alerts-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T01:51:39.502 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-09-30T01:51:39.502 INFO:teuthology.orchestra.run.smithi059.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-09-30T01:51:39.503 INFO:teuthology.orchestra.run.smithi059.stdout: python3-routes-2.4.1-12.el8.noarch 2023-09-30T01:51:39.503 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:51:39.503 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-09-30T01:51:39.634 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-mgr-diskprediction-local 2023-09-30T01:51:39.973 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:01:49 ago on Sat 30 Sep 2023 01:49:50 AM UTC. 2023-09-30T01:51:40.617 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:48 ago on Sat 30 Sep 2023 01:49:52 AM UTC. 2023-09-30T01:51:41.544 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2023-09-30T01:51:41.545 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================================ 2023-09-30T01:51:41.545 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2023-09-30T01:51:41.545 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================================ 2023-09-30T01:51:41.545 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2023-09-30T01:51:41.545 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 7.4 M 2023-09-30T01:51:41.546 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2023-09-30T01:51:41.546 INFO:teuthology.orchestra.run.smithi110.stdout: libgfortran x86_64 8.5.0-20.el8 baseos 645 k 2023-09-30T01:51:41.546 INFO:teuthology.orchestra.run.smithi110.stdout: libquadmath x86_64 8.5.0-20.el8 baseos 172 k 2023-09-30T01:51:41.546 INFO:teuthology.orchestra.run.smithi110.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-09-30T01:51:41.546 INFO:teuthology.orchestra.run.smithi110.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-09-30T01:51:41.546 INFO:teuthology.orchestra.run.smithi110.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-09-30T01:51:41.546 INFO:teuthology.orchestra.run.smithi110.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-09-30T01:51:41.546 INFO:teuthology.orchestra.run.smithi110.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 appstream 14 M 2023-09-30T01:51:41.546 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:51:41.546 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2023-09-30T01:51:41.546 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================================ 2023-09-30T01:51:41.547 INFO:teuthology.orchestra.run.smithi110.stdout:Install 8 Packages 2023-09-30T01:51:41.547 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:51:41.547 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 35 M 2023-09-30T01:51:41.547 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 210 M 2023-09-30T01:51:41.547 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2023-09-30T01:51:42.094 INFO:teuthology.orchestra.run.smithi110.stdout:(1/8): openblas-0.3.15-4.el8.x86_64.rpm 10 MB/s | 4.7 MB 00:00 2023-09-30T01:51:42.161 INFO:teuthology.orchestra.run.smithi110.stdout:(2/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 9.2 MB/s | 4.9 MB 00:00 2023-09-30T01:51:42.253 INFO:teuthology.orchestra.run.smithi110.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.4 MB/s | 225 kB 00:00 2023-09-30T01:51:42.370 INFO:teuthology.orchestra.run.smithi110.stdout:(4/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 14 MB/s | 3.7 MB 00:00 2023-09-30T01:51:42.371 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-09-30T01:51:42.372 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================================ 2023-09-30T01:51:42.373 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2023-09-30T01:51:42.373 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================================ 2023-09-30T01:51:42.373 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2023-09-30T01:51:42.373 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 7.4 M 2023-09-30T01:51:42.373 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2023-09-30T01:51:42.374 INFO:teuthology.orchestra.run.smithi059.stdout: libgfortran x86_64 8.5.0-20.el8 baseos 645 k 2023-09-30T01:51:42.374 INFO:teuthology.orchestra.run.smithi059.stdout: libquadmath x86_64 8.5.0-20.el8 baseos 172 k 2023-09-30T01:51:42.374 INFO:teuthology.orchestra.run.smithi059.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-09-30T01:51:42.374 INFO:teuthology.orchestra.run.smithi059.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-09-30T01:51:42.374 INFO:teuthology.orchestra.run.smithi059.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-09-30T01:51:42.374 INFO:teuthology.orchestra.run.smithi059.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-09-30T01:51:42.375 INFO:teuthology.orchestra.run.smithi059.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 appstream 14 M 2023-09-30T01:51:42.375 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:51:42.375 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2023-09-30T01:51:42.376 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================================ 2023-09-30T01:51:42.376 INFO:teuthology.orchestra.run.smithi059.stdout:Install 8 Packages 2023-09-30T01:51:42.376 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:51:42.377 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 35 M 2023-09-30T01:51:42.377 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 210 M 2023-09-30T01:51:42.377 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2023-09-30T01:51:42.706 INFO:teuthology.orchestra.run.smithi110.stdout:(5/8): ceph-mgr-diskprediction-local-17.2.6-116 6.9 MB/s | 7.4 MB 00:01 2023-09-30T01:51:42.731 INFO:teuthology.orchestra.run.smithi110.stdout:(6/8): libgfortran-8.5.0-20.el8.x86_64.rpm 1.7 MB/s | 645 kB 00:00 2023-09-30T01:51:42.864 INFO:teuthology.orchestra.run.smithi110.stdout:(7/8): python3-scipy-1.0.0-21.module_el8.5.0+77 22 MB/s | 14 MB 00:00 2023-09-30T01:51:43.056 INFO:teuthology.orchestra.run.smithi110.stdout:(8/8): libquadmath-8.5.0-20.el8.x86_64.rpm 492 kB/s | 172 kB 00:00 2023-09-30T01:51:43.057 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:51:43.057 INFO:teuthology.orchestra.run.smithi110.stdout:Total 23 MB/s | 35 MB 00:01 2023-09-30T01:51:43.066 INFO:teuthology.orchestra.run.smithi059.stdout:(1/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 11 MB/s | 4.9 MB 00:00 2023-09-30T01:51:43.166 INFO:teuthology.orchestra.run.smithi059.stdout:(2/8): openblas-0.3.15-4.el8.x86_64.rpm 8.6 MB/s | 4.7 MB 00:00 2023-09-30T01:51:43.266 INFO:teuthology.orchestra.run.smithi059.stdout:(3/8): ceph-mgr-diskprediction-local-17.2.6-116 11 MB/s | 7.4 MB 00:00 2023-09-30T01:51:43.291 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2023-09-30T01:51:43.291 INFO:teuthology.orchestra.run.smithi059.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 1.8 MB/s | 225 kB 00:00 2023-09-30T01:51:43.309 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2023-09-30T01:51:43.310 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2023-09-30T01:51:43.360 INFO:teuthology.orchestra.run.smithi059.stdout:(5/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 13 MB/s | 3.7 MB 00:00 2023-09-30T01:51:43.594 INFO:teuthology.orchestra.run.smithi059.stdout:(6/8): libquadmath-8.5.0-20.el8.x86_64.rpm 739 kB/s | 172 kB 00:00 2023-09-30T01:51:43.651 INFO:teuthology.orchestra.run.smithi059.stdout:(7/8): libgfortran-8.5.0-20.el8.x86_64.rpm 1.8 MB/s | 645 kB 00:00 2023-09-30T01:51:43.692 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2023-09-30T01:51:43.695 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2023-09-30T01:51:43.978 INFO:teuthology.orchestra.run.smithi059.stdout:(8/8): python3-scipy-1.0.0-21.module_el8.5.0+77 19 MB/s | 14 MB 00:00 2023-09-30T01:51:43.978 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:51:43.979 INFO:teuthology.orchestra.run.smithi059.stdout:Total 22 MB/s | 35 MB 00:01 2023-09-30T01:51:44.231 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2023-09-30T01:51:44.238 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2023-09-30T01:51:44.251 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2023-09-30T01:51:44.252 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2023-09-30T01:51:44.440 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 1/8 2023-09-30T01:51:44.574 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 1/8 2023-09-30T01:51:44.724 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 2/8 2023-09-30T01:51:45.183 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2023-09-30T01:51:45.187 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2023-09-30T01:51:45.324 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 2/8 2023-09-30T01:51:45.541 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-09-30T01:51:45.924 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2023-09-30T01:51:46.167 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-09-30T01:51:46.219 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 1/8 2023-09-30T01:51:46.358 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 1/8 2023-09-30T01:51:46.374 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-09-30T01:51:46.493 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 2/8 2023-09-30T01:51:47.082 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-09-30T01:51:47.155 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 2/8 2023-09-30T01:51:47.344 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-09-30T01:51:47.437 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-09-30T01:51:47.682 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-09-30T01:51:48.031 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-09-30T01:51:48.185 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-09-30T01:51:48.967 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-09-30T01:51:49.343 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-09-30T01:51:49.466 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-09-30T01:51:49.593 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-09-30T01:51:50.679 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-09-30T01:51:50.890 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e 8/8 2023-09-30T01:51:51.190 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e 8/8 2023-09-30T01:51:51.191 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e 1/8 2023-09-30T01:51:51.191 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-09-30T01:51:51.191 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-09-30T01:51:51.191 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-09-30T01:51:51.191 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-09-30T01:51:51.191 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-09-30T01:51:51.191 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 7/8 2023-09-30T01:51:51.587 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-09-30T01:51:51.832 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 8/8 2023-09-30T01:51:51.832 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:51:51.832 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2023-09-30T01:51:51.832 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T01:51:51.832 INFO:teuthology.orchestra.run.smithi110.stdout: libgfortran-8.5.0-20.el8.x86_64 2023-09-30T01:51:51.832 INFO:teuthology.orchestra.run.smithi110.stdout: libquadmath-8.5.0-20.el8.x86_64 2023-09-30T01:51:51.833 INFO:teuthology.orchestra.run.smithi110.stdout: openblas-0.3.15-4.el8.x86_64 2023-09-30T01:51:51.833 INFO:teuthology.orchestra.run.smithi110.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-09-30T01:51:51.833 INFO:teuthology.orchestra.run.smithi110.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-09-30T01:51:51.833 INFO:teuthology.orchestra.run.smithi110.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-09-30T01:51:51.833 INFO:teuthology.orchestra.run.smithi110.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-09-30T01:51:51.833 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:51:51.833 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2023-09-30T01:51:51.952 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-mgr-rook 2023-09-30T01:51:52.806 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:02:02 ago on Sat 30 Sep 2023 01:49:50 AM UTC. 2023-09-30T01:51:52.895 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-09-30T01:51:53.118 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e 8/8 2023-09-30T01:51:53.477 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e 8/8 2023-09-30T01:51:53.478 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e 1/8 2023-09-30T01:51:53.478 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-09-30T01:51:53.478 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-09-30T01:51:53.478 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-09-30T01:51:53.478 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-09-30T01:51:53.478 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-09-30T01:51:53.478 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 7/8 2023-09-30T01:51:54.211 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 8/8 2023-09-30T01:51:54.211 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:51:54.211 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2023-09-30T01:51:54.212 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T01:51:54.212 INFO:teuthology.orchestra.run.smithi059.stdout: libgfortran-8.5.0-20.el8.x86_64 2023-09-30T01:51:54.212 INFO:teuthology.orchestra.run.smithi059.stdout: libquadmath-8.5.0-20.el8.x86_64 2023-09-30T01:51:54.212 INFO:teuthology.orchestra.run.smithi059.stdout: openblas-0.3.15-4.el8.x86_64 2023-09-30T01:51:54.212 INFO:teuthology.orchestra.run.smithi059.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-09-30T01:51:54.212 INFO:teuthology.orchestra.run.smithi059.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-09-30T01:51:54.212 INFO:teuthology.orchestra.run.smithi059.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-09-30T01:51:54.212 INFO:teuthology.orchestra.run.smithi059.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-09-30T01:51:54.212 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:51:54.212 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-09-30T01:51:54.341 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-mgr-rook 2023-09-30T01:51:54.392 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2023-09-30T01:51:54.394 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T01:51:54.394 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2023-09-30T01:51:54.394 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T01:51:54.394 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2023-09-30T01:51:54.394 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-rook noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 47 k 2023-09-30T01:51:54.394 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2023-09-30T01:51:54.394 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-09-30T01:51:54.394 INFO:teuthology.orchestra.run.smithi110.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-09-30T01:51:54.394 INFO:teuthology.orchestra.run.smithi110.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-09-30T01:51:54.395 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-09-30T01:51:54.395 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-09-30T01:51:54.395 INFO:teuthology.orchestra.run.smithi110.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-09-30T01:51:54.395 INFO:teuthology.orchestra.run.smithi110.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-09-30T01:51:54.395 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-09-30T01:51:54.395 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-09-30T01:51:54.395 INFO:teuthology.orchestra.run.smithi110.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-09-30T01:51:54.395 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-09-30T01:51:54.395 INFO:teuthology.orchestra.run.smithi110.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-09-30T01:51:54.395 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:51:54.396 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2023-09-30T01:51:54.396 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T01:51:54.396 INFO:teuthology.orchestra.run.smithi110.stdout:Install 13 Packages 2023-09-30T01:51:54.396 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:51:54.397 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 1.9 M 2023-09-30T01:51:54.397 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 25 M 2023-09-30T01:51:54.397 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2023-09-30T01:51:54.766 INFO:teuthology.orchestra.run.smithi110.stdout:(1/13): python3-jsonpointer-1.10-11.el8.noarch. 159 kB/s | 20 kB 00:00 2023-09-30T01:51:54.792 INFO:teuthology.orchestra.run.smithi110.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 176 kB/s | 27 kB 00:00 2023-09-30T01:51:54.817 INFO:teuthology.orchestra.run.smithi110.stdout:(3/13): ceph-mgr-rook-17.2.6-1165.g93dce5e1.el8 268 kB/s | 47 kB 00:00 2023-09-30T01:51:54.884 INFO:teuthology.orchestra.run.smithi110.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.1 MB/s | 126 kB 00:00 2023-09-30T01:51:54.901 INFO:teuthology.orchestra.run.smithi110.stdout:(5/13): python3-pyasn1-modules-0.3.7-6.el8.noar 1.0 MB/s | 110 kB 00:00 2023-09-30T01:51:55.026 INFO:teuthology.orchestra.run.smithi110.stdout:(6/13): python3-cachetools-3.1.1-4.el8.noarch.r 263 kB/s | 33 kB 00:00 2023-09-30T01:51:55.052 INFO:teuthology.orchestra.run.smithi110.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 255 kB/s | 43 kB 00:00 2023-09-30T01:51:55.077 INFO:teuthology.orchestra.run.smithi110.stdout:(8/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 597 kB/s | 155 kB 00:00 2023-09-30T01:51:55.102 INFO:teuthology.orchestra.run.smithi110.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 216 kB/s | 16 kB 00:00 2023-09-30T01:51:55.222 INFO:teuthology.orchestra.run.smithi110.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 486 kB/s | 81 kB 00:00 2023-09-30T01:51:55.236 INFO:teuthology.orchestra.run.smithi110.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 498 kB/s | 64 kB 00:00 2023-09-30T01:51:55.261 INFO:teuthology.orchestra.run.smithi110.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 1.4 MB/s | 61 kB 00:00 2023-09-30T01:51:55.295 INFO:teuthology.orchestra.run.smithi110.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 6.2 MB/s | 1.2 MB 00:00 2023-09-30T01:51:55.297 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:51:55.297 INFO:teuthology.orchestra.run.smithi110.stdout:Total 2.1 MB/s | 1.9 MB 00:00 2023-09-30T01:51:55.324 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:03 ago on Sat 30 Sep 2023 01:49:52 AM UTC. 2023-09-30T01:51:55.353 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2023-09-30T01:51:55.368 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2023-09-30T01:51:55.368 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2023-09-30T01:51:55.454 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2023-09-30T01:51:55.457 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2023-09-30T01:51:55.934 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2023-09-30T01:51:56.200 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-09-30T01:51:56.390 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-09-30T01:51:56.625 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2023-09-30T01:51:56.854 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-09-30T01:51:57.069 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-09-30T01:51:57.091 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-09-30T01:51:57.093 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T01:51:57.093 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2023-09-30T01:51:57.093 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T01:51:57.093 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2023-09-30T01:51:57.093 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-rook noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 47 k 2023-09-30T01:51:57.093 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2023-09-30T01:51:57.093 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-09-30T01:51:57.093 INFO:teuthology.orchestra.run.smithi059.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-09-30T01:51:57.094 INFO:teuthology.orchestra.run.smithi059.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-09-30T01:51:57.094 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-09-30T01:51:57.094 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-09-30T01:51:57.094 INFO:teuthology.orchestra.run.smithi059.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-09-30T01:51:57.094 INFO:teuthology.orchestra.run.smithi059.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-09-30T01:51:57.094 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-09-30T01:51:57.094 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-09-30T01:51:57.094 INFO:teuthology.orchestra.run.smithi059.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-09-30T01:51:57.094 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-09-30T01:51:57.094 INFO:teuthology.orchestra.run.smithi059.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-09-30T01:51:57.094 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:51:57.095 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2023-09-30T01:51:57.095 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T01:51:57.095 INFO:teuthology.orchestra.run.smithi059.stdout:Install 13 Packages 2023-09-30T01:51:57.095 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:51:57.095 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 1.9 M 2023-09-30T01:51:57.095 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 25 M 2023-09-30T01:51:57.096 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2023-09-30T01:51:57.308 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-09-30T01:51:57.600 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-09-30T01:51:57.792 INFO:teuthology.orchestra.run.smithi059.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 199 kB/s | 27 kB 00:00 2023-09-30T01:51:57.817 INFO:teuthology.orchestra.run.smithi059.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 126 kB/s | 20 kB 00:00 2023-09-30T01:51:57.843 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-09-30T01:51:57.917 INFO:teuthology.orchestra.run.smithi059.stdout:(3/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.0 MB/s | 126 kB 00:00 2023-09-30T01:51:57.943 INFO:teuthology.orchestra.run.smithi059.stdout:(4/13): ceph-mgr-rook-17.2.6-1165.g93dce5e1.el8 166 kB/s | 47 kB 00:00 2023-09-30T01:51:57.968 INFO:teuthology.orchestra.run.smithi059.stdout:(5/13): python3-pyasn1-modules-0.3.7-6.el8.noar 729 kB/s | 110 kB 00:00 2023-09-30T01:51:58.093 INFO:teuthology.orchestra.run.smithi059.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 284 kB/s | 43 kB 00:00 2023-09-30T01:51:58.144 INFO:teuthology.orchestra.run.smithi059.stdout:(7/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 685 kB/s | 155 kB 00:00 2023-09-30T01:51:58.236 INFO:teuthology.orchestra.run.smithi059.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 123 kB/s | 33 kB 00:00 2023-09-30T01:51:58.369 INFO:teuthology.orchestra.run.smithi059.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 59 kB/s | 16 kB 00:00 2023-09-30T01:51:58.478 INFO:teuthology.orchestra.run.smithi059.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 247 kB/s | 81 kB 00:00 2023-09-30T01:51:58.503 INFO:teuthology.orchestra.run.smithi059.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 480 kB/s | 64 kB 00:00 2023-09-30T01:51:58.578 INFO:teuthology.orchestra.run.smithi059.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 612 kB/s | 61 kB 00:00 2023-09-30T01:51:58.612 INFO:teuthology.orchestra.run.smithi059.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 3.1 MB/s | 1.2 MB 00:00 2023-09-30T01:51:58.615 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:51:58.615 INFO:teuthology.orchestra.run.smithi059.stdout:Total 1.3 MB/s | 1.9 MB 00:01 2023-09-30T01:51:58.680 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2023-09-30T01:51:58.697 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2023-09-30T01:51:58.697 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2023-09-30T01:51:58.795 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2023-09-30T01:51:58.799 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2023-09-30T01:51:58.879 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-09-30T01:51:59.162 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-09-30T01:51:59.299 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2023-09-30T01:51:59.387 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-09-30T01:51:59.562 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-09-30T01:51:59.605 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-09-30T01:51:59.773 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-09-30T01:51:59.789 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 13/13 2023-09-30T01:52:00.007 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2023-09-30T01:52:00.080 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 13/13 2023-09-30T01:52:00.081 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 1/13 2023-09-30T01:52:00.081 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-09-30T01:52:00.081 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-09-30T01:52:00.081 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-09-30T01:52:00.081 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-09-30T01:52:00.081 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-09-30T01:52:00.081 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-09-30T01:52:00.081 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-09-30T01:52:00.081 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-09-30T01:52:00.081 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-09-30T01:52:00.082 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-09-30T01:52:00.082 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2023-09-30T01:52:00.267 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-09-30T01:52:00.484 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-09-30T01:52:00.770 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-09-30T01:52:00.776 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-09-30T01:52:00.776 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:52:00.777 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2023-09-30T01:52:00.777 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T01:52:00.777 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-09-30T01:52:00.777 INFO:teuthology.orchestra.run.smithi110.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-09-30T01:52:00.777 INFO:teuthology.orchestra.run.smithi110.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-09-30T01:52:00.777 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-09-30T01:52:00.777 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-09-30T01:52:00.777 INFO:teuthology.orchestra.run.smithi110.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-09-30T01:52:00.777 INFO:teuthology.orchestra.run.smithi110.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-09-30T01:52:00.777 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-09-30T01:52:00.777 INFO:teuthology.orchestra.run.smithi110.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-09-30T01:52:00.778 INFO:teuthology.orchestra.run.smithi110.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-09-30T01:52:00.778 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rsa-4.9-2.el8.noarch 2023-09-30T01:52:00.778 INFO:teuthology.orchestra.run.smithi110.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-09-30T01:52:00.778 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:52:00.778 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2023-09-30T01:52:00.900 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-mgr-cephadm 2023-09-30T01:52:01.099 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-09-30T01:52:01.376 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-09-30T01:52:01.760 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:02:11 ago on Sat 30 Sep 2023 01:49:50 AM UTC. 2023-09-30T01:52:02.537 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-09-30T01:52:02.825 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-09-30T01:52:03.067 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-09-30T01:52:03.288 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-09-30T01:52:03.343 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2023-09-30T01:52:03.343 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T01:52:03.343 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2023-09-30T01:52:03.344 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T01:52:03.344 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2023-09-30T01:52:03.344 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-cephadm noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 121 k 2023-09-30T01:52:03.344 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2023-09-30T01:52:03.344 INFO:teuthology.orchestra.run.smithi110.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-09-30T01:52:03.344 INFO:teuthology.orchestra.run.smithi110.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-09-30T01:52:03.344 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-09-30T01:52:03.344 INFO:teuthology.orchestra.run.smithi110.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-09-30T01:52:03.344 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:52:03.344 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2023-09-30T01:52:03.344 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T01:52:03.344 INFO:teuthology.orchestra.run.smithi110.stdout:Install 5 Packages 2023-09-30T01:52:03.345 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:52:03.345 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 6.0 M 2023-09-30T01:52:03.345 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 26 M 2023-09-30T01:52:03.345 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2023-09-30T01:52:03.599 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 13/13 2023-09-30T01:52:03.946 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 13/13 2023-09-30T01:52:03.946 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 1/13 2023-09-30T01:52:03.946 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-09-30T01:52:03.946 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-09-30T01:52:03.946 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-09-30T01:52:03.946 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-09-30T01:52:03.947 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-09-30T01:52:03.947 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-09-30T01:52:03.947 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-09-30T01:52:03.947 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-09-30T01:52:03.947 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-09-30T01:52:03.947 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-09-30T01:52:03.947 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2023-09-30T01:52:04.195 INFO:teuthology.orchestra.run.smithi110.stdout:(1/5): ceph-mgr-cephadm-17.2.6-1165.g93dce5e1.e 614 kB/s | 121 kB 00:00 2023-09-30T01:52:04.337 INFO:teuthology.orchestra.run.smithi110.stdout:(2/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 1.5 MB/s | 538 kB 00:00 2023-09-30T01:52:04.679 INFO:teuthology.orchestra.run.smithi110.stdout:(3/5): python3-natsort-7.1.1-2.el8.noarch.rpm 176 kB/s | 60 kB 00:00 2023-09-30T01:52:04.737 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-09-30T01:52:04.737 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:52:04.737 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2023-09-30T01:52:04.738 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T01:52:04.738 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-09-30T01:52:04.738 INFO:teuthology.orchestra.run.smithi059.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-09-30T01:52:04.738 INFO:teuthology.orchestra.run.smithi059.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-09-30T01:52:04.738 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-09-30T01:52:04.738 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-09-30T01:52:04.738 INFO:teuthology.orchestra.run.smithi059.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-09-30T01:52:04.738 INFO:teuthology.orchestra.run.smithi059.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-09-30T01:52:04.738 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-09-30T01:52:04.738 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-09-30T01:52:04.738 INFO:teuthology.orchestra.run.smithi059.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-09-30T01:52:04.739 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rsa-4.9-2.el8.noarch 2023-09-30T01:52:04.739 INFO:teuthology.orchestra.run.smithi059.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-09-30T01:52:04.739 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:52:04.739 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-09-30T01:52:04.804 INFO:teuthology.orchestra.run.smithi110.stdout:(4/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 781 kB/s | 476 kB 00:00 2023-09-30T01:52:04.864 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-mgr-cephadm 2023-09-30T01:52:05.505 INFO:teuthology.orchestra.run.smithi110.stdout:(5/5): python3-babel-2.5.1-7.el8.noarch.rpm 3.2 MB/s | 4.8 MB 00:01 2023-09-30T01:52:05.505 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:52:05.505 INFO:teuthology.orchestra.run.smithi110.stdout:Total 2.8 MB/s | 6.0 MB 00:02 2023-09-30T01:52:05.560 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2023-09-30T01:52:05.572 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2023-09-30T01:52:05.572 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2023-09-30T01:52:05.673 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2023-09-30T01:52:05.678 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2023-09-30T01:52:05.856 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:13 ago on Sat 30 Sep 2023 01:49:52 AM UTC. 2023-09-30T01:52:05.984 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2023-09-30T01:52:06.296 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 1/5 2023-09-30T01:52:07.275 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2023-09-30T01:52:07.606 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-09-30T01:52:07.608 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T01:52:07.608 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2023-09-30T01:52:07.608 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T01:52:07.608 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2023-09-30T01:52:07.608 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-cephadm noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 121 k 2023-09-30T01:52:07.608 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2023-09-30T01:52:07.608 INFO:teuthology.orchestra.run.smithi059.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-09-30T01:52:07.608 INFO:teuthology.orchestra.run.smithi059.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-09-30T01:52:07.608 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-09-30T01:52:07.608 INFO:teuthology.orchestra.run.smithi059.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-09-30T01:52:07.609 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:52:07.609 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2023-09-30T01:52:07.609 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T01:52:07.609 INFO:teuthology.orchestra.run.smithi059.stdout:Install 5 Packages 2023-09-30T01:52:07.609 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:52:07.609 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 6.0 M 2023-09-30T01:52:07.610 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 26 M 2023-09-30T01:52:07.610 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2023-09-30T01:52:07.618 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 3/5 2023-09-30T01:52:07.873 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 4/5 2023-09-30T01:52:08.054 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 5/5 2023-09-30T01:52:08.343 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 5/5 2023-09-30T01:52:08.343 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/5 2023-09-30T01:52:08.343 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/5 2023-09-30T01:52:08.343 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/5 2023-09-30T01:52:08.343 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2023-09-30T01:52:08.407 INFO:teuthology.orchestra.run.smithi059.stdout:(1/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 1.8 MB/s | 538 kB 00:00 2023-09-30T01:52:08.435 INFO:teuthology.orchestra.run.smithi059.stdout:(2/5): ceph-mgr-cephadm-17.2.6-1165.g93dce5e1.e 376 kB/s | 121 kB 00:00 2023-09-30T01:52:08.703 INFO:teuthology.orchestra.run.smithi059.stdout:(3/5): python3-natsort-7.1.1-2.el8.noarch.rpm 226 kB/s | 60 kB 00:00 2023-09-30T01:52:08.853 INFO:teuthology.orchestra.run.smithi059.stdout:(4/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 1.0 MB/s | 476 kB 00:00 2023-09-30T01:52:08.975 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 5/5 2023-09-30T01:52:08.975 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:52:08.975 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2023-09-30T01:52:08.976 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T01:52:08.976 INFO:teuthology.orchestra.run.smithi110.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-09-30T01:52:08.976 INFO:teuthology.orchestra.run.smithi110.stdout: python3-babel-2.5.1-7.el8.noarch 2023-09-30T01:52:08.976 INFO:teuthology.orchestra.run.smithi110.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-09-30T01:52:08.976 INFO:teuthology.orchestra.run.smithi110.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-09-30T01:52:08.976 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:52:08.976 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2023-09-30T01:52:09.036 INFO:teuthology.orchestra.run.smithi059.stdout:(5/5): python3-babel-2.5.1-7.el8.noarch.rpm 5.2 MB/s | 4.8 MB 00:00 2023-09-30T01:52:09.039 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:52:09.039 INFO:teuthology.orchestra.run.smithi059.stdout:Total 4.2 MB/s | 6.0 MB 00:01 2023-09-30T01:52:09.083 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-fuse 2023-09-30T01:52:09.097 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2023-09-30T01:52:09.110 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2023-09-30T01:52:09.110 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2023-09-30T01:52:09.219 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2023-09-30T01:52:09.223 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2023-09-30T01:52:09.580 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2023-09-30T01:52:09.938 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:02:19 ago on Sat 30 Sep 2023 01:49:50 AM UTC. 2023-09-30T01:52:09.940 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 1/5 2023-09-30T01:52:10.979 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2023-09-30T01:52:11.533 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2023-09-30T01:52:11.534 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T01:52:11.534 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2023-09-30T01:52:11.534 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T01:52:11.534 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2023-09-30T01:52:11.534 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-fuse x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 874 k 2023-09-30T01:52:11.534 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2023-09-30T01:52:11.534 INFO:teuthology.orchestra.run.smithi110.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-09-30T01:52:11.534 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:52:11.534 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2023-09-30T01:52:11.535 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T01:52:11.535 INFO:teuthology.orchestra.run.smithi110.stdout:Install 2 Packages 2023-09-30T01:52:11.535 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:52:11.535 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 957 k 2023-09-30T01:52:11.535 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 3.0 M 2023-09-30T01:52:11.535 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2023-09-30T01:52:11.664 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 3/5 2023-09-30T01:52:11.958 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 4/5 2023-09-30T01:52:12.113 INFO:teuthology.orchestra.run.smithi110.stdout:(1/2): fuse-2.9.7-17.el8.x86_64.rpm 439 kB/s | 83 kB 00:00 2023-09-30T01:52:12.149 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 5/5 2023-09-30T01:52:12.179 INFO:teuthology.orchestra.run.smithi110.stdout:(2/2): ceph-fuse-17.2.6-1165.g93dce5e1.el8.x86_ 3.3 MB/s | 874 kB 00:00 2023-09-30T01:52:12.180 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:52:12.180 INFO:teuthology.orchestra.run.smithi110.stdout:Total 1.4 MB/s | 957 kB 00:00 2023-09-30T01:52:12.191 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2023-09-30T01:52:12.213 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2023-09-30T01:52:12.213 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2023-09-30T01:52:12.302 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2023-09-30T01:52:12.305 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2023-09-30T01:52:12.474 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 5/5 2023-09-30T01:52:12.474 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/5 2023-09-30T01:52:12.474 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/5 2023-09-30T01:52:12.474 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/5 2023-09-30T01:52:12.474 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2023-09-30T01:52:12.475 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2023-09-30T01:52:12.777 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-09-30T01:52:13.020 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/2 2023-09-30T01:52:13.193 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 5/5 2023-09-30T01:52:13.193 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:52:13.193 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2023-09-30T01:52:13.194 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T01:52:13.194 INFO:teuthology.orchestra.run.smithi059.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-09-30T01:52:13.194 INFO:teuthology.orchestra.run.smithi059.stdout: python3-babel-2.5.1-7.el8.noarch 2023-09-30T01:52:13.194 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-09-30T01:52:13.194 INFO:teuthology.orchestra.run.smithi059.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-09-30T01:52:13.194 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:52:13.194 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-09-30T01:52:13.345 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-fuse 2023-09-30T01:52:13.413 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/2 2023-09-30T01:52:13.413 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/2 2023-09-30T01:52:14.051 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-09-30T01:52:14.051 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:52:14.052 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2023-09-30T01:52:14.052 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-09-30T01:52:14.052 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:52:14.052 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2023-09-30T01:52:14.263 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-volume 2023-09-30T01:52:14.342 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:22 ago on Sat 30 Sep 2023 01:49:52 AM UTC. 2023-09-30T01:52:15.127 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:02:25 ago on Sat 30 Sep 2023 01:49:50 AM UTC. 2023-09-30T01:52:16.098 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-09-30T01:52:16.098 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T01:52:16.099 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2023-09-30T01:52:16.099 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T01:52:16.099 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2023-09-30T01:52:16.099 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-fuse x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 874 k 2023-09-30T01:52:16.099 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2023-09-30T01:52:16.099 INFO:teuthology.orchestra.run.smithi059.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-09-30T01:52:16.099 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:52:16.099 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2023-09-30T01:52:16.099 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T01:52:16.100 INFO:teuthology.orchestra.run.smithi059.stdout:Install 2 Packages 2023-09-30T01:52:16.100 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:52:16.100 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 957 k 2023-09-30T01:52:16.100 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 3.0 M 2023-09-30T01:52:16.101 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2023-09-30T01:52:16.494 INFO:teuthology.orchestra.run.smithi059.stdout:(1/2): fuse-2.9.7-17.el8.x86_64.rpm 412 kB/s | 83 kB 00:00 2023-09-30T01:52:16.683 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2023-09-30T01:52:16.683 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T01:52:16.683 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repository Size 2023-09-30T01:52:16.684 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T01:52:16.684 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2023-09-30T01:52:16.684 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-volume noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 267 k 2023-09-30T01:52:16.684 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:52:16.684 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2023-09-30T01:52:16.684 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T01:52:16.684 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2023-09-30T01:52:16.684 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:52:16.685 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 267 k 2023-09-30T01:52:16.685 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 1.3 M 2023-09-30T01:52:16.685 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2023-09-30T01:52:16.694 INFO:teuthology.orchestra.run.smithi059.stdout:(2/2): ceph-fuse-17.2.6-1165.g93dce5e1.el8.x86_ 2.1 MB/s | 874 kB 00:00 2023-09-30T01:52:16.695 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:52:16.695 INFO:teuthology.orchestra.run.smithi059.stdout:Total 1.6 MB/s | 957 kB 00:00 2023-09-30T01:52:16.708 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2023-09-30T01:52:16.732 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2023-09-30T01:52:16.732 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2023-09-30T01:52:16.833 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2023-09-30T01:52:16.836 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2023-09-30T01:52:17.009 INFO:teuthology.orchestra.run.smithi110.stdout:ceph-volume-17.2.6-1165.g93dce5e1.el8.noarch.rp 824 kB/s | 267 kB 00:00 2023-09-30T01:52:17.009 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:52:17.010 INFO:teuthology.orchestra.run.smithi110.stdout:Total 820 kB/s | 267 kB 00:00 2023-09-30T01:52:17.010 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2023-09-30T01:52:17.018 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2023-09-30T01:52:17.020 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2023-09-30T01:52:17.020 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2023-09-30T01:52:17.056 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2023-09-30T01:52:17.059 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2023-09-30T01:52:17.260 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2023-09-30T01:52:17.333 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-09-30T01:52:17.487 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T01:52:17.533 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/2 2023-09-30T01:52:17.879 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T01:52:18.010 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/2 2023-09-30T01:52:18.010 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/2 2023-09-30T01:52:18.489 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T01:52:18.489 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:52:18.489 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2023-09-30T01:52:18.489 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T01:52:18.489 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:52:18.489 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2023-09-30T01:52:18.682 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install librados-devel 2023-09-30T01:52:18.746 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-09-30T01:52:18.746 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:52:18.746 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2023-09-30T01:52:18.746 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-09-30T01:52:18.746 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:52:18.746 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-09-30T01:52:18.972 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-volume 2023-09-30T01:52:19.542 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:02:29 ago on Sat 30 Sep 2023 01:49:50 AM UTC. 2023-09-30T01:52:19.969 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:27 ago on Sat 30 Sep 2023 01:49:52 AM UTC. 2023-09-30T01:52:21.114 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2023-09-30T01:52:21.114 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T01:52:21.114 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2023-09-30T01:52:21.114 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T01:52:21.115 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2023-09-30T01:52:21.115 INFO:teuthology.orchestra.run.smithi110.stdout: librados-devel x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 130 k 2023-09-30T01:52:21.115 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:52:21.115 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2023-09-30T01:52:21.115 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T01:52:21.115 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2023-09-30T01:52:21.115 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:52:21.116 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 130 k 2023-09-30T01:52:21.116 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 464 k 2023-09-30T01:52:21.116 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2023-09-30T01:52:21.717 INFO:teuthology.orchestra.run.smithi110.stdout:librados-devel-17.2.6-1165.g93dce5e1.el8.x86_64 216 kB/s | 130 kB 00:00 2023-09-30T01:52:21.718 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:52:21.719 INFO:teuthology.orchestra.run.smithi110.stdout:Total 215 kB/s | 130 kB 00:00 2023-09-30T01:52:21.719 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2023-09-30T01:52:21.725 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2023-09-30T01:52:21.726 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2023-09-30T01:52:21.727 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-09-30T01:52:21.728 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T01:52:21.728 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2023-09-30T01:52:21.728 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T01:52:21.728 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2023-09-30T01:52:21.728 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-volume noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 267 k 2023-09-30T01:52:21.728 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:52:21.729 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2023-09-30T01:52:21.729 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T01:52:21.729 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2023-09-30T01:52:21.729 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:52:21.729 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 267 k 2023-09-30T01:52:21.730 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 1.3 M 2023-09-30T01:52:21.730 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2023-09-30T01:52:21.778 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2023-09-30T01:52:21.781 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2023-09-30T01:52:21.890 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2023-09-30T01:52:22.168 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:52:22.464 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:52:22.701 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-volume-17.2.6-1165.g93dce5e1.el8.noarch.rp 275 kB/s | 267 kB 00:00 2023-09-30T01:52:22.702 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:52:22.702 INFO:teuthology.orchestra.run.smithi059.stdout:Total 274 kB/s | 267 kB 00:00 2023-09-30T01:52:22.702 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2023-09-30T01:52:22.716 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2023-09-30T01:52:22.716 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2023-09-30T01:52:22.820 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2023-09-30T01:52:22.823 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2023-09-30T01:52:23.078 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:52:23.078 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:52:23.078 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2023-09-30T01:52:23.078 INFO:teuthology.orchestra.run.smithi110.stdout: librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:52:23.078 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:52:23.079 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2023-09-30T01:52:23.325 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install libcephfs2 2023-09-30T01:52:23.406 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2023-09-30T01:52:23.656 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T01:52:24.157 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T01:52:24.186 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:02:34 ago on Sat 30 Sep 2023 01:49:50 AM UTC. 2023-09-30T01:52:24.864 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T01:52:24.864 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:52:24.864 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2023-09-30T01:52:24.864 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T01:52:24.864 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:52:24.865 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-09-30T01:52:25.075 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install librados-devel 2023-09-30T01:52:25.587 INFO:teuthology.orchestra.run.smithi110.stdout:Package libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T01:52:25.774 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2023-09-30T01:52:25.775 INFO:teuthology.orchestra.run.smithi110.stdout:Nothing to do. 2023-09-30T01:52:25.775 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2023-09-30T01:52:25.875 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install libcephfs-devel 2023-09-30T01:52:26.067 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:34 ago on Sat 30 Sep 2023 01:49:52 AM UTC. 2023-09-30T01:52:26.729 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:02:36 ago on Sat 30 Sep 2023 01:49:50 AM UTC. 2023-09-30T01:52:27.815 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-09-30T01:52:27.815 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T01:52:27.815 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repo Size 2023-09-30T01:52:27.815 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T01:52:27.816 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2023-09-30T01:52:27.816 INFO:teuthology.orchestra.run.smithi059.stdout: librados-devel x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 130 k 2023-09-30T01:52:27.816 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:52:27.816 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2023-09-30T01:52:27.816 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T01:52:27.816 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2023-09-30T01:52:27.816 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:52:27.817 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 130 k 2023-09-30T01:52:27.817 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 464 k 2023-09-30T01:52:27.817 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2023-09-30T01:52:28.196 INFO:teuthology.orchestra.run.smithi059.stdout:librados-devel-17.2.6-1165.g93dce5e1.el8.x86_64 343 kB/s | 130 kB 00:00 2023-09-30T01:52:28.196 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:52:28.197 INFO:teuthology.orchestra.run.smithi059.stdout:Total 341 kB/s | 130 kB 00:00 2023-09-30T01:52:28.197 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2023-09-30T01:52:28.204 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2023-09-30T01:52:28.204 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2023-09-30T01:52:28.265 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2023-09-30T01:52:28.268 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2023-09-30T01:52:28.312 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2023-09-30T01:52:28.312 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T01:52:28.312 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2023-09-30T01:52:28.312 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T01:52:28.312 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2023-09-30T01:52:28.313 INFO:teuthology.orchestra.run.smithi110.stdout: libcephfs-devel x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 25 k 2023-09-30T01:52:28.313 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:52:28.313 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2023-09-30T01:52:28.313 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T01:52:28.313 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2023-09-30T01:52:28.313 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:52:28.313 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 25 k 2023-09-30T01:52:28.314 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 109 k 2023-09-30T01:52:28.314 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2023-09-30T01:52:28.385 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2023-09-30T01:52:28.630 INFO:teuthology.orchestra.run.smithi110.stdout:libcephfs-devel-17.2.6-1165.g93dce5e1.el8.x86_6 78 kB/s | 25 kB 00:00 2023-09-30T01:52:28.630 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:52:28.631 INFO:teuthology.orchestra.run.smithi110.stdout:Total 78 kB/s | 25 kB 00:00 2023-09-30T01:52:28.631 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2023-09-30T01:52:28.634 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2023-09-30T01:52:28.634 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2023-09-30T01:52:28.636 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2023-09-30T01:52:28.639 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2023-09-30T01:52:28.689 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2023-09-30T01:52:28.780 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:52:28.900 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:52:29.096 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:52:29.166 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:52:29.739 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:52:29.739 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:52:29.739 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2023-09-30T01:52:29.739 INFO:teuthology.orchestra.run.smithi110.stdout: libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:52:29.739 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:52:29.740 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2023-09-30T01:52:29.774 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:52:29.774 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:52:29.774 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2023-09-30T01:52:29.774 INFO:teuthology.orchestra.run.smithi059.stdout: librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:52:29.774 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:52:29.775 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-09-30T01:52:29.848 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install librados2 2023-09-30T01:52:29.992 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install libcephfs2 2023-09-30T01:52:30.708 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:02:40 ago on Sat 30 Sep 2023 01:49:50 AM UTC. 2023-09-30T01:52:30.979 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:38 ago on Sat 30 Sep 2023 01:49:52 AM UTC. 2023-09-30T01:52:32.107 INFO:teuthology.orchestra.run.smithi110.stdout:Package librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T01:52:32.295 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2023-09-30T01:52:32.296 INFO:teuthology.orchestra.run.smithi110.stdout:Nothing to do. 2023-09-30T01:52:32.296 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2023-09-30T01:52:32.390 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install librbd1 2023-09-30T01:52:32.534 INFO:teuthology.orchestra.run.smithi059.stdout:Package libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T01:52:32.746 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-09-30T01:52:32.747 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2023-09-30T01:52:32.747 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-09-30T01:52:32.853 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install libcephfs-devel 2023-09-30T01:52:33.240 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:02:43 ago on Sat 30 Sep 2023 01:49:50 AM UTC. 2023-09-30T01:52:33.854 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:41 ago on Sat 30 Sep 2023 01:49:52 AM UTC. 2023-09-30T01:52:34.632 INFO:teuthology.orchestra.run.smithi110.stdout:Package librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T01:52:34.822 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2023-09-30T01:52:34.823 INFO:teuthology.orchestra.run.smithi110.stdout:Nothing to do. 2023-09-30T01:52:34.823 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2023-09-30T01:52:34.927 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install python3-rados 2023-09-30T01:52:35.646 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-09-30T01:52:35.647 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T01:52:35.647 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repo Size 2023-09-30T01:52:35.647 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T01:52:35.647 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2023-09-30T01:52:35.647 INFO:teuthology.orchestra.run.smithi059.stdout: libcephfs-devel x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 25 k 2023-09-30T01:52:35.647 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:52:35.647 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2023-09-30T01:52:35.647 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T01:52:35.647 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2023-09-30T01:52:35.647 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:52:35.648 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 25 k 2023-09-30T01:52:35.648 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 109 k 2023-09-30T01:52:35.648 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2023-09-30T01:52:35.774 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:02:45 ago on Sat 30 Sep 2023 01:49:50 AM UTC. 2023-09-30T01:52:35.867 INFO:teuthology.orchestra.run.smithi059.stdout:libcephfs-devel-17.2.6-1165.g93dce5e1.el8.x86_6 113 kB/s | 25 kB 00:00 2023-09-30T01:52:35.868 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:52:35.868 INFO:teuthology.orchestra.run.smithi059.stdout:Total 113 kB/s | 25 kB 00:00 2023-09-30T01:52:35.868 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2023-09-30T01:52:35.873 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2023-09-30T01:52:35.873 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2023-09-30T01:52:35.876 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2023-09-30T01:52:35.879 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2023-09-30T01:52:35.935 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2023-09-30T01:52:36.120 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:52:36.424 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:52:37.075 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:52:37.076 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:52:37.076 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2023-09-30T01:52:37.076 INFO:teuthology.orchestra.run.smithi059.stdout: libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:52:37.076 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:52:37.076 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-09-30T01:52:37.175 INFO:teuthology.orchestra.run.smithi110.stdout:Package python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T01:52:37.208 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install librados2 2023-09-30T01:52:37.362 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2023-09-30T01:52:37.364 INFO:teuthology.orchestra.run.smithi110.stdout:Nothing to do. 2023-09-30T01:52:37.364 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2023-09-30T01:52:37.467 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install python3-rgw 2023-09-30T01:52:38.193 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:46 ago on Sat 30 Sep 2023 01:49:52 AM UTC. 2023-09-30T01:52:38.321 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:02:48 ago on Sat 30 Sep 2023 01:49:50 AM UTC. 2023-09-30T01:52:39.702 INFO:teuthology.orchestra.run.smithi110.stdout:Package python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T01:52:39.753 INFO:teuthology.orchestra.run.smithi059.stdout:Package librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T01:52:39.892 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2023-09-30T01:52:39.894 INFO:teuthology.orchestra.run.smithi110.stdout:Nothing to do. 2023-09-30T01:52:39.894 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2023-09-30T01:52:39.968 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-09-30T01:52:39.969 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2023-09-30T01:52:39.969 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-09-30T01:52:39.994 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install python3-cephfs 2023-09-30T01:52:40.070 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install librbd1 2023-09-30T01:52:40.851 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:02:50 ago on Sat 30 Sep 2023 01:49:50 AM UTC. 2023-09-30T01:52:41.051 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:49 ago on Sat 30 Sep 2023 01:49:52 AM UTC. 2023-09-30T01:52:42.252 INFO:teuthology.orchestra.run.smithi110.stdout:Package python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T01:52:42.439 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2023-09-30T01:52:42.440 INFO:teuthology.orchestra.run.smithi110.stdout:Nothing to do. 2023-09-30T01:52:42.440 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2023-09-30T01:52:42.543 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install python3-rbd 2023-09-30T01:52:42.610 INFO:teuthology.orchestra.run.smithi059.stdout:Package librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T01:52:42.818 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-09-30T01:52:42.819 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2023-09-30T01:52:42.819 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-09-30T01:52:42.921 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install python3-rados 2023-09-30T01:52:43.394 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:02:53 ago on Sat 30 Sep 2023 01:49:50 AM UTC. 2023-09-30T01:52:43.909 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:51 ago on Sat 30 Sep 2023 01:49:52 AM UTC. 2023-09-30T01:52:44.769 INFO:teuthology.orchestra.run.smithi110.stdout:Package python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T01:52:44.959 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2023-09-30T01:52:44.960 INFO:teuthology.orchestra.run.smithi110.stdout:Nothing to do. 2023-09-30T01:52:44.960 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2023-09-30T01:52:45.064 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install rbd-fuse 2023-09-30T01:52:45.457 INFO:teuthology.orchestra.run.smithi059.stdout:Package python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T01:52:45.667 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-09-30T01:52:45.668 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2023-09-30T01:52:45.668 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-09-30T01:52:45.775 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install python3-rgw 2023-09-30T01:52:45.911 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:02:55 ago on Sat 30 Sep 2023 01:49:50 AM UTC. 2023-09-30T01:52:46.755 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:54 ago on Sat 30 Sep 2023 01:49:52 AM UTC. 2023-09-30T01:52:47.492 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2023-09-30T01:52:47.492 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T01:52:47.493 INFO:teuthology.orchestra.run.smithi110.stdout: Package Architecture Version Repository Size 2023-09-30T01:52:47.493 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T01:52:47.493 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2023-09-30T01:52:47.493 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-fuse x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 90 k 2023-09-30T01:52:47.493 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:52:47.493 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2023-09-30T01:52:47.493 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T01:52:47.493 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2023-09-30T01:52:47.493 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:52:47.494 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 90 k 2023-09-30T01:52:47.494 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 264 k 2023-09-30T01:52:47.494 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2023-09-30T01:52:47.762 INFO:teuthology.orchestra.run.smithi110.stdout:rbd-fuse-17.2.6-1165.g93dce5e1.el8.x86_64.rpm 337 kB/s | 90 kB 00:00 2023-09-30T01:52:47.762 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:52:47.763 INFO:teuthology.orchestra.run.smithi110.stdout:Total 335 kB/s | 90 kB 00:00 2023-09-30T01:52:47.763 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2023-09-30T01:52:47.773 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2023-09-30T01:52:47.773 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2023-09-30T01:52:47.822 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2023-09-30T01:52:47.825 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2023-09-30T01:52:47.929 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2023-09-30T01:52:48.115 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:52:48.290 INFO:teuthology.orchestra.run.smithi059.stdout:Package python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T01:52:48.408 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:52:48.498 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-09-30T01:52:48.499 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2023-09-30T01:52:48.499 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-09-30T01:52:48.609 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install python3-cephfs 2023-09-30T01:52:49.068 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:52:49.068 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:52:49.069 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2023-09-30T01:52:49.069 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:52:49.069 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:52:49.069 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2023-09-30T01:52:49.258 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install rbd-mirror 2023-09-30T01:52:49.603 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:57 ago on Sat 30 Sep 2023 01:49:52 AM UTC. 2023-09-30T01:52:50.116 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:03:00 ago on Sat 30 Sep 2023 01:49:50 AM UTC. 2023-09-30T01:52:51.150 INFO:teuthology.orchestra.run.smithi059.stdout:Package python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T01:52:51.358 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-09-30T01:52:51.360 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2023-09-30T01:52:51.360 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-09-30T01:52:51.460 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install python3-rbd 2023-09-30T01:52:51.712 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2023-09-30T01:52:51.713 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T01:52:51.713 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2023-09-30T01:52:51.713 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T01:52:51.713 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2023-09-30T01:52:51.713 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-mirror x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.5 M 2023-09-30T01:52:51.713 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:52:51.713 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2023-09-30T01:52:51.713 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T01:52:51.713 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2023-09-30T01:52:51.714 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:52:51.714 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 3.5 M 2023-09-30T01:52:51.714 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 16 M 2023-09-30T01:52:51.714 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2023-09-30T01:52:52.299 INFO:teuthology.orchestra.run.smithi110.stdout:rbd-mirror-17.2.6-1165.g93dce5e1.el8.x86_64.rpm 5.9 MB/s | 3.5 MB 00:00 2023-09-30T01:52:52.299 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:52:52.300 INFO:teuthology.orchestra.run.smithi110.stdout:Total 5.9 MB/s | 3.5 MB 00:00 2023-09-30T01:52:52.300 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2023-09-30T01:52:52.311 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2023-09-30T01:52:52.312 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2023-09-30T01:52:52.374 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2023-09-30T01:52:52.377 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2023-09-30T01:52:52.453 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:03:00 ago on Sat 30 Sep 2023 01:49:52 AM UTC. 2023-09-30T01:52:52.813 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2023-09-30T01:52:53.024 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:52:53.446 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:52:54.004 INFO:teuthology.orchestra.run.smithi059.stdout:Package python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T01:52:54.162 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:52:54.163 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:52:54.163 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2023-09-30T01:52:54.163 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:52:54.163 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:52:54.163 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2023-09-30T01:52:54.215 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-09-30T01:52:54.216 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2023-09-30T01:52:54.216 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-09-30T01:52:54.318 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install rbd-fuse 2023-09-30T01:52:54.339 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install rbd-nbd 2023-09-30T01:52:55.204 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:03:05 ago on Sat 30 Sep 2023 01:49:50 AM UTC. 2023-09-30T01:52:55.303 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:03:03 ago on Sat 30 Sep 2023 01:49:52 AM UTC. 2023-09-30T01:52:56.799 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2023-09-30T01:52:56.800 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T01:52:56.800 INFO:teuthology.orchestra.run.smithi110.stdout: Package Architecture Version Repository Size 2023-09-30T01:52:56.800 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T01:52:56.800 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2023-09-30T01:52:56.800 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-nbd x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 173 k 2023-09-30T01:52:56.800 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:52:56.800 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2023-09-30T01:52:56.800 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T01:52:56.800 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2023-09-30T01:52:56.800 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:52:56.801 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 173 k 2023-09-30T01:52:56.801 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 552 k 2023-09-30T01:52:56.801 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2023-09-30T01:52:57.069 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-09-30T01:52:57.070 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T01:52:57.070 INFO:teuthology.orchestra.run.smithi059.stdout: Package Architecture Version Repository Size 2023-09-30T01:52:57.070 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T01:52:57.070 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2023-09-30T01:52:57.070 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-fuse x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 90 k 2023-09-30T01:52:57.070 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:52:57.070 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2023-09-30T01:52:57.071 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T01:52:57.071 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2023-09-30T01:52:57.071 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:52:57.071 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 90 k 2023-09-30T01:52:57.071 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 264 k 2023-09-30T01:52:57.071 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2023-09-30T01:52:57.095 INFO:teuthology.orchestra.run.smithi110.stdout:rbd-nbd-17.2.6-1165.g93dce5e1.el8.x86_64.rpm 589 kB/s | 173 kB 00:00 2023-09-30T01:52:57.095 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:52:57.095 INFO:teuthology.orchestra.run.smithi110.stdout:Total 587 kB/s | 173 kB 00:00 2023-09-30T01:52:57.096 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2023-09-30T01:52:57.106 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2023-09-30T01:52:57.106 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2023-09-30T01:52:57.155 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2023-09-30T01:52:57.158 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2023-09-30T01:52:57.238 INFO:teuthology.orchestra.run.smithi059.stdout:rbd-fuse-17.2.6-1165.g93dce5e1.el8.x86_64.rpm 542 kB/s | 90 kB 00:00 2023-09-30T01:52:57.239 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:52:57.239 INFO:teuthology.orchestra.run.smithi059.stdout:Total 537 kB/s | 90 kB 00:00 2023-09-30T01:52:57.239 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2023-09-30T01:52:57.251 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2023-09-30T01:52:57.251 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2023-09-30T01:52:57.287 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2023-09-30T01:52:57.308 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2023-09-30T01:52:57.312 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2023-09-30T01:52:57.472 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:52:57.542 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2023-09-30T01:52:57.734 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:52:57.764 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:52:58.044 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:52:58.393 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:52:58.393 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:52:58.393 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2023-09-30T01:52:58.393 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:52:58.393 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:52:58.393 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2023-09-30T01:52:58.554 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-base-debuginfo 2023-09-30T01:52:58.733 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:52:58.733 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:52:58.733 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2023-09-30T01:52:58.734 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:52:58.734 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:52:58.734 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-09-30T01:52:58.927 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install rbd-mirror 2023-09-30T01:52:59.415 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:03:09 ago on Sat 30 Sep 2023 01:49:50 AM UTC. 2023-09-30T01:52:59.914 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:03:07 ago on Sat 30 Sep 2023 01:49:52 AM UTC. 2023-09-30T01:53:00.986 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2023-09-30T01:53:00.987 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T01:53:00.987 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2023-09-30T01:53:00.987 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T01:53:00.987 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2023-09-30T01:53:00.987 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-base-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 86 M 2023-09-30T01:53:00.987 INFO:teuthology.orchestra.run.smithi110.stdout:Installing dependencies: 2023-09-30T01:53:00.987 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 24 M 2023-09-30T01:53:00.987 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:53:00.987 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2023-09-30T01:53:00.988 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T01:53:00.988 INFO:teuthology.orchestra.run.smithi110.stdout:Install 2 Packages 2023-09-30T01:53:00.988 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:53:00.988 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 110 M 2023-09-30T01:53:00.988 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 659 M 2023-09-30T01:53:00.988 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2023-09-30T01:53:01.674 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-09-30T01:53:01.675 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T01:53:01.675 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repo Size 2023-09-30T01:53:01.675 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T01:53:01.676 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2023-09-30T01:53:01.676 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-mirror x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.5 M 2023-09-30T01:53:01.676 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:53:01.676 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2023-09-30T01:53:01.676 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T01:53:01.676 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2023-09-30T01:53:01.676 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:53:01.676 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 3.5 M 2023-09-30T01:53:01.676 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 16 M 2023-09-30T01:53:01.676 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2023-09-30T01:53:02.841 INFO:teuthology.orchestra.run.smithi059.stdout:rbd-mirror-17.2.6-1165.g93dce5e1.el8.x86_64.rpm 3.0 MB/s | 3.5 MB 00:01 2023-09-30T01:53:02.842 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:53:02.842 INFO:teuthology.orchestra.run.smithi059.stdout:Total 3.0 MB/s | 3.5 MB 00:01 2023-09-30T01:53:02.842 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2023-09-30T01:53:02.857 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2023-09-30T01:53:02.857 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2023-09-30T01:53:02.929 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2023-09-30T01:53:02.933 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2023-09-30T01:53:03.414 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2023-09-30T01:53:03.592 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:53:04.073 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:53:04.759 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:53:04.759 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:53:04.759 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2023-09-30T01:53:04.759 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:53:04.760 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:53:04.760 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-09-30T01:53:04.971 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install rbd-nbd 2023-09-30T01:53:05.397 INFO:teuthology.orchestra.run.smithi110.stdout:(1/2): ceph-debuginfo-17.2.6-1165.g93dce5e1.el8 5.4 MB/s | 24 MB 00:04 2023-09-30T01:53:05.962 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:03:13 ago on Sat 30 Sep 2023 01:49:52 AM UTC. 2023-09-30T01:53:07.718 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-09-30T01:53:07.718 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T01:53:07.718 INFO:teuthology.orchestra.run.smithi059.stdout: Package Architecture Version Repository Size 2023-09-30T01:53:07.718 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T01:53:07.719 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2023-09-30T01:53:07.719 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-nbd x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 173 k 2023-09-30T01:53:07.719 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:53:07.719 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2023-09-30T01:53:07.719 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T01:53:07.719 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2023-09-30T01:53:07.719 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:53:07.720 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 173 k 2023-09-30T01:53:07.720 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 552 k 2023-09-30T01:53:07.720 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2023-09-30T01:53:08.091 INFO:teuthology.orchestra.run.smithi059.stdout:rbd-nbd-17.2.6-1165.g93dce5e1.el8.x86_64.rpm 466 kB/s | 173 kB 00:00 2023-09-30T01:53:08.092 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:53:08.092 INFO:teuthology.orchestra.run.smithi059.stdout:Total 464 kB/s | 173 kB 00:00 2023-09-30T01:53:08.093 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2023-09-30T01:53:08.106 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2023-09-30T01:53:08.107 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2023-09-30T01:53:08.166 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2023-09-30T01:53:08.169 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2023-09-30T01:53:08.308 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2023-09-30T01:53:08.633 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:53:08.963 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:53:09.651 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T01:53:09.651 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:53:09.651 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2023-09-30T01:53:09.651 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:53:09.651 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:53:09.651 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-09-30T01:53:09.861 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-base-debuginfo 2023-09-30T01:53:10.861 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:03:18 ago on Sat 30 Sep 2023 01:49:52 AM UTC. 2023-09-30T01:53:11.072 INFO:teuthology.orchestra.run.smithi110.stdout:(2/2): ceph-base-debuginfo-17.2.6-1165.g93dce5e 8.5 MB/s | 86 MB 00:10 2023-09-30T01:53:11.074 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:53:11.074 INFO:teuthology.orchestra.run.smithi110.stdout:Total 11 MB/s | 110 MB 00:10 2023-09-30T01:53:11.075 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2023-09-30T01:53:11.079 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2023-09-30T01:53:11.079 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2023-09-30T01:53:11.555 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2023-09-30T01:53:11.558 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2023-09-30T01:53:12.625 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-09-30T01:53:12.626 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T01:53:12.626 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repo Size 2023-09-30T01:53:12.626 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T01:53:12.626 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2023-09-30T01:53:12.626 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-base-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 86 M 2023-09-30T01:53:12.626 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2023-09-30T01:53:12.626 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 24 M 2023-09-30T01:53:12.626 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:53:12.626 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2023-09-30T01:53:12.627 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T01:53:12.627 INFO:teuthology.orchestra.run.smithi059.stdout:Install 2 Packages 2023-09-30T01:53:12.627 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:53:12.627 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 110 M 2023-09-30T01:53:12.627 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 659 M 2023-09-30T01:53:12.628 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2023-09-30T01:53:14.543 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2023-09-30T01:53:16.928 INFO:teuthology.orchestra.run.smithi059.stdout:(1/2): ceph-base-debuginfo-17.2.6-1165.g93dce5e 20 MB/s | 86 MB 00:04 2023-09-30T01:53:18.010 INFO:teuthology.orchestra.run.smithi059.stdout:(2/2): ceph-debuginfo-17.2.6-1165.g93dce5e1.el8 4.4 MB/s | 24 MB 00:05 2023-09-30T01:53:18.012 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:53:18.012 INFO:teuthology.orchestra.run.smithi059.stdout:Total 20 MB/s | 110 MB 00:05 2023-09-30T01:53:18.013 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2023-09-30T01:53:18.017 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2023-09-30T01:53:18.018 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2023-09-30T01:53:18.549 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2023-09-30T01:53:18.552 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2023-09-30T01:53:21.862 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2023-09-30T01:53:24.140 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/2 2023-09-30T01:53:24.354 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-base-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_ 2/2 2023-09-30T01:53:24.600 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-base-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_ 2/2 2023-09-30T01:53:24.600 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-base-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_ 1/2 2023-09-30T01:53:25.301 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/2 2023-09-30T01:53:25.302 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:53:25.302 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2023-09-30T01:53:25.302 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-base-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:53:25.302 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:53:25.302 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:53:25.302 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2023-09-30T01:53:25.418 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-common-debuginfo 2023-09-30T01:53:26.273 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:03:36 ago on Sat 30 Sep 2023 01:49:50 AM UTC. 2023-09-30T01:53:27.849 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2023-09-30T01:53:27.849 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T01:53:27.849 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2023-09-30T01:53:27.849 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T01:53:27.850 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2023-09-30T01:53:27.850 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-common-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 557 M 2023-09-30T01:53:27.850 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:53:27.850 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2023-09-30T01:53:27.850 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T01:53:27.850 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2023-09-30T01:53:27.850 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:53:27.851 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 557 M 2023-09-30T01:53:27.851 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 2.9 G 2023-09-30T01:53:27.851 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2023-09-30T01:53:32.524 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/2 2023-09-30T01:53:32.714 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-base-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_ 2/2 2023-09-30T01:53:33.052 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-base-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_ 2/2 2023-09-30T01:53:33.052 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-base-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_ 1/2 2023-09-30T01:53:33.847 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/2 2023-09-30T01:53:33.847 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:53:33.847 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2023-09-30T01:53:33.847 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-base-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:53:33.847 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:53:33.847 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:53:33.847 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-09-30T01:53:33.985 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-common-debuginfo 2023-09-30T01:53:34.977 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:03:42 ago on Sat 30 Sep 2023 01:49:52 AM UTC. 2023-09-30T01:53:36.764 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-09-30T01:53:36.764 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T01:53:36.764 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repo Size 2023-09-30T01:53:36.764 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T01:53:36.765 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2023-09-30T01:53:36.765 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-common-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 557 M 2023-09-30T01:53:36.765 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:53:36.765 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2023-09-30T01:53:36.765 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T01:53:36.765 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2023-09-30T01:53:36.765 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:53:36.766 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 557 M 2023-09-30T01:53:36.766 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 2.9 G 2023-09-30T01:53:36.766 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2023-09-30T01:54:06.683 INFO:teuthology.orchestra.run.smithi110.stdout:ceph-common-debuginfo-17.2.6-1165.g93dce5e1.el8 14 MB/s | 557 MB 00:38 2023-09-30T01:54:06.683 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:54:06.684 INFO:teuthology.orchestra.run.smithi110.stdout:Total 14 MB/s | 557 MB 00:38 2023-09-30T01:54:06.684 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2023-09-30T01:54:06.688 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2023-09-30T01:54:06.688 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2023-09-30T01:54:08.810 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2023-09-30T01:54:08.813 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2023-09-30T01:54:22.253 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-common-debuginfo-17.2.6-1165.g93dce5e1.el8 12 MB/s | 557 MB 00:45 2023-09-30T01:54:22.254 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:54:22.254 INFO:teuthology.orchestra.run.smithi059.stdout:Total 12 MB/s | 557 MB 00:45 2023-09-30T01:54:22.254 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2023-09-30T01:54:22.259 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2023-09-30T01:54:22.259 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2023-09-30T01:54:24.567 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2023-09-30T01:54:24.570 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2023-09-30T01:55:09.267 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2023-09-30T01:55:09.949 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-common-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x8 1/1 2023-09-30T01:55:10.206 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-common-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x8 1/1 2023-09-30T01:55:10.786 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-common-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x8 1/1 2023-09-30T01:55:10.786 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:55:10.786 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2023-09-30T01:55:10.786 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-common-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:55:10.786 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:55:10.786 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2023-09-30T01:55:10.961 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2023-09-30T01:55:11.811 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:05:21 ago on Sat 30 Sep 2023 01:49:50 AM UTC. 2023-09-30T01:55:13.397 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2023-09-30T01:55:13.398 INFO:teuthology.orchestra.run.smithi110.stdout:======================================================================================== 2023-09-30T01:55:13.398 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2023-09-30T01:55:13.398 INFO:teuthology.orchestra.run.smithi110.stdout:======================================================================================== 2023-09-30T01:55:13.398 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2023-09-30T01:55:13.398 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 2.7 M 2023-09-30T01:55:13.398 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:55:13.398 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2023-09-30T01:55:13.399 INFO:teuthology.orchestra.run.smithi110.stdout:======================================================================================== 2023-09-30T01:55:13.399 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2023-09-30T01:55:13.399 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:55:13.399 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 2.7 M 2023-09-30T01:55:13.399 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 13 M 2023-09-30T01:55:13.399 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2023-09-30T01:55:13.940 INFO:teuthology.orchestra.run.smithi110.stdout:ceph-immutable-object-cache-debuginfo-17.2.6-11 5.0 MB/s | 2.7 MB 00:00 2023-09-30T01:55:13.941 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:55:13.941 INFO:teuthology.orchestra.run.smithi110.stdout:Total 5.0 MB/s | 2.7 MB 00:00 2023-09-30T01:55:13.942 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2023-09-30T01:55:13.944 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2023-09-30T01:55:13.945 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2023-09-30T01:55:13.986 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2023-09-30T01:55:13.989 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2023-09-30T01:55:14.370 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2023-09-30T01:55:14.573 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:17.2.6-1165. 1/1 2023-09-30T01:55:14.835 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:17.2.6-1165. 1/1 2023-09-30T01:55:16.271 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:17.2.6-1165. 1/1 2023-09-30T01:55:16.271 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:55:16.271 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2023-09-30T01:55:16.271 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-immutable-object-cache-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:55:16.271 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:55:16.271 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2023-09-30T01:55:18.032 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-radosgw-debuginfo 2023-09-30T01:55:18.891 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:05:28 ago on Sat 30 Sep 2023 01:49:50 AM UTC. 2023-09-30T01:55:20.474 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2023-09-30T01:55:20.474 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T01:55:20.474 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2023-09-30T01:55:20.475 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T01:55:20.475 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2023-09-30T01:55:20.475 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-radosgw-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 319 M 2023-09-30T01:55:20.475 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:55:20.475 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2023-09-30T01:55:20.475 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T01:55:20.475 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2023-09-30T01:55:20.475 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:55:20.476 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 319 M 2023-09-30T01:55:20.476 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 1.7 G 2023-09-30T01:55:20.476 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2023-09-30T01:55:31.778 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2023-09-30T01:55:32.023 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-common-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x8 1/1 2023-09-30T01:55:32.327 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-common-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x8 1/1 2023-09-30T01:55:33.005 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-common-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x8 1/1 2023-09-30T01:55:33.005 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:55:33.005 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2023-09-30T01:55:33.005 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-common-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:55:33.005 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:55:33.005 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-09-30T01:55:33.195 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2023-09-30T01:55:34.173 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:05:42 ago on Sat 30 Sep 2023 01:49:52 AM UTC. 2023-09-30T01:55:35.984 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-09-30T01:55:35.985 INFO:teuthology.orchestra.run.smithi059.stdout:======================================================================================== 2023-09-30T01:55:35.985 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repo Size 2023-09-30T01:55:35.985 INFO:teuthology.orchestra.run.smithi059.stdout:======================================================================================== 2023-09-30T01:55:35.985 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2023-09-30T01:55:35.985 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 2.7 M 2023-09-30T01:55:35.985 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:55:35.985 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2023-09-30T01:55:35.985 INFO:teuthology.orchestra.run.smithi059.stdout:======================================================================================== 2023-09-30T01:55:35.986 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2023-09-30T01:55:35.986 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:55:35.986 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 2.7 M 2023-09-30T01:55:35.986 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 13 M 2023-09-30T01:55:35.986 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2023-09-30T01:55:36.940 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-immutable-object-cache-debuginfo-17.2.6-11 2.8 MB/s | 2.7 MB 00:00 2023-09-30T01:55:36.941 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:55:36.941 INFO:teuthology.orchestra.run.smithi059.stdout:Total 2.8 MB/s | 2.7 MB 00:00 2023-09-30T01:55:36.942 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2023-09-30T01:55:36.946 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2023-09-30T01:55:36.946 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2023-09-30T01:55:36.998 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2023-09-30T01:55:37.002 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2023-09-30T01:55:37.424 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2023-09-30T01:55:37.656 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:17.2.6-1165. 1/1 2023-09-30T01:55:37.666 INFO:teuthology.orchestra.run.smithi110.stdout:ceph-radosgw-debuginfo-17.2.6-1165.g93dce5e1.el 19 MB/s | 319 MB 00:17 2023-09-30T01:55:37.667 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:55:37.667 INFO:teuthology.orchestra.run.smithi110.stdout:Total 19 MB/s | 319 MB 00:17 2023-09-30T01:55:37.668 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2023-09-30T01:55:37.672 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2023-09-30T01:55:37.672 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2023-09-30T01:55:37.965 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:17.2.6-1165. 1/1 2023-09-30T01:55:38.635 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:17.2.6-1165. 1/1 2023-09-30T01:55:38.635 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:55:38.635 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2023-09-30T01:55:38.635 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-immutable-object-cache-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:55:38.635 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:55:38.635 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-09-30T01:55:38.755 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-radosgw-debuginfo 2023-09-30T01:55:38.884 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2023-09-30T01:55:38.887 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2023-09-30T01:55:39.738 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:05:47 ago on Sat 30 Sep 2023 01:49:52 AM UTC. 2023-09-30T01:55:41.505 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-09-30T01:55:41.507 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T01:55:41.507 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repo Size 2023-09-30T01:55:41.507 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T01:55:41.507 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2023-09-30T01:55:41.507 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-radosgw-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 319 M 2023-09-30T01:55:41.507 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:55:41.507 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2023-09-30T01:55:41.507 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T01:55:41.507 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2023-09-30T01:55:41.507 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:55:41.508 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 319 M 2023-09-30T01:55:41.508 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 1.7 G 2023-09-30T01:55:41.508 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2023-09-30T01:56:14.093 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2023-09-30T01:56:14.299 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-radosgw-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x 1/1 2023-09-30T01:56:14.558 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x 1/1 2023-09-30T01:56:15.159 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-radosgw-debuginfo-17.2.6-1165.g93dce5e1.el 9.5 MB/s | 319 MB 00:33 2023-09-30T01:56:15.160 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:56:15.160 INFO:teuthology.orchestra.run.smithi059.stdout:Total 9.5 MB/s | 319 MB 00:33 2023-09-30T01:56:15.161 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2023-09-30T01:56:15.165 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2023-09-30T01:56:15.165 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2023-09-30T01:56:15.633 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-radosgw-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x 1/1 2023-09-30T01:56:15.633 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:56:15.634 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2023-09-30T01:56:15.634 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-radosgw-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:56:15.634 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:56:15.634 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2023-09-30T01:56:16.482 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2023-09-30T01:56:16.486 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2023-09-30T01:56:17.520 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-test-debuginfo 2023-09-30T01:56:18.384 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:06:28 ago on Sat 30 Sep 2023 01:49:50 AM UTC. 2023-09-30T01:56:19.963 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2023-09-30T01:56:19.963 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T01:56:19.964 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2023-09-30T01:56:19.964 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T01:56:19.964 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2023-09-30T01:56:19.964 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-test-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 938 M 2023-09-30T01:56:19.964 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:56:19.964 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2023-09-30T01:56:19.964 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T01:56:19.964 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2023-09-30T01:56:19.964 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:56:19.964 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 938 M 2023-09-30T01:56:19.965 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 0 2023-09-30T01:56:19.965 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2023-09-30T01:56:55.636 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2023-09-30T01:56:55.821 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-radosgw-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x 1/1 2023-09-30T01:56:56.119 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x 1/1 2023-09-30T01:56:56.764 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-radosgw-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x 1/1 2023-09-30T01:56:56.764 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:56:56.764 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2023-09-30T01:56:56.764 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-radosgw-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:56:56.764 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:56:56.764 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-09-30T01:56:56.913 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-test-debuginfo 2023-09-30T01:56:57.892 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:07:05 ago on Sat 30 Sep 2023 01:49:52 AM UTC. 2023-09-30T01:56:59.671 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-09-30T01:56:59.672 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T01:56:59.672 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repo Size 2023-09-30T01:56:59.672 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T01:56:59.672 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2023-09-30T01:56:59.673 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-test-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 938 M 2023-09-30T01:56:59.673 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:56:59.673 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2023-09-30T01:56:59.673 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T01:56:59.673 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2023-09-30T01:56:59.673 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T01:56:59.674 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 938 M 2023-09-30T01:56:59.674 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 0 2023-09-30T01:56:59.674 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2023-09-30T01:58:09.056 INFO:teuthology.orchestra.run.smithi110.stdout:ceph-test-debuginfo-17.2.6-1165.g93dce5e1.el8.x 8.6 MB/s | 938 MB 01:49 2023-09-30T01:58:09.057 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:58:09.057 INFO:teuthology.orchestra.run.smithi110.stdout:Total 8.6 MB/s | 938 MB 01:49 2023-09-30T01:58:09.058 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2023-09-30T01:58:09.062 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2023-09-30T01:58:09.062 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2023-09-30T01:58:12.562 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2023-09-30T01:58:12.565 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2023-09-30T01:58:26.565 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-test-debuginfo-17.2.6-1165.g93dce5e1.el8.x 11 MB/s | 938 MB 01:26 2023-09-30T01:58:26.566 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2023-09-30T01:58:26.566 INFO:teuthology.orchestra.run.smithi059.stdout:Total 11 MB/s | 938 MB 01:26 2023-09-30T01:58:26.567 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2023-09-30T01:58:26.572 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2023-09-30T01:58:26.572 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2023-09-30T01:58:30.376 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2023-09-30T01:58:30.379 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2023-09-30T01:59:56.530 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2023-09-30T01:59:56.778 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-test-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_ 1/1 2023-09-30T01:59:57.045 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-test-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_ 1/1 2023-09-30T01:59:57.620 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-test-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_ 1/1 2023-09-30T01:59:57.620 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:59:57.620 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2023-09-30T01:59:57.620 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-test-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T01:59:57.620 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T01:59:57.620 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2023-09-30T01:59:57.810 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-base-debuginfo 2023-09-30T01:59:58.667 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:10:08 ago on Sat 30 Sep 2023 01:49:50 AM UTC. 2023-09-30T02:00:00.067 INFO:teuthology.orchestra.run.smithi110.stdout:Package ceph-base-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T02:00:00.253 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2023-09-30T02:00:00.254 INFO:teuthology.orchestra.run.smithi110.stdout:Nothing to do. 2023-09-30T02:00:00.254 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2023-09-30T02:00:00.595 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-mgr-debuginfo 2023-09-30T02:00:01.462 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:10:11 ago on Sat 30 Sep 2023 01:49:50 AM UTC. 2023-09-30T02:00:03.031 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2023-09-30T02:00:03.032 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T02:00:03.032 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2023-09-30T02:00:03.032 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T02:00:03.032 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2023-09-30T02:00:03.033 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 31 M 2023-09-30T02:00:03.033 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:00:03.033 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2023-09-30T02:00:03.033 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T02:00:03.033 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2023-09-30T02:00:03.033 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:00:03.033 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 31 M 2023-09-30T02:00:03.033 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 189 M 2023-09-30T02:00:03.034 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2023-09-30T02:00:04.592 INFO:teuthology.orchestra.run.smithi110.stdout:ceph-mgr-debuginfo-17.2.6-1165.g93dce5e1.el8.x8 20 MB/s | 31 MB 00:01 2023-09-30T02:00:04.593 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2023-09-30T02:00:04.593 INFO:teuthology.orchestra.run.smithi110.stdout:Total 20 MB/s | 31 MB 00:01 2023-09-30T02:00:04.594 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2023-09-30T02:00:04.598 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2023-09-30T02:00:04.598 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2023-09-30T02:00:04.746 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2023-09-30T02:00:04.749 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2023-09-30T02:00:08.340 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2023-09-30T02:00:10.369 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mgr-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_6 1/1 2023-09-30T02:00:11.191 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mgr-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_6 1/1 2023-09-30T02:00:11.777 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mgr-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_6 1/1 2023-09-30T02:00:11.778 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:00:11.778 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2023-09-30T02:00:11.778 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mgr-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:00:11.778 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:00:11.778 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2023-09-30T02:00:11.889 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-mds-debuginfo 2023-09-30T02:00:12.740 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:10:22 ago on Sat 30 Sep 2023 01:49:50 AM UTC. 2023-09-30T02:00:14.293 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2023-09-30T02:00:14.294 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T02:00:14.294 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2023-09-30T02:00:14.295 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T02:00:14.295 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2023-09-30T02:00:14.295 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mds-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 58 M 2023-09-30T02:00:14.295 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:00:14.295 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2023-09-30T02:00:14.295 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T02:00:14.295 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2023-09-30T02:00:14.295 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:00:14.295 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 58 M 2023-09-30T02:00:14.296 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 297 M 2023-09-30T02:00:14.296 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2023-09-30T02:00:17.627 INFO:teuthology.orchestra.run.smithi110.stdout:ceph-mds-debuginfo-17.2.6-1165.g93dce5e1.el8.x8 18 MB/s | 58 MB 00:03 2023-09-30T02:00:17.628 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2023-09-30T02:00:17.628 INFO:teuthology.orchestra.run.smithi110.stdout:Total 18 MB/s | 58 MB 00:03 2023-09-30T02:00:17.629 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2023-09-30T02:00:17.633 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2023-09-30T02:00:17.633 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2023-09-30T02:00:17.883 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2023-09-30T02:00:17.886 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2023-09-30T02:00:24.806 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2023-09-30T02:00:25.058 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mds-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_6 1/1 2023-09-30T02:00:25.322 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mds-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_6 1/1 2023-09-30T02:00:25.902 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2023-09-30T02:00:25.909 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mds-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_6 1/1 2023-09-30T02:00:25.909 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:00:25.910 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2023-09-30T02:00:25.910 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mds-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:00:25.910 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:00:25.910 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2023-09-30T02:00:26.022 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-mon-debuginfo 2023-09-30T02:00:26.152 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-test-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_ 1/1 2023-09-30T02:00:26.446 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-test-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_ 1/1 2023-09-30T02:00:26.870 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:10:36 ago on Sat 30 Sep 2023 01:49:50 AM UTC. 2023-09-30T02:00:27.140 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-test-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_ 1/1 2023-09-30T02:00:27.140 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:00:27.140 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2023-09-30T02:00:27.140 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-test-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:00:27.140 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:00:27.140 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-09-30T02:00:27.371 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-base-debuginfo 2023-09-30T02:00:28.352 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:10:36 ago on Sat 30 Sep 2023 01:49:52 AM UTC. 2023-09-30T02:00:28.449 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2023-09-30T02:00:28.449 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T02:00:28.450 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2023-09-30T02:00:28.450 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T02:00:28.450 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2023-09-30T02:00:28.450 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mon-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 92 M 2023-09-30T02:00:28.450 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:00:28.450 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2023-09-30T02:00:28.450 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T02:00:28.450 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2023-09-30T02:00:28.450 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:00:28.451 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 92 M 2023-09-30T02:00:28.451 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 540 M 2023-09-30T02:00:28.451 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2023-09-30T02:00:29.895 INFO:teuthology.orchestra.run.smithi059.stdout:Package ceph-base-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T02:00:30.106 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-09-30T02:00:30.171 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2023-09-30T02:00:30.171 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-09-30T02:00:30.240 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-mgr-debuginfo 2023-09-30T02:00:31.221 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:10:39 ago on Sat 30 Sep 2023 01:49:52 AM UTC. 2023-09-30T02:00:32.713 INFO:teuthology.orchestra.run.smithi110.stdout:ceph-mon-debuginfo-17.2.6-1165.g93dce5e1.el8.x8 22 MB/s | 92 MB 00:04 2023-09-30T02:00:32.714 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2023-09-30T02:00:32.714 INFO:teuthology.orchestra.run.smithi110.stdout:Total 22 MB/s | 92 MB 00:04 2023-09-30T02:00:32.718 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2023-09-30T02:00:32.722 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2023-09-30T02:00:32.722 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2023-09-30T02:00:33.013 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-09-30T02:00:33.014 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T02:00:33.014 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repo Size 2023-09-30T02:00:33.015 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T02:00:33.015 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2023-09-30T02:00:33.015 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 31 M 2023-09-30T02:00:33.015 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:00:33.015 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2023-09-30T02:00:33.015 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T02:00:33.015 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2023-09-30T02:00:33.016 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:00:33.016 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 31 M 2023-09-30T02:00:33.016 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 189 M 2023-09-30T02:00:33.016 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2023-09-30T02:00:33.097 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2023-09-30T02:00:33.100 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2023-09-30T02:00:36.815 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-mgr-debuginfo-17.2.6-1165.g93dce5e1.el8.x8 8.1 MB/s | 31 MB 00:03 2023-09-30T02:00:36.816 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2023-09-30T02:00:36.816 INFO:teuthology.orchestra.run.smithi059.stdout:Total 8.1 MB/s | 31 MB 00:03 2023-09-30T02:00:36.816 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2023-09-30T02:00:36.821 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2023-09-30T02:00:36.821 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2023-09-30T02:00:36.983 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2023-09-30T02:00:36.987 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2023-09-30T02:00:41.385 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2023-09-30T02:00:41.583 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_6 1/1 2023-09-30T02:00:41.914 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mgr-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_6 1/1 2023-09-30T02:00:42.573 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_6 1/1 2023-09-30T02:00:42.573 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:00:42.573 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2023-09-30T02:00:42.573 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:00:42.573 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:00:42.573 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-09-30T02:00:42.725 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-mds-debuginfo 2023-09-30T02:00:43.432 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2023-09-30T02:00:43.647 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-mon-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_6 1/1 2023-09-30T02:00:43.705 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:10:51 ago on Sat 30 Sep 2023 01:49:52 AM UTC. 2023-09-30T02:00:43.905 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-mon-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_6 1/1 2023-09-30T02:00:44.477 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-mon-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_6 1/1 2023-09-30T02:00:44.477 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:00:44.477 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2023-09-30T02:00:44.477 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-mon-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:00:44.478 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:00:44.478 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2023-09-30T02:00:44.596 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-osd-debuginfo 2023-09-30T02:00:45.445 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:10:55 ago on Sat 30 Sep 2023 01:49:50 AM UTC. 2023-09-30T02:00:45.481 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-09-30T02:00:45.482 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T02:00:45.482 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repo Size 2023-09-30T02:00:45.483 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T02:00:45.483 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2023-09-30T02:00:45.483 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mds-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 58 M 2023-09-30T02:00:45.483 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:00:45.483 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2023-09-30T02:00:45.483 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T02:00:45.484 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2023-09-30T02:00:45.484 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:00:45.484 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 58 M 2023-09-30T02:00:45.484 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 297 M 2023-09-30T02:00:45.484 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2023-09-30T02:00:47.028 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2023-09-30T02:00:47.028 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T02:00:47.028 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2023-09-30T02:00:47.028 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T02:00:47.028 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2023-09-30T02:00:47.028 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-osd-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 312 M 2023-09-30T02:00:47.029 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:00:47.029 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2023-09-30T02:00:47.029 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T02:00:47.029 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2023-09-30T02:00:47.029 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:00:47.029 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 312 M 2023-09-30T02:00:47.030 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 1.7 G 2023-09-30T02:00:47.030 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2023-09-30T02:00:48.864 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-mds-debuginfo-17.2.6-1165.g93dce5e1.el8.x8 17 MB/s | 58 MB 00:03 2023-09-30T02:00:48.865 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2023-09-30T02:00:48.865 INFO:teuthology.orchestra.run.smithi059.stdout:Total 17 MB/s | 58 MB 00:03 2023-09-30T02:00:48.865 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2023-09-30T02:00:48.870 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2023-09-30T02:00:48.870 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2023-09-30T02:00:49.143 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2023-09-30T02:00:49.146 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2023-09-30T02:00:56.054 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2023-09-30T02:00:56.281 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mds-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_6 1/1 2023-09-30T02:00:56.578 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mds-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_6 1/1 2023-09-30T02:00:57.237 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mds-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_6 1/1 2023-09-30T02:00:57.237 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:00:57.237 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2023-09-30T02:00:57.237 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mds-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:00:57.238 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:00:57.238 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-09-30T02:00:57.361 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-mon-debuginfo 2023-09-30T02:00:58.338 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:11:06 ago on Sat 30 Sep 2023 01:49:52 AM UTC. 2023-09-30T02:01:00.093 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-09-30T02:01:00.094 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T02:01:00.094 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repo Size 2023-09-30T02:01:00.094 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T02:01:00.094 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2023-09-30T02:01:00.095 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mon-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 92 M 2023-09-30T02:01:00.095 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:01:00.095 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2023-09-30T02:01:00.095 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T02:01:00.095 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2023-09-30T02:01:00.095 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:01:00.096 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 92 M 2023-09-30T02:01:00.096 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 540 M 2023-09-30T02:01:00.096 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2023-09-30T02:01:07.718 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-mon-debuginfo-17.2.6-1165.g93dce5e1.el8.x8 12 MB/s | 92 MB 00:07 2023-09-30T02:01:07.718 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2023-09-30T02:01:07.719 INFO:teuthology.orchestra.run.smithi059.stdout:Total 12 MB/s | 92 MB 00:07 2023-09-30T02:01:07.719 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2023-09-30T02:01:07.724 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2023-09-30T02:01:07.724 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2023-09-30T02:01:07.996 INFO:teuthology.orchestra.run.smithi110.stdout:ceph-osd-debuginfo-17.2.6-1165.g93dce5e1.el8.x8 15 MB/s | 312 MB 00:20 2023-09-30T02:01:07.997 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2023-09-30T02:01:07.997 INFO:teuthology.orchestra.run.smithi110.stdout:Total 15 MB/s | 312 MB 00:20 2023-09-30T02:01:07.998 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2023-09-30T02:01:08.002 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2023-09-30T02:01:08.002 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2023-09-30T02:01:08.132 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2023-09-30T02:01:08.135 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2023-09-30T02:01:09.190 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2023-09-30T02:01:09.193 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2023-09-30T02:01:19.585 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2023-09-30T02:01:19.778 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mon-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_6 1/1 2023-09-30T02:01:20.097 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mon-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_6 1/1 2023-09-30T02:01:20.754 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mon-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_6 1/1 2023-09-30T02:01:20.754 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:01:20.754 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2023-09-30T02:01:20.755 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mon-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:01:20.755 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:01:20.755 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-09-30T02:01:22.493 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-osd-debuginfo 2023-09-30T02:01:23.480 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:11:31 ago on Sat 30 Sep 2023 01:49:52 AM UTC. 2023-09-30T02:01:25.252 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-09-30T02:01:25.253 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T02:01:25.253 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repo Size 2023-09-30T02:01:25.253 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T02:01:25.253 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2023-09-30T02:01:25.254 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-osd-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 312 M 2023-09-30T02:01:25.254 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:01:25.254 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2023-09-30T02:01:25.254 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T02:01:25.254 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2023-09-30T02:01:25.254 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:01:25.255 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 312 M 2023-09-30T02:01:25.255 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 1.7 G 2023-09-30T02:01:25.255 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2023-09-30T02:01:39.268 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-osd-debuginfo-17.2.6-1165.g93dce5e1.el8.x8 22 MB/s | 312 MB 00:14 2023-09-30T02:01:39.268 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2023-09-30T02:01:39.269 INFO:teuthology.orchestra.run.smithi059.stdout:Total 22 MB/s | 312 MB 00:14 2023-09-30T02:01:39.269 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2023-09-30T02:01:39.274 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2023-09-30T02:01:39.274 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2023-09-30T02:01:40.566 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2023-09-30T02:01:40.569 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2023-09-30T02:01:43.179 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2023-09-30T02:01:43.364 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-osd-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_6 1/1 2023-09-30T02:01:43.615 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-osd-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_6 1/1 2023-09-30T02:01:44.210 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-osd-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_6 1/1 2023-09-30T02:01:44.210 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:01:44.211 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2023-09-30T02:01:44.211 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-osd-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:01:44.211 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:01:44.211 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2023-09-30T02:01:44.351 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install ceph-fuse-debuginfo 2023-09-30T02:01:45.204 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:11:55 ago on Sat 30 Sep 2023 01:49:50 AM UTC. 2023-09-30T02:01:46.851 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2023-09-30T02:01:46.851 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T02:01:46.851 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2023-09-30T02:01:46.851 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T02:01:46.851 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2023-09-30T02:01:46.851 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-fuse-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 12 M 2023-09-30T02:01:46.852 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:01:46.852 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2023-09-30T02:01:46.852 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T02:01:46.852 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2023-09-30T02:01:46.852 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:01:46.852 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 12 M 2023-09-30T02:01:46.853 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 59 M 2023-09-30T02:01:46.853 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2023-09-30T02:01:47.979 INFO:teuthology.orchestra.run.smithi110.stdout:ceph-fuse-debuginfo-17.2.6-1165.g93dce5e1.el8.x 11 MB/s | 12 MB 00:01 2023-09-30T02:01:47.980 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2023-09-30T02:01:47.980 INFO:teuthology.orchestra.run.smithi110.stdout:Total 11 MB/s | 12 MB 00:01 2023-09-30T02:01:47.981 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2023-09-30T02:01:47.985 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2023-09-30T02:01:47.985 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2023-09-30T02:01:48.065 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2023-09-30T02:01:48.068 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2023-09-30T02:01:50.069 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2023-09-30T02:01:50.997 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : ceph-fuse-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_ 1/1 2023-09-30T02:01:51.265 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: ceph-fuse-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_ 1/1 2023-09-30T02:01:51.854 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : ceph-fuse-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_ 1/1 2023-09-30T02:01:51.854 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:01:51.854 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2023-09-30T02:01:51.854 INFO:teuthology.orchestra.run.smithi110.stdout: ceph-fuse-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:01:51.854 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:01:51.855 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2023-09-30T02:01:51.970 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install librados-devel-debuginfo 2023-09-30T02:01:52.824 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:12:02 ago on Sat 30 Sep 2023 01:49:50 AM UTC. 2023-09-30T02:01:54.410 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2023-09-30T02:01:54.411 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T02:01:54.411 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2023-09-30T02:01:54.411 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T02:01:54.411 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2023-09-30T02:01:54.412 INFO:teuthology.orchestra.run.smithi110.stdout: librados-devel-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 128 k 2023-09-30T02:01:54.412 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:01:54.412 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2023-09-30T02:01:54.412 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T02:01:54.412 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2023-09-30T02:01:54.412 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:01:54.412 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 128 k 2023-09-30T02:01:54.412 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 568 k 2023-09-30T02:01:54.412 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2023-09-30T02:01:54.606 INFO:teuthology.orchestra.run.smithi110.stdout:librados-devel-debuginfo-17.2.6-1165.g93dce5e1. 661 kB/s | 128 kB 00:00 2023-09-30T02:01:54.607 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2023-09-30T02:01:54.607 INFO:teuthology.orchestra.run.smithi110.stdout:Total 657 kB/s | 128 kB 00:00 2023-09-30T02:01:54.607 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2023-09-30T02:01:54.610 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2023-09-30T02:01:54.611 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2023-09-30T02:01:54.643 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2023-09-30T02:01:54.646 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2023-09-30T02:01:54.735 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2023-09-30T02:01:54.946 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : librados-devel-debuginfo-2:17.2.6-1165.g93dce5e1.el8 1/1 2023-09-30T02:01:55.210 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librados-devel-debuginfo-2:17.2.6-1165.g93dce5e1.el8 1/1 2023-09-30T02:01:55.824 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librados-devel-debuginfo-2:17.2.6-1165.g93dce5e1.el8 1/1 2023-09-30T02:01:55.824 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:01:55.824 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2023-09-30T02:01:55.824 INFO:teuthology.orchestra.run.smithi110.stdout: librados-devel-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:01:55.824 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:01:55.824 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2023-09-30T02:01:55.935 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install libcephfs2-debuginfo 2023-09-30T02:01:56.782 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:12:06 ago on Sat 30 Sep 2023 01:49:50 AM UTC. 2023-09-30T02:01:58.352 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2023-09-30T02:01:58.353 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T02:01:58.353 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2023-09-30T02:01:58.353 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T02:01:58.353 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2023-09-30T02:01:58.353 INFO:teuthology.orchestra.run.smithi110.stdout: libcephfs2-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 11 M 2023-09-30T02:01:58.353 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:01:58.353 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2023-09-30T02:01:58.353 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T02:01:58.353 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2023-09-30T02:01:58.354 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:01:58.354 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 11 M 2023-09-30T02:01:58.354 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 55 M 2023-09-30T02:01:58.354 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2023-09-30T02:01:59.528 INFO:teuthology.orchestra.run.smithi110.stdout:libcephfs2-debuginfo-17.2.6-1165.g93dce5e1.el8. 9.6 MB/s | 11 MB 00:01 2023-09-30T02:01:59.529 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2023-09-30T02:01:59.529 INFO:teuthology.orchestra.run.smithi110.stdout:Total 9.6 MB/s | 11 MB 00:01 2023-09-30T02:01:59.530 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2023-09-30T02:01:59.534 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2023-09-30T02:01:59.534 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2023-09-30T02:01:59.612 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2023-09-30T02:01:59.615 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2023-09-30T02:02:01.431 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2023-09-30T02:02:01.645 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : libcephfs2-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86 1/1 2023-09-30T02:02:01.906 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: libcephfs2-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86 1/1 2023-09-30T02:02:02.481 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : libcephfs2-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86 1/1 2023-09-30T02:02:02.482 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:02:02.482 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2023-09-30T02:02:02.482 INFO:teuthology.orchestra.run.smithi110.stdout: libcephfs2-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:02:02.482 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:02:02.482 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2023-09-30T02:02:02.595 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install librados2-debuginfo 2023-09-30T02:02:03.441 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:12:13 ago on Sat 30 Sep 2023 01:49:50 AM UTC. 2023-09-30T02:02:05.023 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2023-09-30T02:02:05.023 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T02:02:05.023 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2023-09-30T02:02:05.024 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T02:02:05.024 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2023-09-30T02:02:05.024 INFO:teuthology.orchestra.run.smithi110.stdout: librados2-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 63 M 2023-09-30T02:02:05.024 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:02:05.024 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2023-09-30T02:02:05.024 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T02:02:05.024 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2023-09-30T02:02:05.024 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:02:05.025 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 63 M 2023-09-30T02:02:05.025 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 419 M 2023-09-30T02:02:05.025 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2023-09-30T02:02:08.655 INFO:teuthology.orchestra.run.smithi110.stdout:librados2-debuginfo-17.2.6-1165.g93dce5e1.el8.x 17 MB/s | 63 MB 00:03 2023-09-30T02:02:08.655 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2023-09-30T02:02:08.656 INFO:teuthology.orchestra.run.smithi110.stdout:Total 17 MB/s | 63 MB 00:03 2023-09-30T02:02:08.656 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2023-09-30T02:02:08.660 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2023-09-30T02:02:08.661 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2023-09-30T02:02:08.952 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2023-09-30T02:02:08.955 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2023-09-30T02:02:16.449 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2023-09-30T02:02:16.652 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : librados2-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_ 1/1 2023-09-30T02:02:16.915 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librados2-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_ 1/1 2023-09-30T02:02:17.506 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librados2-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_ 1/1 2023-09-30T02:02:17.506 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:02:17.506 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2023-09-30T02:02:17.506 INFO:teuthology.orchestra.run.smithi110.stdout: librados2-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:02:17.506 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:02:17.506 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2023-09-30T02:02:17.622 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install librbd1-debuginfo 2023-09-30T02:02:18.243 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2023-09-30T02:02:18.437 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-osd-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_6 1/1 2023-09-30T02:02:18.474 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:12:28 ago on Sat 30 Sep 2023 01:49:50 AM UTC. 2023-09-30T02:02:18.731 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-osd-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_6 1/1 2023-09-30T02:02:19.385 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-osd-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_6 1/1 2023-09-30T02:02:19.385 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:02:19.385 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2023-09-30T02:02:19.386 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-osd-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:02:19.386 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:02:19.386 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-09-30T02:02:20.053 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2023-09-30T02:02:20.054 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T02:02:20.054 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2023-09-30T02:02:20.054 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T02:02:20.054 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2023-09-30T02:02:20.054 INFO:teuthology.orchestra.run.smithi110.stdout: librbd1-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 81 M 2023-09-30T02:02:20.054 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:02:20.055 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2023-09-30T02:02:20.055 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T02:02:20.055 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2023-09-30T02:02:20.055 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:02:20.055 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 81 M 2023-09-30T02:02:20.055 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 525 M 2023-09-30T02:02:20.056 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2023-09-30T02:02:21.322 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-fuse-debuginfo 2023-09-30T02:02:22.308 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:12:30 ago on Sat 30 Sep 2023 01:49:52 AM UTC. 2023-09-30T02:02:23.914 INFO:teuthology.orchestra.run.smithi110.stdout:librbd1-debuginfo-17.2.6-1165.g93dce5e1.el8.x86 21 MB/s | 81 MB 00:03 2023-09-30T02:02:23.915 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2023-09-30T02:02:23.916 INFO:teuthology.orchestra.run.smithi110.stdout:Total 21 MB/s | 81 MB 00:03 2023-09-30T02:02:23.916 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2023-09-30T02:02:23.920 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2023-09-30T02:02:23.920 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2023-09-30T02:02:24.094 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-09-30T02:02:24.095 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T02:02:24.095 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repo Size 2023-09-30T02:02:24.095 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T02:02:24.095 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2023-09-30T02:02:24.095 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-fuse-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 12 M 2023-09-30T02:02:24.095 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:02:24.095 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2023-09-30T02:02:24.095 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T02:02:24.095 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2023-09-30T02:02:24.095 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:02:24.096 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 12 M 2023-09-30T02:02:24.096 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 59 M 2023-09-30T02:02:24.096 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2023-09-30T02:02:24.280 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2023-09-30T02:02:24.282 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2023-09-30T02:02:24.858 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-fuse-debuginfo-17.2.6-1165.g93dce5e1.el8.x 16 MB/s | 12 MB 00:00 2023-09-30T02:02:24.859 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2023-09-30T02:02:24.859 INFO:teuthology.orchestra.run.smithi059.stdout:Total 16 MB/s | 12 MB 00:00 2023-09-30T02:02:24.860 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2023-09-30T02:02:24.864 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2023-09-30T02:02:24.864 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2023-09-30T02:02:24.952 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2023-09-30T02:02:24.955 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2023-09-30T02:02:26.446 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2023-09-30T02:02:26.628 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-fuse-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_ 1/1 2023-09-30T02:02:26.938 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-fuse-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_ 1/1 2023-09-30T02:02:27.640 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-fuse-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_ 1/1 2023-09-30T02:02:27.641 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:02:27.641 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2023-09-30T02:02:27.641 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-fuse-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:02:27.641 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:02:27.641 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-09-30T02:02:27.767 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install librados-devel-debuginfo 2023-09-30T02:02:28.755 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:12:36 ago on Sat 30 Sep 2023 01:49:52 AM UTC. 2023-09-30T02:02:30.523 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-09-30T02:02:30.523 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T02:02:30.523 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repo Size 2023-09-30T02:02:30.524 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T02:02:30.524 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2023-09-30T02:02:30.524 INFO:teuthology.orchestra.run.smithi059.stdout: librados-devel-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 128 k 2023-09-30T02:02:30.524 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:02:30.524 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2023-09-30T02:02:30.524 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T02:02:30.524 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2023-09-30T02:02:30.524 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:02:30.525 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 128 k 2023-09-30T02:02:30.525 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 568 k 2023-09-30T02:02:30.525 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2023-09-30T02:02:30.885 INFO:teuthology.orchestra.run.smithi059.stdout:librados-devel-debuginfo-17.2.6-1165.g93dce5e1. 356 kB/s | 128 kB 00:00 2023-09-30T02:02:30.885 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2023-09-30T02:02:30.886 INFO:teuthology.orchestra.run.smithi059.stdout:Total 355 kB/s | 128 kB 00:00 2023-09-30T02:02:30.886 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2023-09-30T02:02:30.890 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2023-09-30T02:02:30.891 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2023-09-30T02:02:30.932 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2023-09-30T02:02:30.936 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2023-09-30T02:02:31.034 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2023-09-30T02:02:31.252 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : librados-devel-debuginfo-2:17.2.6-1165.g93dce5e1.el8 1/1 2023-09-30T02:02:31.545 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librados-devel-debuginfo-2:17.2.6-1165.g93dce5e1.el8 1/1 2023-09-30T02:02:32.268 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librados-devel-debuginfo-2:17.2.6-1165.g93dce5e1.el8 1/1 2023-09-30T02:02:32.268 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:02:32.268 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2023-09-30T02:02:32.268 INFO:teuthology.orchestra.run.smithi059.stdout: librados-devel-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:02:32.268 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:02:32.268 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-09-30T02:02:32.398 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install libcephfs2-debuginfo 2023-09-30T02:02:33.376 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:12:41 ago on Sat 30 Sep 2023 01:49:52 AM UTC. 2023-09-30T02:02:33.809 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2023-09-30T02:02:34.300 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : librbd1-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T02:02:34.556 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: librbd1-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T02:02:35.136 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : librbd1-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T02:02:35.137 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:02:35.137 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2023-09-30T02:02:35.137 INFO:teuthology.orchestra.run.smithi110.stdout: librbd1-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:02:35.137 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:02:35.137 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2023-09-30T02:02:35.149 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-09-30T02:02:35.150 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T02:02:35.150 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repo Size 2023-09-30T02:02:35.150 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T02:02:35.150 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2023-09-30T02:02:35.150 INFO:teuthology.orchestra.run.smithi059.stdout: libcephfs2-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 11 M 2023-09-30T02:02:35.150 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:02:35.150 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2023-09-30T02:02:35.151 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T02:02:35.151 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2023-09-30T02:02:35.151 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:02:35.152 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 11 M 2023-09-30T02:02:35.152 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 55 M 2023-09-30T02:02:35.152 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2023-09-30T02:02:35.265 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install python3-cephfs-debuginfo 2023-09-30T02:02:36.131 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:12:46 ago on Sat 30 Sep 2023 01:49:50 AM UTC. 2023-09-30T02:02:37.467 INFO:teuthology.orchestra.run.smithi059.stdout:libcephfs2-debuginfo-17.2.6-1165.g93dce5e1.el8. 4.9 MB/s | 11 MB 00:02 2023-09-30T02:02:37.468 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2023-09-30T02:02:37.469 INFO:teuthology.orchestra.run.smithi059.stdout:Total 4.9 MB/s | 11 MB 00:02 2023-09-30T02:02:37.469 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2023-09-30T02:02:37.473 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2023-09-30T02:02:37.473 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2023-09-30T02:02:37.559 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2023-09-30T02:02:37.562 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2023-09-30T02:02:37.701 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2023-09-30T02:02:37.702 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T02:02:37.702 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2023-09-30T02:02:37.702 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T02:02:37.702 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2023-09-30T02:02:37.702 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cephfs-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 620 k 2023-09-30T02:02:37.702 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:02:37.702 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2023-09-30T02:02:37.702 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T02:02:37.703 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2023-09-30T02:02:37.703 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:02:37.703 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 620 k 2023-09-30T02:02:37.703 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 3.0 M 2023-09-30T02:02:37.703 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2023-09-30T02:02:37.949 INFO:teuthology.orchestra.run.smithi110.stdout:python3-cephfs-debuginfo-17.2.6-1165.g93dce5e1. 2.5 MB/s | 620 kB 00:00 2023-09-30T02:02:37.950 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2023-09-30T02:02:37.950 INFO:teuthology.orchestra.run.smithi110.stdout:Total 2.4 MB/s | 620 kB 00:00 2023-09-30T02:02:37.950 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2023-09-30T02:02:37.953 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2023-09-30T02:02:37.954 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2023-09-30T02:02:37.989 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2023-09-30T02:02:37.992 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2023-09-30T02:02:38.134 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2023-09-30T02:02:38.324 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-cephfs-debuginfo-2:17.2.6-1165.g93dce5e1.el8 1/1 2023-09-30T02:02:38.595 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: python3-cephfs-debuginfo-2:17.2.6-1165.g93dce5e1.el8 1/1 2023-09-30T02:02:39.181 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-cephfs-debuginfo-2:17.2.6-1165.g93dce5e1.el8 1/1 2023-09-30T02:02:39.181 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:02:39.181 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2023-09-30T02:02:39.181 INFO:teuthology.orchestra.run.smithi110.stdout: python3-cephfs-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:02:39.181 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:02:39.181 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2023-09-30T02:02:39.315 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install python3-rados-debuginfo 2023-09-30T02:02:39.345 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2023-09-30T02:02:39.543 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libcephfs2-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86 1/1 2023-09-30T02:02:39.835 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: libcephfs2-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86 1/1 2023-09-30T02:02:40.165 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:12:50 ago on Sat 30 Sep 2023 01:49:50 AM UTC. 2023-09-30T02:02:40.506 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libcephfs2-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86 1/1 2023-09-30T02:02:40.506 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:02:40.506 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2023-09-30T02:02:40.506 INFO:teuthology.orchestra.run.smithi059.stdout: libcephfs2-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:02:40.506 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:02:40.506 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-09-30T02:02:40.632 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install librados2-debuginfo 2023-09-30T02:02:41.609 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:12:49 ago on Sat 30 Sep 2023 01:49:52 AM UTC. 2023-09-30T02:02:41.739 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2023-09-30T02:02:41.739 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T02:02:41.739 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2023-09-30T02:02:41.740 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T02:02:41.740 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2023-09-30T02:02:41.740 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rados-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 721 k 2023-09-30T02:02:41.740 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:02:41.740 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2023-09-30T02:02:41.740 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T02:02:41.740 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2023-09-30T02:02:41.740 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:02:41.741 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 721 k 2023-09-30T02:02:41.741 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 3.3 M 2023-09-30T02:02:41.741 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2023-09-30T02:02:42.002 INFO:teuthology.orchestra.run.smithi110.stdout:python3-rados-debuginfo-17.2.6-1165.g93dce5e1.e 2.7 MB/s | 721 kB 00:00 2023-09-30T02:02:42.002 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2023-09-30T02:02:42.002 INFO:teuthology.orchestra.run.smithi110.stdout:Total 2.7 MB/s | 721 kB 00:00 2023-09-30T02:02:42.003 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2023-09-30T02:02:42.006 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2023-09-30T02:02:42.006 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2023-09-30T02:02:42.041 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2023-09-30T02:02:42.043 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2023-09-30T02:02:42.191 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2023-09-30T02:02:42.373 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-rados-debuginfo-2:17.2.6-1165.g93dce5e1.el8. 1/1 2023-09-30T02:02:42.637 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: python3-rados-debuginfo-2:17.2.6-1165.g93dce5e1.el8. 1/1 2023-09-30T02:02:43.216 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-rados-debuginfo-2:17.2.6-1165.g93dce5e1.el8. 1/1 2023-09-30T02:02:43.216 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:02:43.217 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2023-09-30T02:02:43.217 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rados-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:02:43.217 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:02:43.217 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2023-09-30T02:02:43.340 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install python3-rbd-debuginfo 2023-09-30T02:02:43.365 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-09-30T02:02:43.366 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T02:02:43.366 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repo Size 2023-09-30T02:02:43.366 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T02:02:43.366 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2023-09-30T02:02:43.366 INFO:teuthology.orchestra.run.smithi059.stdout: librados2-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 63 M 2023-09-30T02:02:43.366 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:02:43.367 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2023-09-30T02:02:43.367 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T02:02:43.367 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2023-09-30T02:02:43.367 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:02:43.367 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 63 M 2023-09-30T02:02:43.367 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 419 M 2023-09-30T02:02:43.368 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2023-09-30T02:02:44.190 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:12:54 ago on Sat 30 Sep 2023 01:49:50 AM UTC. 2023-09-30T02:02:45.775 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2023-09-30T02:02:45.775 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T02:02:45.775 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2023-09-30T02:02:45.776 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T02:02:45.776 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2023-09-30T02:02:45.776 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rbd-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 1.2 M 2023-09-30T02:02:45.776 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:02:45.776 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2023-09-30T02:02:45.776 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T02:02:45.776 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2023-09-30T02:02:45.776 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:02:45.777 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 1.2 M 2023-09-30T02:02:45.777 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 6.2 M 2023-09-30T02:02:45.777 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2023-09-30T02:02:46.472 INFO:teuthology.orchestra.run.smithi059.stdout:librados2-debuginfo-17.2.6-1165.g93dce5e1.el8.x 20 MB/s | 63 MB 00:03 2023-09-30T02:02:46.473 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2023-09-30T02:02:46.473 INFO:teuthology.orchestra.run.smithi059.stdout:Total 20 MB/s | 63 MB 00:03 2023-09-30T02:02:46.474 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2023-09-30T02:02:46.478 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2023-09-30T02:02:46.478 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2023-09-30T02:02:46.513 INFO:teuthology.orchestra.run.smithi110.stdout:python3-rbd-debuginfo-17.2.6-1165.g93dce5e1.el8 1.6 MB/s | 1.2 MB 00:00 2023-09-30T02:02:46.514 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2023-09-30T02:02:46.514 INFO:teuthology.orchestra.run.smithi110.stdout:Total 1.6 MB/s | 1.2 MB 00:00 2023-09-30T02:02:46.514 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2023-09-30T02:02:46.519 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2023-09-30T02:02:46.519 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2023-09-30T02:02:46.560 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2023-09-30T02:02:46.563 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2023-09-30T02:02:46.799 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2023-09-30T02:02:46.803 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2023-09-30T02:02:47.153 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2023-09-30T02:02:47.373 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-rbd-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x8 1/1 2023-09-30T02:02:47.619 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: python3-rbd-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x8 1/1 2023-09-30T02:02:48.189 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-rbd-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x8 1/1 2023-09-30T02:02:48.190 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:02:48.190 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2023-09-30T02:02:48.190 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rbd-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:02:48.190 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:02:48.190 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2023-09-30T02:02:48.303 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install python3-rgw-debuginfo 2023-09-30T02:02:49.166 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:12:59 ago on Sat 30 Sep 2023 01:49:50 AM UTC. 2023-09-30T02:02:50.749 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2023-09-30T02:02:50.749 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T02:02:50.750 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2023-09-30T02:02:50.750 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T02:02:50.750 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2023-09-30T02:02:50.750 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rgw-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 318 k 2023-09-30T02:02:50.750 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:02:50.751 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2023-09-30T02:02:50.751 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T02:02:50.751 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2023-09-30T02:02:50.751 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:02:50.751 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 318 k 2023-09-30T02:02:50.751 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 1.3 M 2023-09-30T02:02:50.751 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2023-09-30T02:02:51.082 INFO:teuthology.orchestra.run.smithi110.stdout:python3-rgw-debuginfo-17.2.6-1165.g93dce5e1.el8 961 kB/s | 318 kB 00:00 2023-09-30T02:02:51.083 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2023-09-30T02:02:51.083 INFO:teuthology.orchestra.run.smithi110.stdout:Total 957 kB/s | 318 kB 00:00 2023-09-30T02:02:51.083 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2023-09-30T02:02:51.086 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2023-09-30T02:02:51.086 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2023-09-30T02:02:51.121 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2023-09-30T02:02:51.124 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2023-09-30T02:02:51.233 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2023-09-30T02:02:51.447 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : python3-rgw-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x8 1/1 2023-09-30T02:02:51.710 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: python3-rgw-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x8 1/1 2023-09-30T02:02:52.367 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : python3-rgw-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x8 1/1 2023-09-30T02:02:52.367 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:02:52.367 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2023-09-30T02:02:52.367 INFO:teuthology.orchestra.run.smithi110.stdout: python3-rgw-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:02:52.367 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:02:52.367 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2023-09-30T02:02:52.479 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install rbd-fuse-debuginfo 2023-09-30T02:02:53.332 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:13:03 ago on Sat 30 Sep 2023 01:49:50 AM UTC. 2023-09-30T02:02:54.909 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2023-09-30T02:02:54.909 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T02:02:54.909 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2023-09-30T02:02:54.909 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T02:02:54.910 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2023-09-30T02:02:54.910 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-fuse-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 1.2 M 2023-09-30T02:02:54.910 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:02:54.910 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2023-09-30T02:02:54.910 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T02:02:54.910 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2023-09-30T02:02:54.910 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:02:54.911 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 1.2 M 2023-09-30T02:02:54.911 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 5.6 M 2023-09-30T02:02:54.911 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2023-09-30T02:02:55.106 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2023-09-30T02:02:55.307 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : librados2-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_ 1/1 2023-09-30T02:02:55.392 INFO:teuthology.orchestra.run.smithi110.stdout:rbd-fuse-debuginfo-17.2.6-1165.g93dce5e1.el8.x8 2.5 MB/s | 1.2 MB 00:00 2023-09-30T02:02:55.393 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2023-09-30T02:02:55.393 INFO:teuthology.orchestra.run.smithi110.stdout:Total 2.5 MB/s | 1.2 MB 00:00 2023-09-30T02:02:55.393 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2023-09-30T02:02:55.397 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2023-09-30T02:02:55.397 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2023-09-30T02:02:55.434 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2023-09-30T02:02:55.437 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2023-09-30T02:02:55.610 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librados2-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_ 1/1 2023-09-30T02:02:55.640 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2023-09-30T02:02:55.846 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : rbd-fuse-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_6 1/1 2023-09-30T02:02:56.115 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: rbd-fuse-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_6 1/1 2023-09-30T02:02:56.284 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librados2-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_ 1/1 2023-09-30T02:02:56.285 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:02:56.285 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2023-09-30T02:02:56.285 INFO:teuthology.orchestra.run.smithi059.stdout: librados2-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:02:56.285 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:02:56.285 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-09-30T02:02:56.412 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install librbd1-debuginfo 2023-09-30T02:02:56.700 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : rbd-fuse-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_6 1/1 2023-09-30T02:02:56.701 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:02:56.701 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2023-09-30T02:02:56.701 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-fuse-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:02:56.701 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:02:56.701 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2023-09-30T02:02:56.810 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install rbd-mirror-debuginfo 2023-09-30T02:02:57.401 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:13:05 ago on Sat 30 Sep 2023 01:49:52 AM UTC. 2023-09-30T02:02:57.660 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:13:07 ago on Sat 30 Sep 2023 01:49:50 AM UTC. 2023-09-30T02:02:59.156 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-09-30T02:02:59.157 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T02:02:59.157 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repo Size 2023-09-30T02:02:59.157 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T02:02:59.157 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2023-09-30T02:02:59.157 INFO:teuthology.orchestra.run.smithi059.stdout: librbd1-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 81 M 2023-09-30T02:02:59.157 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:02:59.157 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2023-09-30T02:02:59.157 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T02:02:59.158 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2023-09-30T02:02:59.158 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:02:59.158 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 81 M 2023-09-30T02:02:59.158 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 525 M 2023-09-30T02:02:59.158 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2023-09-30T02:02:59.240 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2023-09-30T02:02:59.241 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T02:02:59.241 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2023-09-30T02:02:59.241 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T02:02:59.241 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2023-09-30T02:02:59.241 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-mirror-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 72 M 2023-09-30T02:02:59.241 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:02:59.241 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2023-09-30T02:02:59.242 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T02:02:59.242 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2023-09-30T02:02:59.242 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:02:59.242 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 72 M 2023-09-30T02:02:59.242 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 480 M 2023-09-30T02:02:59.242 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2023-09-30T02:03:05.644 INFO:teuthology.orchestra.run.smithi110.stdout:rbd-mirror-debuginfo-17.2.6-1165.g93dce5e1.el8. 11 MB/s | 72 MB 00:06 2023-09-30T02:03:05.645 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2023-09-30T02:03:05.645 INFO:teuthology.orchestra.run.smithi110.stdout:Total 11 MB/s | 72 MB 00:06 2023-09-30T02:03:05.646 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2023-09-30T02:03:05.650 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2023-09-30T02:03:05.650 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2023-09-30T02:03:05.890 INFO:teuthology.orchestra.run.smithi059.stdout:librbd1-debuginfo-17.2.6-1165.g93dce5e1.el8.x86 12 MB/s | 81 MB 00:06 2023-09-30T02:03:05.891 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2023-09-30T02:03:05.891 INFO:teuthology.orchestra.run.smithi059.stdout:Total 12 MB/s | 81 MB 00:06 2023-09-30T02:03:05.891 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2023-09-30T02:03:05.896 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2023-09-30T02:03:05.896 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2023-09-30T02:03:05.949 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2023-09-30T02:03:05.952 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2023-09-30T02:03:06.291 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2023-09-30T02:03:06.294 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2023-09-30T02:03:14.504 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2023-09-30T02:03:14.722 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : rbd-mirror-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86 1/1 2023-09-30T02:03:14.980 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: rbd-mirror-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86 1/1 2023-09-30T02:03:15.562 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : rbd-mirror-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86 1/1 2023-09-30T02:03:15.563 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:03:15.563 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2023-09-30T02:03:15.563 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-mirror-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:03:15.563 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:03:15.563 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2023-09-30T02:03:15.679 DEBUG:teuthology.orchestra.run.smithi110:> sudo yum -y install rbd-nbd-debuginfo 2023-09-30T02:03:16.529 INFO:teuthology.orchestra.run.smithi110.stdout:Last metadata expiration check: 0:13:26 ago on Sat 30 Sep 2023 01:49:50 AM UTC. 2023-09-30T02:03:16.830 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2023-09-30T02:03:17.088 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : librbd1-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T02:03:17.388 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librbd1-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T02:03:18.071 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librbd1-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T02:03:18.071 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:03:18.071 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2023-09-30T02:03:18.071 INFO:teuthology.orchestra.run.smithi059.stdout: librbd1-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:03:18.071 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:03:18.071 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-09-30T02:03:18.165 INFO:teuthology.orchestra.run.smithi110.stdout:Dependencies resolved. 2023-09-30T02:03:18.166 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T02:03:18.166 INFO:teuthology.orchestra.run.smithi110.stdout: Package Arch Version Repo Size 2023-09-30T02:03:18.166 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T02:03:18.166 INFO:teuthology.orchestra.run.smithi110.stdout:Installing: 2023-09-30T02:03:18.166 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-nbd-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 2.0 M 2023-09-30T02:03:18.166 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:03:18.166 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction Summary 2023-09-30T02:03:18.166 INFO:teuthology.orchestra.run.smithi110.stdout:================================================================================ 2023-09-30T02:03:18.166 INFO:teuthology.orchestra.run.smithi110.stdout:Install 1 Package 2023-09-30T02:03:18.167 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:03:18.167 INFO:teuthology.orchestra.run.smithi110.stdout:Total download size: 2.0 M 2023-09-30T02:03:18.167 INFO:teuthology.orchestra.run.smithi110.stdout:Installed size: 9.8 M 2023-09-30T02:03:18.168 INFO:teuthology.orchestra.run.smithi110.stdout:Downloading Packages: 2023-09-30T02:03:18.210 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install python3-cephfs-debuginfo 2023-09-30T02:03:18.637 INFO:teuthology.orchestra.run.smithi110.stdout:rbd-nbd-debuginfo-17.2.6-1165.g93dce5e1.el8.x86 4.3 MB/s | 2.0 MB 00:00 2023-09-30T02:03:18.638 INFO:teuthology.orchestra.run.smithi110.stdout:-------------------------------------------------------------------------------- 2023-09-30T02:03:18.638 INFO:teuthology.orchestra.run.smithi110.stdout:Total 4.3 MB/s | 2.0 MB 00:00 2023-09-30T02:03:18.638 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction check 2023-09-30T02:03:18.642 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction check succeeded. 2023-09-30T02:03:18.642 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction test 2023-09-30T02:03:18.682 INFO:teuthology.orchestra.run.smithi110.stdout:Transaction test succeeded. 2023-09-30T02:03:18.685 INFO:teuthology.orchestra.run.smithi110.stdout:Running transaction 2023-09-30T02:03:18.974 INFO:teuthology.orchestra.run.smithi110.stdout: Preparing : 1/1 2023-09-30T02:03:19.169 INFO:teuthology.orchestra.run.smithi110.stdout: Installing : rbd-nbd-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T02:03:19.198 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:13:27 ago on Sat 30 Sep 2023 01:49:52 AM UTC. 2023-09-30T02:03:19.414 INFO:teuthology.orchestra.run.smithi110.stdout: Running scriptlet: rbd-nbd-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T02:03:20.003 INFO:teuthology.orchestra.run.smithi110.stdout: Verifying : rbd-nbd-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T02:03:20.003 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:03:20.003 INFO:teuthology.orchestra.run.smithi110.stdout:Installed: 2023-09-30T02:03:20.003 INFO:teuthology.orchestra.run.smithi110.stdout: rbd-nbd-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:03:20.004 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:03:20.004 INFO:teuthology.orchestra.run.smithi110.stdout:Complete! 2023-09-30T02:03:20.116 DEBUG:teuthology.parallel:result is None 2023-09-30T02:03:20.994 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-09-30T02:03:20.995 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T02:03:20.995 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repo Size 2023-09-30T02:03:20.995 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T02:03:20.995 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2023-09-30T02:03:20.995 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cephfs-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 620 k 2023-09-30T02:03:20.995 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:03:20.995 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2023-09-30T02:03:20.995 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T02:03:20.996 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2023-09-30T02:03:20.996 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:03:20.996 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 620 k 2023-09-30T02:03:20.996 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 3.0 M 2023-09-30T02:03:20.996 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2023-09-30T02:03:21.649 INFO:teuthology.orchestra.run.smithi059.stdout:python3-cephfs-debuginfo-17.2.6-1165.g93dce5e1. 950 kB/s | 620 kB 00:00 2023-09-30T02:03:21.650 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2023-09-30T02:03:21.650 INFO:teuthology.orchestra.run.smithi059.stdout:Total 948 kB/s | 620 kB 00:00 2023-09-30T02:03:21.651 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2023-09-30T02:03:21.655 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2023-09-30T02:03:21.655 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2023-09-30T02:03:21.701 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2023-09-30T02:03:21.703 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2023-09-30T02:03:22.398 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2023-09-30T02:03:24.112 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cephfs-debuginfo-2:17.2.6-1165.g93dce5e1.el8 1/1 2023-09-30T02:03:24.409 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: python3-cephfs-debuginfo-2:17.2.6-1165.g93dce5e1.el8 1/1 2023-09-30T02:03:25.078 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cephfs-debuginfo-2:17.2.6-1165.g93dce5e1.el8 1/1 2023-09-30T02:03:25.078 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:03:25.079 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2023-09-30T02:03:25.079 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cephfs-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:03:25.079 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:03:25.079 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-09-30T02:03:25.219 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install python3-rados-debuginfo 2023-09-30T02:03:26.204 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:13:34 ago on Sat 30 Sep 2023 01:49:52 AM UTC. 2023-09-30T02:03:27.953 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-09-30T02:03:27.953 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T02:03:27.953 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repo Size 2023-09-30T02:03:27.954 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T02:03:27.954 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2023-09-30T02:03:27.954 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rados-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 721 k 2023-09-30T02:03:27.954 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:03:27.954 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2023-09-30T02:03:27.954 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T02:03:27.954 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2023-09-30T02:03:27.954 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:03:27.955 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 721 k 2023-09-30T02:03:27.955 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 3.3 M 2023-09-30T02:03:27.955 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2023-09-30T02:03:28.327 INFO:teuthology.orchestra.run.smithi059.stdout:python3-rados-debuginfo-17.2.6-1165.g93dce5e1.e 1.9 MB/s | 721 kB 00:00 2023-09-30T02:03:28.327 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2023-09-30T02:03:28.328 INFO:teuthology.orchestra.run.smithi059.stdout:Total 1.9 MB/s | 721 kB 00:00 2023-09-30T02:03:28.328 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2023-09-30T02:03:28.332 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2023-09-30T02:03:28.333 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2023-09-30T02:03:28.377 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2023-09-30T02:03:28.380 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2023-09-30T02:03:28.554 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2023-09-30T02:03:28.728 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-rados-debuginfo-2:17.2.6-1165.g93dce5e1.el8. 1/1 2023-09-30T02:03:29.025 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: python3-rados-debuginfo-2:17.2.6-1165.g93dce5e1.el8. 1/1 2023-09-30T02:03:29.716 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-rados-debuginfo-2:17.2.6-1165.g93dce5e1.el8. 1/1 2023-09-30T02:03:29.717 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:03:29.717 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2023-09-30T02:03:29.717 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rados-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:03:29.717 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:03:29.717 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-09-30T02:03:29.985 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install python3-rbd-debuginfo 2023-09-30T02:03:30.963 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:13:38 ago on Sat 30 Sep 2023 01:49:52 AM UTC. 2023-09-30T02:03:32.728 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-09-30T02:03:32.728 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T02:03:32.728 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repo Size 2023-09-30T02:03:32.728 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T02:03:32.729 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2023-09-30T02:03:32.729 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rbd-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 1.2 M 2023-09-30T02:03:32.729 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:03:32.729 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2023-09-30T02:03:32.729 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T02:03:32.729 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2023-09-30T02:03:32.729 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:03:32.730 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 1.2 M 2023-09-30T02:03:32.730 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 6.2 M 2023-09-30T02:03:32.730 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2023-09-30T02:03:33.146 INFO:teuthology.orchestra.run.smithi059.stdout:python3-rbd-debuginfo-17.2.6-1165.g93dce5e1.el8 2.8 MB/s | 1.2 MB 00:00 2023-09-30T02:03:33.147 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2023-09-30T02:03:33.147 INFO:teuthology.orchestra.run.smithi059.stdout:Total 2.8 MB/s | 1.2 MB 00:00 2023-09-30T02:03:33.148 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2023-09-30T02:03:33.152 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2023-09-30T02:03:33.152 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2023-09-30T02:03:33.198 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2023-09-30T02:03:33.202 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2023-09-30T02:03:33.439 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2023-09-30T02:03:33.630 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-rbd-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x8 1/1 2023-09-30T02:03:33.963 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: python3-rbd-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x8 1/1 2023-09-30T02:03:34.620 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-rbd-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x8 1/1 2023-09-30T02:03:34.620 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:03:34.620 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2023-09-30T02:03:34.620 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rbd-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:03:34.620 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:03:34.620 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-09-30T02:03:34.772 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install python3-rgw-debuginfo 2023-09-30T02:03:35.768 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:13:43 ago on Sat 30 Sep 2023 01:49:52 AM UTC. 2023-09-30T02:03:37.530 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-09-30T02:03:37.531 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T02:03:37.531 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repo Size 2023-09-30T02:03:37.531 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T02:03:37.531 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2023-09-30T02:03:37.531 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rgw-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 318 k 2023-09-30T02:03:37.532 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:03:37.532 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2023-09-30T02:03:37.532 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T02:03:37.532 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2023-09-30T02:03:37.532 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:03:37.532 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 318 k 2023-09-30T02:03:37.533 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 1.3 M 2023-09-30T02:03:37.533 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2023-09-30T02:03:37.748 INFO:teuthology.orchestra.run.smithi059.stdout:python3-rgw-debuginfo-17.2.6-1165.g93dce5e1.el8 1.4 MB/s | 318 kB 00:00 2023-09-30T02:03:37.749 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2023-09-30T02:03:37.749 INFO:teuthology.orchestra.run.smithi059.stdout:Total 1.4 MB/s | 318 kB 00:00 2023-09-30T02:03:37.750 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2023-09-30T02:03:37.754 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2023-09-30T02:03:37.754 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2023-09-30T02:03:37.797 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2023-09-30T02:03:37.800 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2023-09-30T02:03:37.923 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2023-09-30T02:03:38.152 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-rgw-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x8 1/1 2023-09-30T02:03:38.438 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: python3-rgw-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x8 1/1 2023-09-30T02:03:39.097 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-rgw-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x8 1/1 2023-09-30T02:03:39.097 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:03:39.097 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2023-09-30T02:03:39.097 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rgw-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:03:39.097 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:03:39.097 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-09-30T02:03:39.217 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install rbd-fuse-debuginfo 2023-09-30T02:03:40.209 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:13:48 ago on Sat 30 Sep 2023 01:49:52 AM UTC. 2023-09-30T02:03:41.981 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-09-30T02:03:41.981 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T02:03:41.981 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repo Size 2023-09-30T02:03:41.981 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T02:03:41.981 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2023-09-30T02:03:41.982 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-fuse-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 1.2 M 2023-09-30T02:03:41.982 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:03:41.982 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2023-09-30T02:03:41.982 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T02:03:41.982 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2023-09-30T02:03:41.982 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:03:41.983 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 1.2 M 2023-09-30T02:03:41.983 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 5.6 M 2023-09-30T02:03:41.983 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2023-09-30T02:03:42.442 INFO:teuthology.orchestra.run.smithi059.stdout:rbd-fuse-debuginfo-17.2.6-1165.g93dce5e1.el8.x8 2.6 MB/s | 1.2 MB 00:00 2023-09-30T02:03:42.443 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2023-09-30T02:03:42.444 INFO:teuthology.orchestra.run.smithi059.stdout:Total 2.6 MB/s | 1.2 MB 00:00 2023-09-30T02:03:42.444 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2023-09-30T02:03:42.449 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2023-09-30T02:03:42.449 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2023-09-30T02:03:42.495 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2023-09-30T02:03:42.498 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2023-09-30T02:03:43.171 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2023-09-30T02:03:43.610 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : rbd-fuse-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_6 1/1 2023-09-30T02:03:43.936 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: rbd-fuse-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_6 1/1 2023-09-30T02:03:44.633 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : rbd-fuse-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_6 1/1 2023-09-30T02:03:44.633 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:03:44.633 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2023-09-30T02:03:44.633 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-fuse-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:03:44.634 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:03:44.634 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-09-30T02:03:44.753 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install rbd-mirror-debuginfo 2023-09-30T02:03:45.725 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:13:53 ago on Sat 30 Sep 2023 01:49:52 AM UTC. 2023-09-30T02:03:47.481 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-09-30T02:03:47.482 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T02:03:47.482 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repo Size 2023-09-30T02:03:47.482 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T02:03:47.482 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2023-09-30T02:03:47.482 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-mirror-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 72 M 2023-09-30T02:03:47.482 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:03:47.482 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2023-09-30T02:03:47.482 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T02:03:47.482 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2023-09-30T02:03:47.483 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:03:47.483 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 72 M 2023-09-30T02:03:47.483 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 480 M 2023-09-30T02:03:47.483 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2023-09-30T02:03:59.652 INFO:teuthology.orchestra.run.smithi059.stdout:rbd-mirror-debuginfo-17.2.6-1165.g93dce5e1.el8. 5.9 MB/s | 72 MB 00:12 2023-09-30T02:03:59.653 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2023-09-30T02:03:59.653 INFO:teuthology.orchestra.run.smithi059.stdout:Total 5.9 MB/s | 72 MB 00:12 2023-09-30T02:03:59.654 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2023-09-30T02:03:59.658 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2023-09-30T02:03:59.658 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2023-09-30T02:03:59.984 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2023-09-30T02:03:59.987 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2023-09-30T02:04:09.463 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2023-09-30T02:04:09.648 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : rbd-mirror-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86 1/1 2023-09-30T02:04:09.964 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: rbd-mirror-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86 1/1 2023-09-30T02:04:10.631 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : rbd-mirror-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86 1/1 2023-09-30T02:04:10.631 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:04:10.631 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2023-09-30T02:04:10.631 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-mirror-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:04:10.632 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:04:10.632 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-09-30T02:04:10.762 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install rbd-nbd-debuginfo 2023-09-30T02:04:11.754 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:14:19 ago on Sat 30 Sep 2023 01:49:52 AM UTC. 2023-09-30T02:04:13.522 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-09-30T02:04:13.522 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T02:04:13.522 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repo Size 2023-09-30T02:04:13.523 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T02:04:13.523 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2023-09-30T02:04:13.523 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-nbd-debuginfo x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 2.0 M 2023-09-30T02:04:13.523 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:04:13.523 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2023-09-30T02:04:13.523 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-09-30T02:04:13.523 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2023-09-30T02:04:13.523 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:04:13.524 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 2.0 M 2023-09-30T02:04:13.524 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 9.8 M 2023-09-30T02:04:13.524 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2023-09-30T02:04:13.861 INFO:teuthology.orchestra.run.smithi059.stdout:rbd-nbd-debuginfo-17.2.6-1165.g93dce5e1.el8.x86 6.0 MB/s | 2.0 MB 00:00 2023-09-30T02:04:13.862 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2023-09-30T02:04:13.862 INFO:teuthology.orchestra.run.smithi059.stdout:Total 5.9 MB/s | 2.0 MB 00:00 2023-09-30T02:04:13.863 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2023-09-30T02:04:13.867 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2023-09-30T02:04:13.867 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2023-09-30T02:04:13.916 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2023-09-30T02:04:13.920 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2023-09-30T02:04:14.242 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2023-09-30T02:04:14.439 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : rbd-nbd-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T02:04:14.803 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: rbd-nbd-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T02:04:15.479 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : rbd-nbd-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T02:04:15.479 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:04:15.479 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2023-09-30T02:04:15.479 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-nbd-debuginfo-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T02:04:15.479 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:04:15.479 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-09-30T02:04:15.604 DEBUG:teuthology.parallel:result is None 2023-09-30T02:04:15.605 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-30T02:04:15.755 DEBUG:teuthology.orchestra.run.smithi059:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-09-30T02:04:15.783 INFO:teuthology.orchestra.run.smithi059.stdout:17.2.6-1165.g93dce5e1.el8 2023-09-30T02:04:15.783 INFO:teuthology.packaging:The installed version of ceph is 17.2.6-1165.g93dce5e1.el8 2023-09-30T02:04:15.784 INFO:teuthology.task.install:The correct ceph version 17.2.6-1165.g93dce5e1 is installed. 2023-09-30T02:04:15.784 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-30T02:04:15.935 DEBUG:teuthology.orchestra.run.smithi110:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-09-30T02:04:15.962 INFO:teuthology.orchestra.run.smithi110.stdout:17.2.6-1165.g93dce5e1.el8 2023-09-30T02:04:15.962 INFO:teuthology.packaging:The installed version of ceph is 17.2.6-1165.g93dce5e1.el8 2023-09-30T02:04:15.962 INFO:teuthology.task.install:The correct ceph version 17.2.6-1165.g93dce5e1 is installed. 2023-09-30T02:04:15.963 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-09-30T02:04:15.963 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2023-09-30T02:04:15.963 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-09-30T02:04:15.998 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-09-30T02:04:15.998 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-09-30T02:04:16.035 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-09-30T02:04:16.035 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2023-09-30T02:04:16.035 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/usr/bin/daemon-helper 2023-09-30T02:04:16.073 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-09-30T02:04:16.148 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-09-30T02:04:16.148 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/usr/bin/daemon-helper 2023-09-30T02:04:16.180 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-09-30T02:04:16.251 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-09-30T02:04:16.251 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2023-09-30T02:04:16.252 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/usr/bin/adjust-ulimits 2023-09-30T02:04:16.286 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-09-30T02:04:16.361 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-09-30T02:04:16.361 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/usr/bin/adjust-ulimits 2023-09-30T02:04:16.392 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-09-30T02:04:16.461 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-09-30T02:04:16.462 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2023-09-30T02:04:16.462 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/usr/bin/stdin-killer 2023-09-30T02:04:16.496 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-09-30T02:04:16.571 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-09-30T02:04:16.571 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/usr/bin/stdin-killer 2023-09-30T02:04:16.601 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-09-30T02:04:16.670 INFO:teuthology.run_tasks:Running task ceph... 2023-09-30T02:04:16.734 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-09-30T02:04:16.734 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 777 /var/log/ceph 2023-09-30T02:04:16.737 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 777 /var/log/ceph 2023-09-30T02:04:16.769 INFO:tasks.ceph:Disabling ceph logrotate... 2023-09-30T02:04:16.769 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm -f -- /etc/logrotate.d/ceph 2023-09-30T02:04:16.812 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -f -- /etc/logrotate.d/ceph 2023-09-30T02:04:16.844 INFO:tasks.ceph:Creating extra log directories... 2023-09-30T02:04:16.844 DEBUG:teuthology.orchestra.run.smithi059:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-09-30T02:04:16.888 DEBUG:teuthology.orchestra.run.smithi110:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-09-30T02:04:16.930 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-09-30T02:04:16.930 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-09-30T02:04:16.931 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7407087', 'branch': 'wip-yuri4-testing-2023-09-29-0928-quincy', 'description': 'rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados tasks/rados_api_tests validater/valgrind}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7407087', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '8.stream', 'overrides': {'admin_socket': {'branch': 'wip-yuri4-testing-2023-09-29-0928-quincy'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri4-testing-2023-09-29-0928-quincy', 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 9810, 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-yuri4-testing-2023-09-29-0928-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481', 'targets': {'smithi059.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEIBRHJ0E9yId33pAQ6zaKUIboMamQ5eavNMIuT0xJxpdZT67asfI4+/iMT8fjQz+Bty8+qBIkRMYPZtedOmIbM=', 'smithi110.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEoQZBS3c/f0LpnsOM7gYX1QUT6fO3qGKPBNFX4nLXfv7pJqFWKHqO/grsNaWh8iLnuAcpGMUTMEOtrac/4nX/Y='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'thrashosds': {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200}}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '54e62bcbac4e53d9685e08328b790d3b20d71cae', 'timestamp': '2023-09-29_19:38:56', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-09-30T02:04:16.931 DEBUG:teuthology.orchestra.run.smithi059:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-09-30T02:04:16.963 DEBUG:teuthology.orchestra.run.smithi110:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-09-30T02:04:16.983 DEBUG:teuthology.orchestra.run.smithi059:> sudo install -d -m0777 -- /var/run/ceph 2023-09-30T02:04:17.027 DEBUG:teuthology.orchestra.run.smithi110:> sudo install -d -m0777 -- /var/run/ceph 2023-09-30T02:04:17.059 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2023-09-30T02:04:17.059 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/scratch_devs of=/dev/stdout 2023-09-30T02:04:17.123 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-09-30T02:04:17.123 DEBUG:teuthology.orchestra.run.smithi059:> stat /dev/vg_nvme/lv_1 2023-09-30T02:04:17.186 INFO:teuthology.orchestra.run.smithi059.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-09-30T02:04:17.187 INFO:teuthology.orchestra.run.smithi059.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T02:04:17.187 INFO:teuthology.orchestra.run.smithi059.stdout:Device: 6h/6d Inode: 390019 Links: 1 2023-09-30T02:04:17.187 INFO:teuthology.orchestra.run.smithi059.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T02:04:17.187 INFO:teuthology.orchestra.run.smithi059.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T02:04:17.187 INFO:teuthology.orchestra.run.smithi059.stdout:Access: 2023-09-30 01:53:03.796671627 +0000 2023-09-30T02:04:17.187 INFO:teuthology.orchestra.run.smithi059.stdout:Modify: 2023-09-30 01:48:41.759204867 +0000 2023-09-30T02:04:17.187 INFO:teuthology.orchestra.run.smithi059.stdout:Change: 2023-09-30 01:48:41.759204867 +0000 2023-09-30T02:04:17.187 INFO:teuthology.orchestra.run.smithi059.stdout: Birth: - 2023-09-30T02:04:17.187 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-09-30T02:04:17.260 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records in 2023-09-30T02:04:17.260 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records out 2023-09-30T02:04:17.260 INFO:teuthology.orchestra.run.smithi059.stderr:512 bytes copied, 0.000229054 s, 2.2 MB/s 2023-09-30T02:04:17.261 DEBUG:teuthology.orchestra.run.smithi059:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-09-30T02:04:17.326 DEBUG:teuthology.orchestra.run.smithi059:> stat /dev/vg_nvme/lv_2 2023-09-30T02:04:17.389 INFO:teuthology.orchestra.run.smithi059.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-09-30T02:04:17.389 INFO:teuthology.orchestra.run.smithi059.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T02:04:17.389 INFO:teuthology.orchestra.run.smithi059.stdout:Device: 6h/6d Inode: 390099 Links: 1 2023-09-30T02:04:17.389 INFO:teuthology.orchestra.run.smithi059.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T02:04:17.389 INFO:teuthology.orchestra.run.smithi059.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T02:04:17.389 INFO:teuthology.orchestra.run.smithi059.stdout:Access: 2023-09-30 01:53:03.796671627 +0000 2023-09-30T02:04:17.389 INFO:teuthology.orchestra.run.smithi059.stdout:Modify: 2023-09-30 01:48:42.332198008 +0000 2023-09-30T02:04:17.390 INFO:teuthology.orchestra.run.smithi059.stdout:Change: 2023-09-30 01:48:42.332198008 +0000 2023-09-30T02:04:17.390 INFO:teuthology.orchestra.run.smithi059.stdout: Birth: - 2023-09-30T02:04:17.390 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-09-30T02:04:17.463 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records in 2023-09-30T02:04:17.463 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records out 2023-09-30T02:04:17.463 INFO:teuthology.orchestra.run.smithi059.stderr:512 bytes copied, 0.000234949 s, 2.2 MB/s 2023-09-30T02:04:17.464 DEBUG:teuthology.orchestra.run.smithi059:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-09-30T02:04:17.529 DEBUG:teuthology.orchestra.run.smithi059:> stat /dev/vg_nvme/lv_3 2023-09-30T02:04:17.593 INFO:teuthology.orchestra.run.smithi059.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-09-30T02:04:17.593 INFO:teuthology.orchestra.run.smithi059.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T02:04:17.593 INFO:teuthology.orchestra.run.smithi059.stdout:Device: 6h/6d Inode: 386933 Links: 1 2023-09-30T02:04:17.593 INFO:teuthology.orchestra.run.smithi059.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T02:04:17.593 INFO:teuthology.orchestra.run.smithi059.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T02:04:17.593 INFO:teuthology.orchestra.run.smithi059.stdout:Access: 2023-09-30 01:53:03.796671627 +0000 2023-09-30T02:04:17.593 INFO:teuthology.orchestra.run.smithi059.stdout:Modify: 2023-09-30 01:48:42.938190755 +0000 2023-09-30T02:04:17.594 INFO:teuthology.orchestra.run.smithi059.stdout:Change: 2023-09-30 01:48:42.938190755 +0000 2023-09-30T02:04:17.594 INFO:teuthology.orchestra.run.smithi059.stdout: Birth: - 2023-09-30T02:04:17.594 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-09-30T02:04:17.666 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records in 2023-09-30T02:04:17.667 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records out 2023-09-30T02:04:17.667 INFO:teuthology.orchestra.run.smithi059.stderr:512 bytes copied, 0.000234892 s, 2.2 MB/s 2023-09-30T02:04:17.668 DEBUG:teuthology.orchestra.run.smithi059:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-09-30T02:04:17.733 DEBUG:teuthology.orchestra.run.smithi059:> stat /dev/vg_nvme/lv_4 2023-09-30T02:04:17.796 INFO:teuthology.orchestra.run.smithi059.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-09-30T02:04:17.796 INFO:teuthology.orchestra.run.smithi059.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T02:04:17.796 INFO:teuthology.orchestra.run.smithi059.stdout:Device: 6h/6d Inode: 391591 Links: 1 2023-09-30T02:04:17.797 INFO:teuthology.orchestra.run.smithi059.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T02:04:17.797 INFO:teuthology.orchestra.run.smithi059.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T02:04:17.797 INFO:teuthology.orchestra.run.smithi059.stdout:Access: 2023-09-30 01:53:03.796671627 +0000 2023-09-30T02:04:17.797 INFO:teuthology.orchestra.run.smithi059.stdout:Modify: 2023-09-30 01:48:43.506183956 +0000 2023-09-30T02:04:17.797 INFO:teuthology.orchestra.run.smithi059.stdout:Change: 2023-09-30 01:48:43.506183956 +0000 2023-09-30T02:04:17.797 INFO:teuthology.orchestra.run.smithi059.stdout: Birth: - 2023-09-30T02:04:17.797 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-09-30T02:04:17.869 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records in 2023-09-30T02:04:17.870 INFO:teuthology.orchestra.run.smithi059.stderr:1+0 records out 2023-09-30T02:04:17.870 INFO:teuthology.orchestra.run.smithi059.stderr:512 bytes copied, 0.000341106 s, 1.5 MB/s 2023-09-30T02:04:17.871 DEBUG:teuthology.orchestra.run.smithi059:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-09-30T02:04:17.936 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-09-30T02:04:17.936 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-09-30T02:04:17.936 DEBUG:teuthology.orchestra.run.smithi110:> dd if=/scratch_devs of=/dev/stdout 2023-09-30T02:04:17.956 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-09-30T02:04:17.956 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_1 2023-09-30T02:04:18.016 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-09-30T02:04:18.016 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T02:04:18.016 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 6h/6d Inode: 394298 Links: 1 2023-09-30T02:04:18.016 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T02:04:18.016 INFO:teuthology.orchestra.run.smithi110.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T02:04:18.017 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2023-09-30 01:52:53.201058502 +0000 2023-09-30T02:04:18.017 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2023-09-30 01:47:56.347170822 +0000 2023-09-30T02:04:18.017 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2023-09-30 01:47:56.347170822 +0000 2023-09-30T02:04:18.017 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: - 2023-09-30T02:04:18.017 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-09-30T02:04:18.084 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2023-09-30T02:04:18.085 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2023-09-30T02:04:18.085 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 9.3841e-05 s, 5.5 MB/s 2023-09-30T02:04:18.086 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-09-30T02:04:18.146 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_2 2023-09-30T02:04:18.206 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-09-30T02:04:18.206 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T02:04:18.206 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 6h/6d Inode: 380707 Links: 1 2023-09-30T02:04:18.206 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T02:04:18.206 INFO:teuthology.orchestra.run.smithi110.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T02:04:18.206 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2023-09-30 01:52:53.201058502 +0000 2023-09-30T02:04:18.206 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2023-09-30 01:47:56.877162291 +0000 2023-09-30T02:04:18.206 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2023-09-30 01:47:56.877162291 +0000 2023-09-30T02:04:18.206 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: - 2023-09-30T02:04:18.207 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-09-30T02:04:18.273 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2023-09-30T02:04:18.273 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2023-09-30T02:04:18.273 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 0.000131034 s, 3.9 MB/s 2023-09-30T02:04:18.274 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-09-30T02:04:18.334 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_3 2023-09-30T02:04:18.393 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-09-30T02:04:18.393 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T02:04:18.393 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 6h/6d Inode: 396392 Links: 1 2023-09-30T02:04:18.393 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T02:04:18.393 INFO:teuthology.orchestra.run.smithi110.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T02:04:18.393 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2023-09-30 01:52:53.202058486 +0000 2023-09-30T02:04:18.394 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2023-09-30 01:47:57.374154292 +0000 2023-09-30T02:04:18.394 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2023-09-30 01:47:57.374154292 +0000 2023-09-30T02:04:18.394 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: - 2023-09-30T02:04:18.394 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-09-30T02:04:18.459 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2023-09-30T02:04:18.459 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2023-09-30T02:04:18.459 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 0.000120622 s, 4.2 MB/s 2023-09-30T02:04:18.460 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-09-30T02:04:18.519 DEBUG:teuthology.orchestra.run.smithi110:> stat /dev/vg_nvme/lv_4 2023-09-30T02:04:18.577 INFO:teuthology.orchestra.run.smithi110.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-09-30T02:04:18.578 INFO:teuthology.orchestra.run.smithi110.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T02:04:18.578 INFO:teuthology.orchestra.run.smithi110.stdout:Device: 6h/6d Inode: 380853 Links: 1 2023-09-30T02:04:18.578 INFO:teuthology.orchestra.run.smithi110.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T02:04:18.578 INFO:teuthology.orchestra.run.smithi110.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T02:04:18.578 INFO:teuthology.orchestra.run.smithi110.stdout:Access: 2023-09-30 01:52:53.202058486 +0000 2023-09-30T02:04:18.578 INFO:teuthology.orchestra.run.smithi110.stdout:Modify: 2023-09-30 01:47:57.891145971 +0000 2023-09-30T02:04:18.578 INFO:teuthology.orchestra.run.smithi110.stdout:Change: 2023-09-30 01:47:57.891145971 +0000 2023-09-30T02:04:18.578 INFO:teuthology.orchestra.run.smithi110.stdout: Birth: - 2023-09-30T02:04:18.579 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-09-30T02:04:18.643 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records in 2023-09-30T02:04:18.643 INFO:teuthology.orchestra.run.smithi110.stderr:1+0 records out 2023-09-30T02:04:18.643 INFO:teuthology.orchestra.run.smithi110.stderr:512 bytes copied, 9.8015e-05 s, 5.2 MB/s 2023-09-30T02:04:18.644 DEBUG:teuthology.orchestra.run.smithi110:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-09-30T02:04:18.703 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-09-30T02:04:18.703 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi059.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi110.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-09-30T02:04:18.703 INFO:tasks.ceph:Generating config... 2023-09-30T02:04:18.704 INFO:tasks.ceph:[client] debug monc = 20 2023-09-30T02:04:18.705 INFO:tasks.ceph:[client] debug ms = 1 2023-09-30T02:04:18.705 INFO:tasks.ceph:[client] debug objecter = 20 2023-09-30T02:04:18.705 INFO:tasks.ceph:[client] debug rados = 20 2023-09-30T02:04:18.705 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-09-30T02:04:18.705 INFO:tasks.ceph:[global] mon election default strategy = 3 2023-09-30T02:04:18.705 INFO:tasks.ceph:[global] ms bind msgr1 = False 2023-09-30T02:04:18.705 INFO:tasks.ceph:[global] ms bind msgr2 = True 2023-09-30T02:04:18.705 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2023-09-30T02:04:18.705 INFO:tasks.ceph:[global] ms type = async 2023-09-30T02:04:18.705 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2023-09-30T02:04:18.706 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-09-30T02:04:18.706 INFO:tasks.ceph:[mgr] debug ms = 1 2023-09-30T02:04:18.706 INFO:tasks.ceph:[mon] debug mon = 20 2023-09-30T02:04:18.706 INFO:tasks.ceph:[mon] debug ms = 1 2023-09-30T02:04:18.706 INFO:tasks.ceph:[mon] debug paxos = 20 2023-09-30T02:04:18.706 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2023-09-30T02:04:18.706 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2023-09-30T02:04:18.706 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2023-09-30T02:04:18.706 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2023-09-30T02:04:18.706 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2023-09-30T02:04:18.706 INFO:tasks.ceph:[mon] mon scrub interval = 300 2023-09-30T02:04:18.706 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2023-09-30T02:04:18.707 INFO:tasks.ceph:[mon] paxos service trim min = 10 2023-09-30T02:04:18.707 INFO:tasks.ceph:[osd] bdev async discard = True 2023-09-30T02:04:18.707 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-09-30T02:04:18.707 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-09-30T02:04:18.707 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-09-30T02:04:18.707 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2023-09-30T02:04:18.707 INFO:tasks.ceph:[osd] debug bluefs = 1 2023-09-30T02:04:18.707 INFO:tasks.ceph:[osd] debug bluestore = 1 2023-09-30T02:04:18.707 INFO:tasks.ceph:[osd] debug monc = 20 2023-09-30T02:04:18.707 INFO:tasks.ceph:[osd] debug ms = 1 2023-09-30T02:04:18.707 INFO:tasks.ceph:[osd] debug osd = 20 2023-09-30T02:04:18.707 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-09-30T02:04:18.708 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-09-30T02:04:18.708 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-09-30T02:04:18.708 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-09-30T02:04:18.708 INFO:tasks.ceph:[osd] osd class default list = * 2023-09-30T02:04:18.708 INFO:tasks.ceph:[osd] osd class load list = * 2023-09-30T02:04:18.708 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2023-09-30T02:04:18.708 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2023-09-30T02:04:18.708 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2023-09-30T02:04:18.708 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-09-30T02:04:18.708 INFO:tasks.ceph:[osd] osd fast shutdown = False 2023-09-30T02:04:18.708 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2023-09-30T02:04:18.708 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2023-09-30T02:04:18.708 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2023-09-30T02:04:18.709 INFO:tasks.ceph:[osd] osd memory target = 1610612736 2023-09-30T02:04:18.709 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-09-30T02:04:18.709 INFO:tasks.ceph:[osd] osd op queue = debug_random 2023-09-30T02:04:18.709 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2023-09-30T02:04:18.709 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2023-09-30T02:04:18.709 INFO:tasks.ceph:Setting up mon.a... 2023-09-30T02:04:18.709 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-09-30T02:04:18.772 INFO:teuthology.orchestra.run.smithi059.stdout:creating /etc/ceph/ceph.keyring 2023-09-30T02:04:18.775 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-09-30T02:04:18.825 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-09-30T02:04:18.858 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.59'), ('mon.c', '[v2:172.21.15.59:3301,v1:172.21.15.59:6790]'), ('mon.b', '172.21.15.110')] 2023-09-30T02:04:18.858 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.59,[v2:172.21.15.59:3301,v1:172.21.15.59:6790],172.21.15.110', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2023-09-30T02:04:18.859 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2023-09-30T02:04:18.859 DEBUG:teuthology.orchestra.run.smithi059:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-09-30T02:04:18.921 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.59 --addv c '[v2:172.21.15.59:3301,v1:172.21.15.59:6790]' --add b 172.21.15.110 --print /home/ubuntu/cephtest/ceph.monmap 2023-09-30T02:04:19.016 INFO:teuthology.orchestra.run.smithi059.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-09-30T02:04:19.017 INFO:teuthology.orchestra.run.smithi059.stdout:monmaptool: generated fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:04:19.017 INFO:teuthology.orchestra.run.smithi059.stdout:setting min_mon_release = octopus 2023-09-30T02:04:19.017 INFO:teuthology.orchestra.run.smithi059.stdout:epoch 0 2023-09-30T02:04:19.017 INFO:teuthology.orchestra.run.smithi059.stdout:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:04:19.017 INFO:teuthology.orchestra.run.smithi059.stdout:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:04:19.017 INFO:teuthology.orchestra.run.smithi059.stdout:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:04:19.017 INFO:teuthology.orchestra.run.smithi059.stdout:min_mon_release 15 (octopus) 2023-09-30T02:04:19.017 INFO:teuthology.orchestra.run.smithi059.stdout:election_strategy: 3 2023-09-30T02:04:19.017 INFO:teuthology.orchestra.run.smithi059.stdout:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:04:19.017 INFO:teuthology.orchestra.run.smithi059.stdout:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:04:19.017 INFO:teuthology.orchestra.run.smithi059.stdout:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:04:19.018 INFO:teuthology.orchestra.run.smithi059.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-09-30T02:04:19.019 DEBUG:teuthology.orchestra.run.smithi059:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-09-30T02:04:19.040 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 623052cb-4f82-4b2b-b72b-3ed5be614613... 2023-09-30T02:04:19.041 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-09-30T02:04:19.083 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-09-30T02:04:19.128 INFO:teuthology.orchestra.run.smithi110.stdout:[global] 2023-09-30T02:04:19.128 INFO:teuthology.orchestra.run.smithi110.stdout: chdir = "" 2023-09-30T02:04:19.128 INFO:teuthology.orchestra.run.smithi110.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-09-30T02:04:19.128 INFO:teuthology.orchestra.run.smithi110.stdout: auth supported = cephx 2023-09-30T02:04:19.128 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:04:19.129 INFO:teuthology.orchestra.run.smithi110.stdout: filestore xattr use omap = true 2023-09-30T02:04:19.129 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:04:19.129 INFO:teuthology.orchestra.run.smithi110.stdout: mon clock drift allowed = 1.000 2023-09-30T02:04:19.129 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:04:19.129 INFO:teuthology.orchestra.run.smithi110.stdout: osd crush chooseleaf type = 0 2023-09-30T02:04:19.129 INFO:teuthology.orchestra.run.smithi110.stdout: auth debug = true 2023-09-30T02:04:19.129 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:04:19.129 INFO:teuthology.orchestra.run.smithi110.stdout: ms die on old message = true 2023-09-30T02:04:19.129 INFO:teuthology.orchestra.run.smithi110.stdout: ms die on bug = true 2023-09-30T02:04:19.129 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:04:19.130 INFO:teuthology.orchestra.run.smithi110.stdout: mon max pg per osd = 10000 # >= luminous 2023-09-30T02:04:19.130 INFO:teuthology.orchestra.run.smithi110.stdout: mon pg warn max object skew = 0 2023-09-30T02:04:19.130 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:04:19.130 INFO:teuthology.orchestra.run.smithi110.stdout: # disable pg_autoscaler by default for new pools 2023-09-30T02:04:19.130 INFO:teuthology.orchestra.run.smithi110.stdout: osd_pool_default_pg_autoscale_mode = off 2023-09-30T02:04:19.130 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:04:19.130 INFO:teuthology.orchestra.run.smithi110.stdout: osd pool default size = 2 2023-09-30T02:04:19.130 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:04:19.130 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd allow primary affinity = true 2023-09-30T02:04:19.130 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd allow pg remap = true 2023-09-30T02:04:19.130 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on legacy crush tunables = false 2023-09-30T02:04:19.130 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on crush straw calc version zero = false 2023-09-30T02:04:19.131 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on no sortbitwise = false 2023-09-30T02:04:19.131 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on osd down out interval zero = false 2023-09-30T02:04:19.131 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on too few osds = false 2023-09-30T02:04:19.131 INFO:teuthology.orchestra.run.smithi110.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-09-30T02:04:19.131 INFO:teuthology.orchestra.run.smithi110.stdout: mon_warn_on_pool_no_redundancy = false 2023-09-30T02:04:19.131 INFO:teuthology.orchestra.run.smithi110.stdout: mon_allow_pool_size_one = true 2023-09-30T02:04:19.131 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:04:19.131 INFO:teuthology.orchestra.run.smithi110.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-09-30T02:04:19.131 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:04:19.131 INFO:teuthology.orchestra.run.smithi110.stdout: osd default data pool replay window = 5 2023-09-30T02:04:19.132 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:04:19.132 INFO:teuthology.orchestra.run.smithi110.stdout: mon allow pool delete = true 2023-09-30T02:04:19.132 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:04:19.132 INFO:teuthology.orchestra.run.smithi110.stdout: mon cluster log file level = debug 2023-09-30T02:04:19.132 INFO:teuthology.orchestra.run.smithi110.stdout: debug asserts on shutdown = true 2023-09-30T02:04:19.132 INFO:teuthology.orchestra.run.smithi110.stdout: mon health detail to clog = false 2023-09-30T02:04:19.133 INFO:teuthology.orchestra.run.smithi110.stdout: mon host = "172.21.15.59,[v2:172.21.15.59:3301,v1:172.21.15.59:6790],172.21.15.110" 2023-09-30T02:04:19.133 INFO:teuthology.orchestra.run.smithi110.stdout: mon client directed command retry = 5 2023-09-30T02:04:19.133 INFO:teuthology.orchestra.run.smithi110.stdout: mon election default strategy = 3 2023-09-30T02:04:19.133 INFO:teuthology.orchestra.run.smithi110.stdout: ms bind msgr1 = False 2023-09-30T02:04:19.133 INFO:teuthology.orchestra.run.smithi110.stdout: ms bind msgr2 = True 2023-09-30T02:04:19.133 INFO:teuthology.orchestra.run.smithi110.stdout: ms inject socket failures = 5000 2023-09-30T02:04:19.133 INFO:teuthology.orchestra.run.smithi110.stdout: ms type = async 2023-09-30T02:04:19.134 INFO:teuthology.orchestra.run.smithi110.stdout: osd heartbeat grace = 80 2023-09-30T02:04:19.134 INFO:teuthology.orchestra.run.smithi110.stdout: fsid = 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:04:19.134 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:04:19.134 INFO:teuthology.orchestra.run.smithi110.stdout:[osd] 2023-09-30T02:04:19.134 INFO:teuthology.orchestra.run.smithi110.stdout: osd journal size = 100 2023-09-30T02:04:19.134 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:04:19.135 INFO:teuthology.orchestra.run.smithi110.stdout: osd scrub load threshold = 5.0 2023-09-30T02:04:19.135 INFO:teuthology.orchestra.run.smithi110.stdout: osd scrub max interval = 600 2023-09-30T02:04:19.135 INFO:teuthology.orchestra.run.smithi110.stdout: osd mclock profile = high_recovery_ops 2023-09-30T02:04:19.135 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:04:19.135 INFO:teuthology.orchestra.run.smithi110.stdout: osd recover clone overlap = true 2023-09-30T02:04:19.135 INFO:teuthology.orchestra.run.smithi110.stdout: osd recovery max chunk = 1048576 2023-09-30T02:04:19.135 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:04:19.136 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug shutdown = true 2023-09-30T02:04:19.136 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug op order = true 2023-09-30T02:04:19.136 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug verify stray on activate = true 2023-09-30T02:04:19.136 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:04:19.136 INFO:teuthology.orchestra.run.smithi110.stdout: osd open classes on start = true 2023-09-30T02:04:19.136 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug pg log writeout = true 2023-09-30T02:04:19.137 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:04:19.137 INFO:teuthology.orchestra.run.smithi110.stdout: osd deep scrub update digest min age = 30 2023-09-30T02:04:19.137 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:04:19.137 INFO:teuthology.orchestra.run.smithi110.stdout: osd map max advance = 10 2023-09-30T02:04:19.137 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:04:19.137 INFO:teuthology.orchestra.run.smithi110.stdout: journal zero on create = true 2023-09-30T02:04:19.137 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:04:19.138 INFO:teuthology.orchestra.run.smithi110.stdout: filestore ondisk finisher threads = 3 2023-09-30T02:04:19.138 INFO:teuthology.orchestra.run.smithi110.stdout: filestore apply finisher threads = 3 2023-09-30T02:04:19.138 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:04:19.138 INFO:teuthology.orchestra.run.smithi110.stdout: bdev debug aio = true 2023-09-30T02:04:19.138 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug misdirected ops = true 2023-09-30T02:04:19.138 INFO:teuthology.orchestra.run.smithi110.stdout: bdev async discard = True 2023-09-30T02:04:19.138 INFO:teuthology.orchestra.run.smithi110.stdout: bdev enable discard = True 2023-09-30T02:04:19.139 INFO:teuthology.orchestra.run.smithi110.stdout: bluestore block size = 96636764160 2023-09-30T02:04:19.139 INFO:teuthology.orchestra.run.smithi110.stdout: bluestore fsck on mount = True 2023-09-30T02:04:19.139 INFO:teuthology.orchestra.run.smithi110.stdout: bluestore zero block detection = True 2023-09-30T02:04:19.139 INFO:teuthology.orchestra.run.smithi110.stdout: debug bluefs = 1 2023-09-30T02:04:19.139 INFO:teuthology.orchestra.run.smithi110.stdout: debug bluestore = 1 2023-09-30T02:04:19.139 INFO:teuthology.orchestra.run.smithi110.stdout: debug monc = 20 2023-09-30T02:04:19.140 INFO:teuthology.orchestra.run.smithi110.stdout: debug ms = 1 2023-09-30T02:04:19.140 INFO:teuthology.orchestra.run.smithi110.stdout: debug osd = 20 2023-09-30T02:04:19.140 INFO:teuthology.orchestra.run.smithi110.stdout: debug rocksdb = 10 2023-09-30T02:04:19.140 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd backfillfull_ratio = 0.85 2023-09-30T02:04:19.140 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd full ratio = 0.9 2023-09-30T02:04:19.140 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd nearfull ratio = 0.8 2023-09-30T02:04:19.141 INFO:teuthology.orchestra.run.smithi110.stdout: osd class default list = * 2023-09-30T02:04:19.141 INFO:teuthology.orchestra.run.smithi110.stdout: osd class load list = * 2023-09-30T02:04:19.141 INFO:teuthology.orchestra.run.smithi110.stdout: osd client watch timeout = 120 2023-09-30T02:04:19.141 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug verify cached snaps = True 2023-09-30T02:04:19.141 INFO:teuthology.orchestra.run.smithi110.stdout: osd debug verify missing on start = True 2023-09-30T02:04:19.141 INFO:teuthology.orchestra.run.smithi110.stdout: osd failsafe full ratio = 0.95 2023-09-30T02:04:19.142 INFO:teuthology.orchestra.run.smithi110.stdout: osd fast shutdown = False 2023-09-30T02:04:19.142 INFO:teuthology.orchestra.run.smithi110.stdout: osd max markdown count = 1000 2023-09-30T02:04:19.142 INFO:teuthology.orchestra.run.smithi110.stdout: osd mclock override recovery settings = True 2023-09-30T02:04:19.142 INFO:teuthology.orchestra.run.smithi110.stdout: osd memory target = 1610612736 2023-09-30T02:04:19.142 INFO:teuthology.orchestra.run.smithi110.stdout: osd objectstore = bluestore 2023-09-30T02:04:19.143 INFO:teuthology.orchestra.run.smithi110.stdout: osd op queue = debug_random 2023-09-30T02:04:19.143 INFO:teuthology.orchestra.run.smithi110.stdout: osd op queue cut off = debug_random 2023-09-30T02:04:19.143 INFO:teuthology.orchestra.run.smithi110.stdout: osd shutdown pgref assert = True 2023-09-30T02:04:19.143 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:04:19.143 INFO:teuthology.orchestra.run.smithi110.stdout:[mgr] 2023-09-30T02:04:19.144 INFO:teuthology.orchestra.run.smithi110.stdout: debug ms = 1 2023-09-30T02:04:19.144 INFO:teuthology.orchestra.run.smithi110.stdout: debug mgr = 20 2023-09-30T02:04:19.144 INFO:teuthology.orchestra.run.smithi110.stdout: debug mon = 20 2023-09-30T02:04:19.144 INFO:teuthology.orchestra.run.smithi110.stdout: debug auth = 20 2023-09-30T02:04:19.144 INFO:teuthology.orchestra.run.smithi110.stdout: mon reweight min pgs per osd = 4 2023-09-30T02:04:19.144 INFO:teuthology.orchestra.run.smithi110.stdout: mon reweight min bytes per osd = 10 2023-09-30T02:04:19.144 INFO:teuthology.orchestra.run.smithi110.stdout: mgr/telemetry/nag = false 2023-09-30T02:04:19.144 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:04:19.145 INFO:teuthology.orchestra.run.smithi110.stdout:[mon] 2023-09-30T02:04:19.145 INFO:teuthology.orchestra.run.smithi110.stdout: debug ms = 1 2023-09-30T02:04:19.145 INFO:teuthology.orchestra.run.smithi110.stdout: debug mon = 20 2023-09-30T02:04:19.145 INFO:teuthology.orchestra.run.smithi110.stdout: debug paxos = 20 2023-09-30T02:04:19.145 INFO:teuthology.orchestra.run.smithi110.stdout: debug auth = 20 2023-09-30T02:04:19.145 INFO:teuthology.orchestra.run.smithi110.stdout: mon data avail warn = 5 2023-09-30T02:04:19.146 INFO:teuthology.orchestra.run.smithi110.stdout: mon mgr mkfs grace = 240 2023-09-30T02:04:19.146 INFO:teuthology.orchestra.run.smithi110.stdout: mon reweight min pgs per osd = 4 2023-09-30T02:04:19.146 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd reporter subtree level = osd 2023-09-30T02:04:19.146 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd prime pg temp = true 2023-09-30T02:04:19.146 INFO:teuthology.orchestra.run.smithi110.stdout: mon reweight min bytes per osd = 10 2023-09-30T02:04:19.146 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:04:19.147 INFO:teuthology.orchestra.run.smithi110.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-09-30T02:04:19.147 INFO:teuthology.orchestra.run.smithi110.stdout: auth mon ticket ttl = 660 # 11m 2023-09-30T02:04:19.147 INFO:teuthology.orchestra.run.smithi110.stdout: auth service ticket ttl = 240 # 4m 2023-09-30T02:04:19.147 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:04:19.147 INFO:teuthology.orchestra.run.smithi110.stdout: # don't complain about insecure global_id in the test suite 2023-09-30T02:04:19.147 INFO:teuthology.orchestra.run.smithi110.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-09-30T02:04:19.147 INFO:teuthology.orchestra.run.smithi110.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-09-30T02:04:19.148 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:04:19.148 INFO:teuthology.orchestra.run.smithi110.stdout: # 1m isn't quite enough 2023-09-30T02:04:19.148 INFO:teuthology.orchestra.run.smithi110.stdout: mon_down_mkfs_grace = 2m 2023-09-30T02:04:19.148 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:04:19.148 INFO:teuthology.orchestra.run.smithi110.stdout: mon_warn_on_filestore_osds = false 2023-09-30T02:04:19.148 INFO:teuthology.orchestra.run.smithi110.stdout: mon min osdmap epochs = 50 2023-09-30T02:04:19.149 INFO:teuthology.orchestra.run.smithi110.stdout: mon osd crush smoke test = False 2023-09-30T02:04:19.149 INFO:teuthology.orchestra.run.smithi110.stdout: mon osdmap full prune interval = 2 2023-09-30T02:04:19.149 INFO:teuthology.orchestra.run.smithi110.stdout: mon osdmap full prune min = 15 2023-09-30T02:04:19.149 INFO:teuthology.orchestra.run.smithi110.stdout: mon osdmap full prune txsize = 2 2023-09-30T02:04:19.149 INFO:teuthology.orchestra.run.smithi110.stdout: mon scrub interval = 300 2023-09-30T02:04:19.149 INFO:teuthology.orchestra.run.smithi110.stdout: mon warn on pool no app = False 2023-09-30T02:04:19.149 INFO:teuthology.orchestra.run.smithi110.stdout: paxos service trim min = 10 2023-09-30T02:04:19.149 INFO:teuthology.orchestra.run.smithi110.stdout: 2023-09-30T02:04:19.150 INFO:teuthology.orchestra.run.smithi110.stdout:[client] 2023-09-30T02:04:19.150 INFO:teuthology.orchestra.run.smithi110.stdout: rgw cache enabled = true 2023-09-30T02:04:19.150 INFO:teuthology.orchestra.run.smithi110.stdout: rgw enable ops log = true 2023-09-30T02:04:19.150 INFO:teuthology.orchestra.run.smithi110.stdout: rgw enable usage log = true 2023-09-30T02:04:19.150 INFO:teuthology.orchestra.run.smithi110.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-09-30T02:04:19.150 INFO:teuthology.orchestra.run.smithi110.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-09-30T02:04:19.151 INFO:teuthology.orchestra.run.smithi110.stdout: debug monc = 20 2023-09-30T02:04:19.151 INFO:teuthology.orchestra.run.smithi110.stdout: debug ms = 1 2023-09-30T02:04:19.151 INFO:teuthology.orchestra.run.smithi110.stdout: debug objecter = 20 2023-09-30T02:04:19.151 INFO:teuthology.orchestra.run.smithi110.stdout: debug rados = 20 2023-09-30T02:04:19.151 INFO:teuthology.orchestra.run.smithi110.stdout:[mon.a] 2023-09-30T02:04:19.151 INFO:teuthology.orchestra.run.smithi110.stdout:[mon.c] 2023-09-30T02:04:19.152 INFO:teuthology.orchestra.run.smithi110.stdout:[mon.b] 2023-09-30T02:04:19.154 INFO:teuthology.orchestra.run.smithi059.stdout:[global] 2023-09-30T02:04:19.154 INFO:teuthology.orchestra.run.smithi059.stdout: chdir = "" 2023-09-30T02:04:19.154 INFO:teuthology.orchestra.run.smithi059.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-09-30T02:04:19.154 INFO:teuthology.orchestra.run.smithi059.stdout: auth supported = cephx 2023-09-30T02:04:19.154 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:04:19.154 INFO:teuthology.orchestra.run.smithi059.stdout: filestore xattr use omap = true 2023-09-30T02:04:19.155 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:04:19.155 INFO:teuthology.orchestra.run.smithi059.stdout: mon clock drift allowed = 1.000 2023-09-30T02:04:19.155 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:04:19.155 INFO:teuthology.orchestra.run.smithi059.stdout: osd crush chooseleaf type = 0 2023-09-30T02:04:19.155 INFO:teuthology.orchestra.run.smithi059.stdout: auth debug = true 2023-09-30T02:04:19.155 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:04:19.155 INFO:teuthology.orchestra.run.smithi059.stdout: ms die on old message = true 2023-09-30T02:04:19.155 INFO:teuthology.orchestra.run.smithi059.stdout: ms die on bug = true 2023-09-30T02:04:19.156 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:04:19.156 INFO:teuthology.orchestra.run.smithi059.stdout: mon max pg per osd = 10000 # >= luminous 2023-09-30T02:04:19.156 INFO:teuthology.orchestra.run.smithi059.stdout: mon pg warn max object skew = 0 2023-09-30T02:04:19.156 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:04:19.156 INFO:teuthology.orchestra.run.smithi059.stdout: # disable pg_autoscaler by default for new pools 2023-09-30T02:04:19.156 INFO:teuthology.orchestra.run.smithi059.stdout: osd_pool_default_pg_autoscale_mode = off 2023-09-30T02:04:19.156 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:04:19.157 INFO:teuthology.orchestra.run.smithi059.stdout: osd pool default size = 2 2023-09-30T02:04:19.157 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:04:19.157 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd allow primary affinity = true 2023-09-30T02:04:19.157 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd allow pg remap = true 2023-09-30T02:04:19.157 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on legacy crush tunables = false 2023-09-30T02:04:19.157 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on crush straw calc version zero = false 2023-09-30T02:04:19.157 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on no sortbitwise = false 2023-09-30T02:04:19.157 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on osd down out interval zero = false 2023-09-30T02:04:19.158 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on too few osds = false 2023-09-30T02:04:19.158 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-09-30T02:04:19.158 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_pool_no_redundancy = false 2023-09-30T02:04:19.158 INFO:teuthology.orchestra.run.smithi059.stdout: mon_allow_pool_size_one = true 2023-09-30T02:04:19.158 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:04:19.158 INFO:teuthology.orchestra.run.smithi059.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-09-30T02:04:19.158 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:04:19.159 INFO:teuthology.orchestra.run.smithi059.stdout: osd default data pool replay window = 5 2023-09-30T02:04:19.159 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:04:19.159 INFO:teuthology.orchestra.run.smithi059.stdout: mon allow pool delete = true 2023-09-30T02:04:19.159 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:04:19.159 INFO:teuthology.orchestra.run.smithi059.stdout: mon cluster log file level = debug 2023-09-30T02:04:19.159 INFO:teuthology.orchestra.run.smithi059.stdout: debug asserts on shutdown = true 2023-09-30T02:04:19.159 INFO:teuthology.orchestra.run.smithi059.stdout: mon health detail to clog = false 2023-09-30T02:04:19.159 INFO:teuthology.orchestra.run.smithi059.stdout: mon host = "172.21.15.59,[v2:172.21.15.59:3301,v1:172.21.15.59:6790],172.21.15.110" 2023-09-30T02:04:19.160 INFO:teuthology.orchestra.run.smithi059.stdout: mon client directed command retry = 5 2023-09-30T02:04:19.160 INFO:teuthology.orchestra.run.smithi059.stdout: mon election default strategy = 3 2023-09-30T02:04:19.160 INFO:teuthology.orchestra.run.smithi059.stdout: ms bind msgr1 = False 2023-09-30T02:04:19.160 INFO:teuthology.orchestra.run.smithi059.stdout: ms bind msgr2 = True 2023-09-30T02:04:19.160 INFO:teuthology.orchestra.run.smithi059.stdout: ms inject socket failures = 5000 2023-09-30T02:04:19.160 INFO:teuthology.orchestra.run.smithi059.stdout: ms type = async 2023-09-30T02:04:19.160 INFO:teuthology.orchestra.run.smithi059.stdout: osd heartbeat grace = 80 2023-09-30T02:04:19.160 INFO:teuthology.orchestra.run.smithi059.stdout: fsid = 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:04:19.160 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:04:19.161 INFO:teuthology.orchestra.run.smithi059.stdout:[osd] 2023-09-30T02:04:19.161 INFO:teuthology.orchestra.run.smithi059.stdout: osd journal size = 100 2023-09-30T02:04:19.161 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:04:19.161 INFO:teuthology.orchestra.run.smithi059.stdout: osd scrub load threshold = 5.0 2023-09-30T02:04:19.161 INFO:teuthology.orchestra.run.smithi059.stdout: osd scrub max interval = 600 2023-09-30T02:04:19.161 INFO:teuthology.orchestra.run.smithi059.stdout: osd mclock profile = high_recovery_ops 2023-09-30T02:04:19.161 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:04:19.161 INFO:teuthology.orchestra.run.smithi059.stdout: osd recover clone overlap = true 2023-09-30T02:04:19.161 INFO:teuthology.orchestra.run.smithi059.stdout: osd recovery max chunk = 1048576 2023-09-30T02:04:19.161 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:04:19.161 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug shutdown = true 2023-09-30T02:04:19.161 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug op order = true 2023-09-30T02:04:19.162 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug verify stray on activate = true 2023-09-30T02:04:19.162 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:04:19.162 INFO:teuthology.orchestra.run.smithi059.stdout: osd open classes on start = true 2023-09-30T02:04:19.162 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug pg log writeout = true 2023-09-30T02:04:19.162 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:04:19.162 INFO:teuthology.orchestra.run.smithi059.stdout: osd deep scrub update digest min age = 30 2023-09-30T02:04:19.162 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:04:19.162 INFO:teuthology.orchestra.run.smithi059.stdout: osd map max advance = 10 2023-09-30T02:04:19.162 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:04:19.162 INFO:teuthology.orchestra.run.smithi059.stdout: journal zero on create = true 2023-09-30T02:04:19.162 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:04:19.163 INFO:teuthology.orchestra.run.smithi059.stdout: filestore ondisk finisher threads = 3 2023-09-30T02:04:19.163 INFO:teuthology.orchestra.run.smithi059.stdout: filestore apply finisher threads = 3 2023-09-30T02:04:19.163 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:04:19.163 INFO:teuthology.orchestra.run.smithi059.stdout: bdev debug aio = true 2023-09-30T02:04:19.163 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug misdirected ops = true 2023-09-30T02:04:19.163 INFO:teuthology.orchestra.run.smithi059.stdout: bdev async discard = True 2023-09-30T02:04:19.163 INFO:teuthology.orchestra.run.smithi059.stdout: bdev enable discard = True 2023-09-30T02:04:19.163 INFO:teuthology.orchestra.run.smithi059.stdout: bluestore block size = 96636764160 2023-09-30T02:04:19.163 INFO:teuthology.orchestra.run.smithi059.stdout: bluestore fsck on mount = True 2023-09-30T02:04:19.163 INFO:teuthology.orchestra.run.smithi059.stdout: bluestore zero block detection = True 2023-09-30T02:04:19.164 INFO:teuthology.orchestra.run.smithi059.stdout: debug bluefs = 1 2023-09-30T02:04:19.164 INFO:teuthology.orchestra.run.smithi059.stdout: debug bluestore = 1 2023-09-30T02:04:19.164 INFO:teuthology.orchestra.run.smithi059.stdout: debug monc = 20 2023-09-30T02:04:19.164 INFO:teuthology.orchestra.run.smithi059.stdout: debug ms = 1 2023-09-30T02:04:19.164 INFO:teuthology.orchestra.run.smithi059.stdout: debug osd = 20 2023-09-30T02:04:19.164 INFO:teuthology.orchestra.run.smithi059.stdout: debug rocksdb = 10 2023-09-30T02:04:19.164 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd backfillfull_ratio = 0.85 2023-09-30T02:04:19.164 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd full ratio = 0.9 2023-09-30T02:04:19.165 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd nearfull ratio = 0.8 2023-09-30T02:04:19.165 INFO:teuthology.orchestra.run.smithi059.stdout: osd class default list = * 2023-09-30T02:04:19.165 INFO:teuthology.orchestra.run.smithi059.stdout: osd class load list = * 2023-09-30T02:04:19.165 INFO:teuthology.orchestra.run.smithi059.stdout: osd client watch timeout = 120 2023-09-30T02:04:19.165 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug verify cached snaps = True 2023-09-30T02:04:19.165 INFO:teuthology.orchestra.run.smithi059.stdout: osd debug verify missing on start = True 2023-09-30T02:04:19.165 INFO:teuthology.orchestra.run.smithi059.stdout: osd failsafe full ratio = 0.95 2023-09-30T02:04:19.165 INFO:teuthology.orchestra.run.smithi059.stdout: osd fast shutdown = False 2023-09-30T02:04:19.166 INFO:teuthology.orchestra.run.smithi059.stdout: osd max markdown count = 1000 2023-09-30T02:04:19.166 INFO:teuthology.orchestra.run.smithi059.stdout: osd mclock override recovery settings = True 2023-09-30T02:04:19.166 INFO:teuthology.orchestra.run.smithi059.stdout: osd memory target = 1610612736 2023-09-30T02:04:19.166 INFO:teuthology.orchestra.run.smithi059.stdout: osd objectstore = bluestore 2023-09-30T02:04:19.166 INFO:teuthology.orchestra.run.smithi059.stdout: osd op queue = debug_random 2023-09-30T02:04:19.166 INFO:teuthology.orchestra.run.smithi059.stdout: osd op queue cut off = debug_random 2023-09-30T02:04:19.167 INFO:teuthology.orchestra.run.smithi059.stdout: osd shutdown pgref assert = True 2023-09-30T02:04:19.167 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:04:19.167 INFO:teuthology.orchestra.run.smithi059.stdout:[mgr] 2023-09-30T02:04:19.167 INFO:teuthology.orchestra.run.smithi059.stdout: debug ms = 1 2023-09-30T02:04:19.167 INFO:teuthology.orchestra.run.smithi059.stdout: debug mgr = 20 2023-09-30T02:04:19.167 INFO:teuthology.orchestra.run.smithi059.stdout: debug mon = 20 2023-09-30T02:04:19.167 INFO:teuthology.orchestra.run.smithi059.stdout: debug auth = 20 2023-09-30T02:04:19.168 INFO:teuthology.orchestra.run.smithi059.stdout: mon reweight min pgs per osd = 4 2023-09-30T02:04:19.168 INFO:teuthology.orchestra.run.smithi059.stdout: mon reweight min bytes per osd = 10 2023-09-30T02:04:19.168 INFO:teuthology.orchestra.run.smithi059.stdout: mgr/telemetry/nag = false 2023-09-30T02:04:19.168 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:04:19.168 INFO:teuthology.orchestra.run.smithi059.stdout:[mon] 2023-09-30T02:04:19.168 INFO:teuthology.orchestra.run.smithi059.stdout: debug ms = 1 2023-09-30T02:04:19.169 INFO:teuthology.orchestra.run.smithi059.stdout: debug mon = 20 2023-09-30T02:04:19.169 INFO:teuthology.orchestra.run.smithi059.stdout: debug paxos = 20 2023-09-30T02:04:19.169 INFO:teuthology.orchestra.run.smithi059.stdout: debug auth = 20 2023-09-30T02:04:19.169 INFO:teuthology.orchestra.run.smithi059.stdout: mon data avail warn = 5 2023-09-30T02:04:19.169 INFO:teuthology.orchestra.run.smithi059.stdout: mon mgr mkfs grace = 240 2023-09-30T02:04:19.169 INFO:teuthology.orchestra.run.smithi059.stdout: mon reweight min pgs per osd = 4 2023-09-30T02:04:19.169 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd reporter subtree level = osd 2023-09-30T02:04:19.170 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd prime pg temp = true 2023-09-30T02:04:19.170 INFO:teuthology.orchestra.run.smithi059.stdout: mon reweight min bytes per osd = 10 2023-09-30T02:04:19.170 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:04:19.170 INFO:teuthology.orchestra.run.smithi059.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-09-30T02:04:19.170 INFO:teuthology.orchestra.run.smithi059.stdout: auth mon ticket ttl = 660 # 11m 2023-09-30T02:04:19.170 INFO:teuthology.orchestra.run.smithi059.stdout: auth service ticket ttl = 240 # 4m 2023-09-30T02:04:19.170 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:04:19.171 INFO:teuthology.orchestra.run.smithi059.stdout: # don't complain about insecure global_id in the test suite 2023-09-30T02:04:19.171 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-09-30T02:04:19.171 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-09-30T02:04:19.171 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:04:19.171 INFO:teuthology.orchestra.run.smithi059.stdout: # 1m isn't quite enough 2023-09-30T02:04:19.171 INFO:teuthology.orchestra.run.smithi059.stdout: mon_down_mkfs_grace = 2m 2023-09-30T02:04:19.172 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:04:19.172 INFO:teuthology.orchestra.run.smithi059.stdout: mon_warn_on_filestore_osds = false 2023-09-30T02:04:19.172 INFO:teuthology.orchestra.run.smithi059.stdout: mon min osdmap epochs = 50 2023-09-30T02:04:19.172 INFO:teuthology.orchestra.run.smithi059.stdout: mon osd crush smoke test = False 2023-09-30T02:04:19.172 INFO:teuthology.orchestra.run.smithi059.stdout: mon osdmap full prune interval = 2 2023-09-30T02:04:19.172 INFO:teuthology.orchestra.run.smithi059.stdout: mon osdmap full prune min = 15 2023-09-30T02:04:19.173 INFO:teuthology.orchestra.run.smithi059.stdout: mon osdmap full prune txsize = 2 2023-09-30T02:04:19.173 INFO:teuthology.orchestra.run.smithi059.stdout: mon scrub interval = 300 2023-09-30T02:04:19.173 INFO:teuthology.orchestra.run.smithi059.stdout: mon warn on pool no app = False 2023-09-30T02:04:19.173 INFO:teuthology.orchestra.run.smithi059.stdout: paxos service trim min = 10 2023-09-30T02:04:19.173 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:04:19.173 INFO:teuthology.orchestra.run.smithi059.stdout:[client] 2023-09-30T02:04:19.174 INFO:teuthology.orchestra.run.smithi059.stdout: rgw cache enabled = true 2023-09-30T02:04:19.174 INFO:teuthology.orchestra.run.smithi059.stdout: rgw enable ops log = true 2023-09-30T02:04:19.174 INFO:teuthology.orchestra.run.smithi059.stdout: rgw enable usage log = true 2023-09-30T02:04:19.174 INFO:teuthology.orchestra.run.smithi059.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-09-30T02:04:19.174 INFO:teuthology.orchestra.run.smithi059.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-09-30T02:04:19.174 INFO:teuthology.orchestra.run.smithi059.stdout: debug monc = 20 2023-09-30T02:04:19.175 INFO:teuthology.orchestra.run.smithi059.stdout: debug ms = 1 2023-09-30T02:04:19.175 INFO:teuthology.orchestra.run.smithi059.stdout: debug objecter = 20 2023-09-30T02:04:19.175 INFO:teuthology.orchestra.run.smithi059.stdout: debug rados = 20 2023-09-30T02:04:19.175 INFO:teuthology.orchestra.run.smithi059.stdout:[mon.a] 2023-09-30T02:04:19.175 INFO:teuthology.orchestra.run.smithi059.stdout:[mon.c] 2023-09-30T02:04:19.175 INFO:teuthology.orchestra.run.smithi059.stdout:[mon.b] 2023-09-30T02:04:19.175 INFO:tasks.ceph:Creating admin key on mon.a... 2023-09-30T02:04:19.176 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-09-30T02:04:19.240 INFO:tasks.ceph:Copying monmap to all nodes... 2023-09-30T02:04:19.240 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2023-09-30T02:04:19.240 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-09-30T02:04:19.261 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2023-09-30T02:04:19.261 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-09-30T02:04:19.323 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi059.front.sepia.ceph.com 2023-09-30T02:04:19.323 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2023-09-30T02:04:19.323 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/etc/ceph/ceph.keyring 2023-09-30T02:04:19.324 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-09-30T02:04:19.411 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2023-09-30T02:04:19.411 DEBUG:teuthology.orchestra.run.smithi059:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-09-30T02:04:19.474 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi110.front.sepia.ceph.com 2023-09-30T02:04:19.474 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-09-30T02:04:19.474 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd of=/etc/ceph/ceph.keyring 2023-09-30T02:04:19.474 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-09-30T02:04:19.512 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-09-30T02:04:19.512 DEBUG:teuthology.orchestra.run.smithi110:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-09-30T02:04:19.570 INFO:tasks.ceph:Setting up mon nodes... 2023-09-30T02:04:19.570 INFO:tasks.ceph:Setting up mgr nodes... 2023-09-30T02:04:19.570 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-09-30T02:04:19.632 INFO:teuthology.orchestra.run.smithi059.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-09-30T02:04:19.635 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-09-30T02:04:19.693 INFO:teuthology.orchestra.run.smithi110.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-09-30T02:04:19.695 INFO:tasks.ceph:Setting up mds nodes... 2023-09-30T02:04:19.695 INFO:tasks.ceph_client:Setting up client nodes... 2023-09-30T02:04:19.695 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-09-30T02:04:19.747 INFO:teuthology.orchestra.run.smithi059.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-09-30T02:04:19.762 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-09-30T02:04:19.803 INFO:teuthology.orchestra.run.smithi110.stdout:creating /etc/ceph/ceph.client.1.keyring 2023-09-30T02:04:19.815 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-09-30T02:04:19.815 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi059.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi110.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-09-30T02:04:19.815 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-09-30T02:04:19.850 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-09-30T02:04:19.850 INFO:tasks.ceph:role: osd.0 2023-09-30T02:04:19.851 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi059.front.sepia.ceph.com 2023-09-30T02:04:19.851 DEBUG:teuthology.orchestra.run.smithi059:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-09-30T02:04:19.933 INFO:teuthology.orchestra.run.smithi059.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T02:04:19.933 INFO:teuthology.orchestra.run.smithi059.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T02:04:19.933 INFO:teuthology.orchestra.run.smithi059.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T02:04:19.933 INFO:teuthology.orchestra.run.smithi059.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-09-30T02:04:19.933 INFO:teuthology.orchestra.run.smithi059.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T02:04:19.934 INFO:teuthology.orchestra.run.smithi059.stdout: = sunit=0 swidth=0 blks 2023-09-30T02:04:19.934 INFO:teuthology.orchestra.run.smithi059.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T02:04:19.934 INFO:teuthology.orchestra.run.smithi059.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T02:04:19.934 INFO:teuthology.orchestra.run.smithi059.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T02:04:19.934 INFO:teuthology.orchestra.run.smithi059.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T02:04:20.362 INFO:teuthology.orchestra.run.smithi059.stdout:Discarding blocks...Done. 2023-09-30T02:04:20.364 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi059.front.sepia.ceph.com -o noatime 2023-09-30T02:04:20.364 DEBUG:teuthology.orchestra.run.smithi059:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-09-30T02:04:20.519 DEBUG:teuthology.orchestra.run.smithi059:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-09-30T02:04:20.562 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-09-30T02:04:20.637 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-09-30T02:04:20.638 INFO:tasks.ceph:role: osd.1 2023-09-30T02:04:20.638 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi059.front.sepia.ceph.com 2023-09-30T02:04:20.638 DEBUG:teuthology.orchestra.run.smithi059:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-09-30T02:04:20.720 INFO:teuthology.orchestra.run.smithi059.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T02:04:20.721 INFO:teuthology.orchestra.run.smithi059.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T02:04:20.721 INFO:teuthology.orchestra.run.smithi059.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T02:04:20.721 INFO:teuthology.orchestra.run.smithi059.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-09-30T02:04:20.721 INFO:teuthology.orchestra.run.smithi059.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T02:04:20.721 INFO:teuthology.orchestra.run.smithi059.stdout: = sunit=0 swidth=0 blks 2023-09-30T02:04:20.721 INFO:teuthology.orchestra.run.smithi059.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T02:04:20.721 INFO:teuthology.orchestra.run.smithi059.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T02:04:20.721 INFO:teuthology.orchestra.run.smithi059.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T02:04:20.721 INFO:teuthology.orchestra.run.smithi059.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T02:04:21.143 INFO:teuthology.orchestra.run.smithi059.stdout:Discarding blocks...Done. 2023-09-30T02:04:21.145 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi059.front.sepia.ceph.com -o noatime 2023-09-30T02:04:21.145 DEBUG:teuthology.orchestra.run.smithi059:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-09-30T02:04:21.263 DEBUG:teuthology.orchestra.run.smithi059:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-09-30T02:04:21.301 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-09-30T02:04:21.378 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-09-30T02:04:21.378 INFO:tasks.ceph:role: osd.2 2023-09-30T02:04:21.378 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi059.front.sepia.ceph.com 2023-09-30T02:04:21.378 DEBUG:teuthology.orchestra.run.smithi059:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-09-30T02:04:21.462 INFO:teuthology.orchestra.run.smithi059.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T02:04:21.462 INFO:teuthology.orchestra.run.smithi059.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T02:04:21.462 INFO:teuthology.orchestra.run.smithi059.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T02:04:21.462 INFO:teuthology.orchestra.run.smithi059.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-09-30T02:04:21.462 INFO:teuthology.orchestra.run.smithi059.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T02:04:21.462 INFO:teuthology.orchestra.run.smithi059.stdout: = sunit=0 swidth=0 blks 2023-09-30T02:04:21.462 INFO:teuthology.orchestra.run.smithi059.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T02:04:21.462 INFO:teuthology.orchestra.run.smithi059.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T02:04:21.462 INFO:teuthology.orchestra.run.smithi059.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T02:04:21.462 INFO:teuthology.orchestra.run.smithi059.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T02:04:21.905 INFO:teuthology.orchestra.run.smithi059.stdout:Discarding blocks...Done. 2023-09-30T02:04:21.907 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi059.front.sepia.ceph.com -o noatime 2023-09-30T02:04:21.907 DEBUG:teuthology.orchestra.run.smithi059:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-09-30T02:04:22.085 DEBUG:teuthology.orchestra.run.smithi059:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-09-30T02:04:22.124 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-09-30T02:04:22.200 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-09-30T02:04:22.200 INFO:tasks.ceph:role: osd.3 2023-09-30T02:04:22.200 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi059.front.sepia.ceph.com 2023-09-30T02:04:22.200 DEBUG:teuthology.orchestra.run.smithi059:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-09-30T02:04:22.283 INFO:teuthology.orchestra.run.smithi059.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T02:04:22.284 INFO:teuthology.orchestra.run.smithi059.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T02:04:22.284 INFO:teuthology.orchestra.run.smithi059.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T02:04:22.284 INFO:teuthology.orchestra.run.smithi059.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-09-30T02:04:22.284 INFO:teuthology.orchestra.run.smithi059.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T02:04:22.284 INFO:teuthology.orchestra.run.smithi059.stdout: = sunit=0 swidth=0 blks 2023-09-30T02:04:22.284 INFO:teuthology.orchestra.run.smithi059.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T02:04:22.284 INFO:teuthology.orchestra.run.smithi059.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T02:04:22.284 INFO:teuthology.orchestra.run.smithi059.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T02:04:22.284 INFO:teuthology.orchestra.run.smithi059.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T02:04:22.719 INFO:teuthology.orchestra.run.smithi059.stdout:Discarding blocks...Done. 2023-09-30T02:04:22.721 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi059.front.sepia.ceph.com -o noatime 2023-09-30T02:04:22.721 DEBUG:teuthology.orchestra.run.smithi059:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-09-30T02:04:22.856 DEBUG:teuthology.orchestra.run.smithi059:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-09-30T02:04:22.896 DEBUG:teuthology.orchestra.run.smithi059:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T02:04:23.017 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:23.016+0000 7f038febf540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-09-30T02:04:23.017 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:23.016+0000 7f038febf540 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-09-30T02:04:23.017 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:23.016+0000 7f038febf540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-09-30T02:04:23.017 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:23.016+0000 7f038febf540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-09-30T02:04:23.018 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:23.016+0000 7f038febf540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-09-30T02:04:23.018 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:23.017+0000 7f038febf540 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-09-30T02:04:23.544 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:23.543+0000 7f038febf540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T02:04:24.307 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:24.306+0000 7f038febf540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T02:04:25.070 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:25.069+0000 7f038febf540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T02:04:25.861 DEBUG:teuthology.orchestra.run.smithi059:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-09-30T02:04:25.896 DEBUG:teuthology.orchestra.run.smithi059:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T02:04:25.991 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:25.991+0000 7f9979ad3540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-09-30T02:04:25.991 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:25.991+0000 7f9979ad3540 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-09-30T02:04:25.992 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:25.991+0000 7f9979ad3540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-09-30T02:04:25.992 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:25.991+0000 7f9979ad3540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-09-30T02:04:25.992 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:25.991+0000 7f9979ad3540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-09-30T02:04:25.992 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:25.991+0000 7f9979ad3540 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-09-30T02:04:26.519 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:26.519+0000 7f9979ad3540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T02:04:27.282 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:27.281+0000 7f9979ad3540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T02:04:28.044 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:28.044+0000 7f9979ad3540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T02:04:28.837 DEBUG:teuthology.orchestra.run.smithi059:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-09-30T02:04:28.873 DEBUG:teuthology.orchestra.run.smithi059:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T02:04:28.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:28.968+0000 7faf17cff540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-09-30T02:04:28.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:28.968+0000 7faf17cff540 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-09-30T02:04:28.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:28.968+0000 7faf17cff540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-09-30T02:04:28.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:28.968+0000 7faf17cff540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-09-30T02:04:28.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:28.968+0000 7faf17cff540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-09-30T02:04:28.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:28.969+0000 7faf17cff540 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-09-30T02:04:29.495 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:29.494+0000 7faf17cff540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T02:04:30.260 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:30.260+0000 7faf17cff540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T02:04:31.023 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:31.023+0000 7faf17cff540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T02:04:31.820 DEBUG:teuthology.orchestra.run.smithi059:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-09-30T02:04:31.855 DEBUG:teuthology.orchestra.run.smithi059:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T02:04:31.951 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:31.950+0000 7f148fb0b540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-09-30T02:04:31.951 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:31.951+0000 7f148fb0b540 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-09-30T02:04:31.951 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:31.951+0000 7f148fb0b540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-09-30T02:04:31.951 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:31.951+0000 7f148fb0b540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-09-30T02:04:31.951 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:31.951+0000 7f148fb0b540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-09-30T02:04:31.952 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:31.951+0000 7f148fb0b540 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-09-30T02:04:32.479 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:32.478+0000 7f148fb0b540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T02:04:33.238 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:33.237+0000 7f148fb0b540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T02:04:34.000 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:34.000+0000 7f148fb0b540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T02:04:34.788 DEBUG:teuthology.orchestra.run.smithi059:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-09-30T02:04:34.824 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-09-30T02:04:34.857 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-09-30T02:04:34.857 INFO:tasks.ceph:role: osd.4 2023-09-30T02:04:34.857 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi110.front.sepia.ceph.com 2023-09-30T02:04:34.857 DEBUG:teuthology.orchestra.run.smithi110:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-09-30T02:04:34.928 INFO:teuthology.orchestra.run.smithi110.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T02:04:34.928 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T02:04:34.928 INFO:teuthology.orchestra.run.smithi110.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T02:04:34.928 INFO:teuthology.orchestra.run.smithi110.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-09-30T02:04:34.929 INFO:teuthology.orchestra.run.smithi110.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T02:04:34.929 INFO:teuthology.orchestra.run.smithi110.stdout: = sunit=0 swidth=0 blks 2023-09-30T02:04:34.929 INFO:teuthology.orchestra.run.smithi110.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T02:04:34.929 INFO:teuthology.orchestra.run.smithi110.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T02:04:34.929 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T02:04:34.929 INFO:teuthology.orchestra.run.smithi110.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T02:04:34.951 INFO:teuthology.orchestra.run.smithi110.stdout:Discarding blocks...Done. 2023-09-30T02:04:34.953 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi110.front.sepia.ceph.com -o noatime 2023-09-30T02:04:34.953 DEBUG:teuthology.orchestra.run.smithi110:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-09-30T02:04:35.035 DEBUG:teuthology.orchestra.run.smithi110:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-09-30T02:04:35.107 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-09-30T02:04:35.175 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-09-30T02:04:35.175 INFO:tasks.ceph:role: osd.5 2023-09-30T02:04:35.175 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi110.front.sepia.ceph.com 2023-09-30T02:04:35.175 DEBUG:teuthology.orchestra.run.smithi110:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-09-30T02:04:35.242 INFO:teuthology.orchestra.run.smithi110.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T02:04:35.242 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T02:04:35.242 INFO:teuthology.orchestra.run.smithi110.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T02:04:35.242 INFO:teuthology.orchestra.run.smithi110.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-09-30T02:04:35.242 INFO:teuthology.orchestra.run.smithi110.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T02:04:35.243 INFO:teuthology.orchestra.run.smithi110.stdout: = sunit=0 swidth=0 blks 2023-09-30T02:04:35.243 INFO:teuthology.orchestra.run.smithi110.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T02:04:35.243 INFO:teuthology.orchestra.run.smithi110.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T02:04:35.243 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T02:04:35.243 INFO:teuthology.orchestra.run.smithi110.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T02:04:35.265 INFO:teuthology.orchestra.run.smithi110.stdout:Discarding blocks...Done. 2023-09-30T02:04:35.266 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi110.front.sepia.ceph.com -o noatime 2023-09-30T02:04:35.266 DEBUG:teuthology.orchestra.run.smithi110:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-09-30T02:04:35.346 DEBUG:teuthology.orchestra.run.smithi110:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-09-30T02:04:35.417 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-09-30T02:04:35.484 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-09-30T02:04:35.484 INFO:tasks.ceph:role: osd.6 2023-09-30T02:04:35.485 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi110.front.sepia.ceph.com 2023-09-30T02:04:35.485 DEBUG:teuthology.orchestra.run.smithi110:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-09-30T02:04:35.551 INFO:teuthology.orchestra.run.smithi110.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T02:04:35.551 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T02:04:35.551 INFO:teuthology.orchestra.run.smithi110.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T02:04:35.551 INFO:teuthology.orchestra.run.smithi110.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-09-30T02:04:35.551 INFO:teuthology.orchestra.run.smithi110.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T02:04:35.551 INFO:teuthology.orchestra.run.smithi110.stdout: = sunit=0 swidth=0 blks 2023-09-30T02:04:35.551 INFO:teuthology.orchestra.run.smithi110.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T02:04:35.552 INFO:teuthology.orchestra.run.smithi110.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T02:04:35.552 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T02:04:35.552 INFO:teuthology.orchestra.run.smithi110.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T02:04:35.574 INFO:teuthology.orchestra.run.smithi110.stdout:Discarding blocks...Done. 2023-09-30T02:04:35.575 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi110.front.sepia.ceph.com -o noatime 2023-09-30T02:04:35.576 DEBUG:teuthology.orchestra.run.smithi110:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-09-30T02:04:35.655 DEBUG:teuthology.orchestra.run.smithi110:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-09-30T02:04:35.726 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-09-30T02:04:35.793 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-09-30T02:04:35.793 INFO:tasks.ceph:role: osd.7 2023-09-30T02:04:35.794 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi110.front.sepia.ceph.com 2023-09-30T02:04:35.794 DEBUG:teuthology.orchestra.run.smithi110:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-09-30T02:04:35.860 INFO:teuthology.orchestra.run.smithi110.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T02:04:35.860 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T02:04:35.860 INFO:teuthology.orchestra.run.smithi110.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T02:04:35.860 INFO:teuthology.orchestra.run.smithi110.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-09-30T02:04:35.860 INFO:teuthology.orchestra.run.smithi110.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T02:04:35.860 INFO:teuthology.orchestra.run.smithi110.stdout: = sunit=0 swidth=0 blks 2023-09-30T02:04:35.861 INFO:teuthology.orchestra.run.smithi110.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T02:04:35.861 INFO:teuthology.orchestra.run.smithi110.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T02:04:35.861 INFO:teuthology.orchestra.run.smithi110.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T02:04:35.861 INFO:teuthology.orchestra.run.smithi110.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T02:04:35.883 INFO:teuthology.orchestra.run.smithi110.stdout:Discarding blocks...Done. 2023-09-30T02:04:35.885 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi110.front.sepia.ceph.com -o noatime 2023-09-30T02:04:35.885 DEBUG:teuthology.orchestra.run.smithi110:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-09-30T02:04:35.965 DEBUG:teuthology.orchestra.run.smithi110:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-09-30T02:04:36.035 DEBUG:teuthology.orchestra.run.smithi110:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T02:04:36.131 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:04:36.131+0000 7f02efd1e540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-09-30T02:04:36.131 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:04:36.131+0000 7f02efd1e540 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-09-30T02:04:36.131 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:04:36.131+0000 7f02efd1e540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-09-30T02:04:36.131 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:04:36.131+0000 7f02efd1e540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-09-30T02:04:36.131 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:04:36.131+0000 7f02efd1e540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-09-30T02:04:36.131 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:04:36.131+0000 7f02efd1e540 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-09-30T02:04:36.657 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:04:36.658+0000 7f02efd1e540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T02:04:37.421 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:04:37.421+0000 7f02efd1e540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T02:04:38.184 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:04:38.184+0000 7f02efd1e540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T02:04:38.974 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-09-30T02:04:39.006 DEBUG:teuthology.orchestra.run.smithi110:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T02:04:39.095 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:04:39.094+0000 7fbdba670540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-09-30T02:04:39.095 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:04:39.095+0000 7fbdba670540 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-09-30T02:04:39.095 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:04:39.095+0000 7fbdba670540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-09-30T02:04:39.095 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:04:39.095+0000 7fbdba670540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-09-30T02:04:39.095 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:04:39.095+0000 7fbdba670540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-09-30T02:04:39.095 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:04:39.095+0000 7fbdba670540 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-09-30T02:04:39.620 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:04:39.620+0000 7fbdba670540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T02:04:40.385 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:04:40.385+0000 7fbdba670540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T02:04:41.155 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:04:41.155+0000 7fbdba670540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T02:04:41.945 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-09-30T02:04:41.977 DEBUG:teuthology.orchestra.run.smithi110:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T02:04:42.066 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:04:42.066+0000 7fea5627e540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-09-30T02:04:42.066 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:04:42.066+0000 7fea5627e540 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-09-30T02:04:42.066 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:04:42.066+0000 7fea5627e540 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-09-30T02:04:42.067 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:04:42.066+0000 7fea5627e540 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-09-30T02:04:42.067 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:04:42.066+0000 7fea5627e540 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-09-30T02:04:42.067 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:04:42.066+0000 7fea5627e540 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-09-30T02:04:42.589 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:04:42.589+0000 7fea5627e540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T02:04:43.350 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:04:43.350+0000 7fea5627e540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T02:04:44.116 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:04:44.116+0000 7fea5627e540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T02:04:44.909 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-09-30T02:04:44.942 DEBUG:teuthology.orchestra.run.smithi110:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T02:04:45.032 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:04:45.032+0000 7f60eb91f540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-09-30T02:04:45.032 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:04:45.032+0000 7f60eb91f540 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-09-30T02:04:45.032 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:04:45.032+0000 7f60eb91f540 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-09-30T02:04:45.032 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:04:45.032+0000 7f60eb91f540 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-09-30T02:04:45.032 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:04:45.032+0000 7f60eb91f540 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-09-30T02:04:45.033 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:04:45.032+0000 7f60eb91f540 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-09-30T02:04:45.555 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:04:45.555+0000 7f60eb91f540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T02:04:46.317 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:04:46.317+0000 7f60eb91f540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T02:04:47.080 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:04:47.080+0000 7f60eb91f540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T02:04:47.868 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-09-30T02:04:47.900 INFO:tasks.ceph:Reading keys from all nodes... 2023-09-30T02:04:47.900 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2023-09-30T02:04:47.901 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-09-30T02:04:47.936 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2023-09-30T02:04:47.936 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-09-30T02:04:48.011 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2023-09-30T02:04:48.011 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-09-30T02:04:48.086 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2023-09-30T02:04:48.086 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-09-30T02:04:48.161 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2023-09-30T02:04:48.161 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-09-30T02:04:48.236 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-09-30T02:04:48.236 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-09-30T02:04:48.269 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-09-30T02:04:48.269 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-09-30T02:04:48.342 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-09-30T02:04:48.342 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-09-30T02:04:48.414 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-09-30T02:04:48.414 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-09-30T02:04:48.485 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-09-30T02:04:48.485 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-09-30T02:04:48.554 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2023-09-30T02:04:48.554 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-09-30T02:04:48.577 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-09-30T02:04:48.577 DEBUG:teuthology.orchestra.run.smithi110:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2023-09-30T02:04:48.613 INFO:tasks.ceph:Adding keys to all mons... 2023-09-30T02:04:48.613 DEBUG:teuthology.orchestra.run.smithi059:> sudo tee -a /etc/ceph/ceph.keyring 2023-09-30T02:04:48.619 DEBUG:teuthology.orchestra.run.smithi110:> sudo tee -a /etc/ceph/ceph.keyring 2023-09-30T02:04:48.655 INFO:teuthology.orchestra.run.smithi059.stdout:[mgr.y] 2023-09-30T02:04:48.655 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQAjghdlrODJJRAASs9hIiJ1k48opVW1G1Xqhw== 2023-09-30T02:04:48.656 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.0] 2023-09-30T02:04:48.656 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQAnghdlmIISARAATXbEmhBxpT8aobfYvA5fOw== 2023-09-30T02:04:48.656 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.1] 2023-09-30T02:04:48.656 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQApghdlufcnOxAAInJYohhMshK8WJUA9OXIkA== 2023-09-30T02:04:48.656 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.2] 2023-09-30T02:04:48.656 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQAsghdlrm7MORAA/yHlvEuVM6vTThiW34ycuQ== 2023-09-30T02:04:48.656 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.3] 2023-09-30T02:04:48.656 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQAvghdli27DOBAAdJk6in+Lh6CZtnXUJ3SUtA== 2023-09-30T02:04:48.656 INFO:teuthology.orchestra.run.smithi059.stdout:[mgr.x] 2023-09-30T02:04:48.656 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQAjghdlVyFfKRAA1kZHL8sShPAhOZiTHbOvFQ== 2023-09-30T02:04:48.656 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.4] 2023-09-30T02:04:48.656 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQA0ghdlsXffBxAAhpMgO6jciT+DLmo9m3BM/g== 2023-09-30T02:04:48.657 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.5] 2023-09-30T02:04:48.657 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQA3ghdltgC3BRAAUQIjCBkXM9R7GLh6CNfeBw== 2023-09-30T02:04:48.657 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.6] 2023-09-30T02:04:48.657 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQA6ghdlts//AxAAdIwPSs0bctF+ujo9HT4bIQ== 2023-09-30T02:04:48.657 INFO:teuthology.orchestra.run.smithi059.stdout:[osd.7] 2023-09-30T02:04:48.657 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQA9ghdli6j3ARAA4H2BDl/7a8F7LmNE3br1sw== 2023-09-30T02:04:48.657 INFO:teuthology.orchestra.run.smithi059.stdout:[client.0] 2023-09-30T02:04:48.657 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQAjghdl0GCcLBAAoEj7l4q2QFNgayFeIIho8Q== 2023-09-30T02:04:48.657 INFO:teuthology.orchestra.run.smithi059.stdout:[client.1] 2023-09-30T02:04:48.657 INFO:teuthology.orchestra.run.smithi059.stdout: key = AQAjghdlkwPzLxAAKoOaj7y9XcIvdaSZEgJAlg== 2023-09-30T02:04:48.679 INFO:teuthology.orchestra.run.smithi110.stdout:[mgr.y] 2023-09-30T02:04:48.679 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQAjghdlrODJJRAASs9hIiJ1k48opVW1G1Xqhw== 2023-09-30T02:04:48.679 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.0] 2023-09-30T02:04:48.680 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQAnghdlmIISARAATXbEmhBxpT8aobfYvA5fOw== 2023-09-30T02:04:48.680 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.1] 2023-09-30T02:04:48.680 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQApghdlufcnOxAAInJYohhMshK8WJUA9OXIkA== 2023-09-30T02:04:48.680 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.2] 2023-09-30T02:04:48.680 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQAsghdlrm7MORAA/yHlvEuVM6vTThiW34ycuQ== 2023-09-30T02:04:48.680 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.3] 2023-09-30T02:04:48.680 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQAvghdli27DOBAAdJk6in+Lh6CZtnXUJ3SUtA== 2023-09-30T02:04:48.680 INFO:teuthology.orchestra.run.smithi110.stdout:[mgr.x] 2023-09-30T02:04:48.680 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQAjghdlVyFfKRAA1kZHL8sShPAhOZiTHbOvFQ== 2023-09-30T02:04:48.680 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.4] 2023-09-30T02:04:48.680 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQA0ghdlsXffBxAAhpMgO6jciT+DLmo9m3BM/g== 2023-09-30T02:04:48.680 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.5] 2023-09-30T02:04:48.680 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQA3ghdltgC3BRAAUQIjCBkXM9R7GLh6CNfeBw== 2023-09-30T02:04:48.681 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.6] 2023-09-30T02:04:48.681 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQA6ghdlts//AxAAdIwPSs0bctF+ujo9HT4bIQ== 2023-09-30T02:04:48.681 INFO:teuthology.orchestra.run.smithi110.stdout:[osd.7] 2023-09-30T02:04:48.681 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQA9ghdli6j3ARAA4H2BDl/7a8F7LmNE3br1sw== 2023-09-30T02:04:48.681 INFO:teuthology.orchestra.run.smithi110.stdout:[client.0] 2023-09-30T02:04:48.681 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQAjghdl0GCcLBAAoEj7l4q2QFNgayFeIIho8Q== 2023-09-30T02:04:48.681 INFO:teuthology.orchestra.run.smithi110.stdout:[client.1] 2023-09-30T02:04:48.681 INFO:teuthology.orchestra.run.smithi110.stdout: key = AQAjghdlkwPzLxAAKoOaj7y9XcIvdaSZEgJAlg== 2023-09-30T02:04:48.682 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-09-30T02:04:48.700 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-09-30T02:04:48.773 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T02:04:48.776 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T02:04:48.832 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T02:04:48.835 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T02:04:48.890 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T02:04:48.892 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T02:04:48.947 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T02:04:48.949 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T02:04:49.004 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-09-30T02:04:49.006 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-09-30T02:04:49.060 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T02:04:49.062 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T02:04:49.115 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T02:04:49.118 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T02:04:49.171 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T02:04:49.174 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T02:04:49.227 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T02:04:49.229 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T02:04:49.282 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-09-30T02:04:49.285 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-09-30T02:04:49.341 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-09-30T02:04:49.344 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-09-30T02:04:49.400 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-09-30T02:04:49.400 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-09-30T02:04:49.430 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-09-30T02:04:49.535 DEBUG:teuthology.orchestra.run.smithi059:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-09-30T02:04:49.567 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-09-30T02:04:49.641 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-09-30T02:04:49.749 DEBUG:teuthology.orchestra.run.smithi059:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-09-30T02:04:49.782 DEBUG:teuthology.orchestra.run.smithi110:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-09-30T02:04:49.811 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-09-30T02:04:49.906 DEBUG:teuthology.orchestra.run.smithi110:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-09-30T02:04:49.934 DEBUG:teuthology.orchestra.run.smithi059:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-09-30T02:04:49.937 DEBUG:teuthology.orchestra.run.smithi110:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-09-30T02:04:49.993 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-09-30T02:04:49.993 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2023-09-30T02:04:49.993 INFO:tasks.ceph.mon.a:Restarting daemon 2023-09-30T02:04:49.993 DEBUG:teuthology.orchestra.run.smithi059:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2023-09-30T02:04:49.999 INFO:tasks.ceph.mon.a:Started 2023-09-30T02:04:50.000 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2023-09-30T02:04:50.000 INFO:tasks.ceph.mon.c:Restarting daemon 2023-09-30T02:04:50.000 DEBUG:teuthology.orchestra.run.smithi059:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2023-09-30T02:04:50.002 INFO:tasks.ceph.mon.c:Started 2023-09-30T02:04:50.003 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2023-09-30T02:04:50.003 INFO:tasks.ceph.mon.b:Restarting daemon 2023-09-30T02:04:50.003 DEBUG:teuthology.orchestra.run.smithi110:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2023-09-30T02:04:50.035 INFO:tasks.ceph.mon.b:Started 2023-09-30T02:04:50.035 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-09-30T02:04:50.035 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-09-30T02:04:50.036 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2023-09-30T02:04:50.038 INFO:tasks.ceph.mgr.y:Started 2023-09-30T02:04:50.038 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-09-30T02:04:50.038 DEBUG:teuthology.orchestra.run.smithi110:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2023-09-30T02:04:50.040 INFO:tasks.ceph.mgr.x:Started 2023-09-30T02:04:50.040 INFO:tasks.ceph:Setting crush tunables to default 2023-09-30T02:04:50.040 DEBUG:teuthology.orchestra.run.smithi059:> sudo ceph --cluster ceph osd crush tunables default 2023-09-30T02:04:50.186 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:50.187+0000 7f4cae509700 10 monclient: get_monmap_and_config 2023-09-30T02:04:50.187 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:50.187+0000 7f4cae509700 10 monclient: build_initial_monmap 2023-09-30T02:04:50.187 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:50.187+0000 7f4cae509700 10 monclient: monmap: 2023-09-30T02:04:50.187 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:04:50.187 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:04:50.187 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:50.187808+0000 2023-09-30T02:04:50.187 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:50.187808+0000 2023-09-30T02:04:50.187 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:04:50.187 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:04:50.187 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:04:50.187 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:04:50.188 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:04:50.188 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:04:50.188 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:04:50.188 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:04:50.189 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:50.188+0000 7f4cae509700 1 Processor -- start 2023-09-30T02:04:50.189 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:50.188+0000 7f4cae509700 1 -- start start 2023-09-30T02:04:50.189 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:50.188+0000 7f4cae509700 10 monclient: init 2023-09-30T02:04:50.189 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:50.188+0000 7f4cae509700 10 monclient: _reopen_session rank -1 2023-09-30T02:04:50.189 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:50.188+0000 7f4cae509700 10 monclient: _add_conns ranks=[0,4,3,1,2] 2023-09-30T02:04:50.189 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:50.188+0000 7f4cae509700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4ca80a5b70 addr v1:172.21.15.59:6789/0 2023-09-30T02:04:50.189 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:50.188+0000 7f4cae509700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4ca8142290 0x7f4ca80b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:04:50.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:50.188+0000 7f4cae509700 10 monclient(hunting): picked mon.noname-c con 0x7f4ca8142290 addr v2:172.21.15.110:3300/0 2023-09-30T02:04:50.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:50.189+0000 7f4cae509700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f4ca80a5f40 0x7f4ca813af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:04:50.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:50.189+0000 7f4cae509700 10 monclient(hunting): picked mon.noname-a con 0x7f4ca80a5f40 addr v2:172.21.15.59:3300/0 2023-09-30T02:04:50.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:50.189+0000 7f4cae509700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ca80a75a0 con 0x7f4ca80a5b70 2023-09-30T02:04:50.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:50.189+0000 7f4cae509700 10 monclient(hunting): start opening mon connection 2023-09-30T02:04:50.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:50.189+0000 7f4cae509700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ca80a6360 con 0x7f4ca80a5f40 2023-09-30T02:04:50.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:50.189+0000 7f4cae509700 10 monclient(hunting): start opening mon connection 2023-09-30T02:04:50.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:50.189+0000 7f4cae509700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ca80af3f0 con 0x7f4ca8142290 2023-09-30T02:04:50.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:50.189+0000 7f4cae509700 10 monclient(hunting): _renew_subs 2023-09-30T02:04:50.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:50.189+0000 7f4cae509700 10 monclient(hunting): authenticate will time out at 1735.491058s 2023-09-30T02:04:50.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:50.189+0000 7f4ca7fff700 1 -- >> v1:172.21.15.59:6789/0 conn(0x7f4ca80a5b70 legacy=0x7f4ca8141b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.59:6789/0 2023-09-30T02:04:50.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:50.189+0000 7f4cacaa6700 1 -- >> v2:172.21.15.59:3300/0 conn(0x7f4ca80a5f40 msgr2=0x7f4ca813af30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.59:3300/0 2023-09-30T02:04:50.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:50.189+0000 7f4cacaa6700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f4ca80a5f40 0x7f4ca813af30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-30T02:04:50.191 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:50.189+0000 7f4ca77fe700 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f4ca8142290 msgr2=0x7f4ca80b1500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2023-09-30T02:04:50.191 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:50.189+0000 7f4ca77fe700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4ca8142290 0x7f4ca80b1500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-30T02:04:50.389 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:50.390+0000 7f4cacaa6700 1 -- >> v2:172.21.15.59:3300/0 conn(0x7f4ca80a5f40 msgr2=0x7f4ca813af30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.59:3300/0 2023-09-30T02:04:50.389 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:50.390+0000 7f4ca7fff700 1 -- >> v1:172.21.15.59:6789/0 conn(0x7f4ca80a5b70 legacy=0x7f4ca8141b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.59:6789/0 2023-09-30T02:04:50.389 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:50.390+0000 7f4ca77fe700 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f4ca8142290 msgr2=0x7f4ca80b1500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2023-09-30T02:04:50.390 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:50.390+0000 7f4cacaa6700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f4ca80a5f40 0x7f4ca813af30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-09-30T02:04:50.390 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:50.390+0000 7f4ca77fe700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4ca8142290 0x7f4ca80b1500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-09-30T02:04:50.790 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:50.791+0000 7f4cacaa6700 1 -- >> v2:172.21.15.59:3300/0 conn(0x7f4ca80a5f40 msgr2=0x7f4ca813af30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.59:3300/0 2023-09-30T02:04:50.790 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:50.791+0000 7f4ca7fff700 1 -- >> v1:172.21.15.59:6789/0 conn(0x7f4ca80a5b70 legacy=0x7f4ca8141b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.59:6789/0 2023-09-30T02:04:50.791 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:50.791+0000 7f4cacaa6700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f4ca80a5f40 0x7f4ca813af30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-09-30T02:04:50.791 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:50.791+0000 7f4ca77fe700 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f4ca8142290 msgr2=0x7f4ca80b1500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2023-09-30T02:04:50.791 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:50.791+0000 7f4ca77fe700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4ca8142290 0x7f4ca80b1500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-09-30T02:04:51.591 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:51.592+0000 7f4cacaa6700 1 -- >> v2:172.21.15.59:3300/0 conn(0x7f4ca80a5f40 msgr2=0x7f4ca813af30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.59:3300/0 2023-09-30T02:04:51.591 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:51.592+0000 7f4ca7fff700 1 -- >> v1:172.21.15.59:6789/0 conn(0x7f4ca80a5b70 legacy=0x7f4ca8141b80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.59:6789/0 2023-09-30T02:04:51.591 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:51.592+0000 7f4ca77fe700 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f4ca8142290 msgr2=0x7f4ca80b1500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2023-09-30T02:04:51.592 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:51.592+0000 7f4cacaa6700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f4ca80a5f40 0x7f4ca813af30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-09-30T02:04:51.592 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:51.592+0000 7f4ca77fe700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4ca8142290 0x7f4ca80b1500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-09-30T02:04:53.189 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:53.189+0000 7f4ca5ffb700 10 monclient(hunting): tick 2023-09-30T02:04:53.189 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:53.189+0000 7f4ca5ffb700 1 monclient(hunting): continuing hunt 2023-09-30T02:04:53.189 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:53.189+0000 7f4ca5ffb700 10 monclient(hunting): _reopen_session rank -1 2023-09-30T02:04:53.189 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:53.189+0000 7f4ca5ffb700 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f4ca8142290 msgr2=0x7f4ca80b1500 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T02:04:53.189 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:53.189+0000 7f4ca5ffb700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4ca8142290 0x7f4ca80b1500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:04:53.189 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:53.189+0000 7f4ca5ffb700 1 -- >> v2:172.21.15.59:3300/0 conn(0x7f4ca80a5f40 msgr2=0x7f4ca813af30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T02:04:53.189 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:53.189+0000 7f4ca5ffb700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f4ca80a5f40 0x7f4ca813af30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:04:53.189 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:53.189+0000 7f4ca5ffb700 1 -- >> v1:172.21.15.59:6789/0 conn(0x7f4ca80a5b70 legacy=0x7f4ca8141b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T02:04:53.189 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:53.189+0000 7f4ca5ffb700 10 monclient: _add_conns ranks=[2,1] 2023-09-30T02:04:53.189 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:53.189+0000 7f4ca5ffb700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c940036e0 0x7f4c94005b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:04:53.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:53.189+0000 7f4ca5ffb700 10 monclient(hunting): picked mon.noname-b con 0x7f4c940036e0 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:04:53.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:53.189+0000 7f4ca5ffb700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4c94006160 addr v1:172.21.15.110:6789/0 2023-09-30T02:04:53.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:53.189+0000 7f4ca5ffb700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ca813b470 con 0x7f4c94006160 2023-09-30T02:04:53.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:53.189+0000 7f4ca5ffb700 10 monclient(hunting): start opening mon connection 2023-09-30T02:04:53.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:53.189+0000 7f4ca5ffb700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca80a6360 con 0x7f4c940036e0 2023-09-30T02:04:53.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:53.189+0000 7f4ca5ffb700 10 monclient(hunting): _renew_subs 2023-09-30T02:04:53.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:53.189+0000 7f4ca7fff700 1 -- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c940036e0 msgr2=0x7f4c94005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.59:3301/0 2023-09-30T02:04:53.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:53.189+0000 7f4ca7fff700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c940036e0 0x7f4c94005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-30T02:04:53.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:53.189+0000 7f4ca77fe700 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f4c94006160 legacy=0x7f4c94008610 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2023-09-30T02:04:53.389 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:53.390+0000 7f4ca7fff700 1 -- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c940036e0 msgr2=0x7f4c94005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.59:3301/0 2023-09-30T02:04:53.389 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:53.390+0000 7f4ca7fff700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c940036e0 0x7f4c94005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-09-30T02:04:53.389 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:53.390+0000 7f4ca77fe700 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f4c94006160 legacy=0x7f4c94008610 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2023-09-30T02:04:53.790 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:53.791+0000 7f4ca7fff700 1 -- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c940036e0 msgr2=0x7f4c94005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.59:3301/0 2023-09-30T02:04:53.791 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:53.791+0000 7f4ca77fe700 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f4c94006160 legacy=0x7f4c94008610 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2023-09-30T02:04:53.791 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:53.791+0000 7f4ca7fff700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c940036e0 0x7f4c94005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-09-30T02:04:54.591 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:54.592+0000 7f4ca7fff700 1 -- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c940036e0 msgr2=0x7f4c94005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.59:3301/0 2023-09-30T02:04:54.591 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:54.592+0000 7f4ca77fe700 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f4c94006160 legacy=0x7f4c94008610 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2023-09-30T02:04:54.591 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:54.592+0000 7f4ca7fff700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c940036e0 0x7f4c94005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-09-30T02:04:56.189 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:56.189+0000 7f4ca5ffb700 10 monclient(hunting): tick 2023-09-30T02:04:56.189 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:56.189+0000 7f4ca5ffb700 1 monclient(hunting): continuing hunt 2023-09-30T02:04:56.189 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:56.189+0000 7f4ca5ffb700 10 monclient(hunting): _reopen_session rank -1 2023-09-30T02:04:56.189 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:56.189+0000 7f4ca5ffb700 1 -- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c940036e0 msgr2=0x7f4c94005b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T02:04:56.189 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:56.189+0000 7f4ca5ffb700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c940036e0 0x7f4c94005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:04:56.189 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:56.189+0000 7f4ca5ffb700 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f4c94006160 legacy=0x7f4c94008610 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T02:04:56.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:56.189+0000 7f4cacaa6700 1 -- reap_dead start 2023-09-30T02:04:56.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:56.189+0000 7f4ca5ffb700 10 monclient: _add_conns ranks=[3,2,0,1,4] 2023-09-30T02:04:56.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:56.189+0000 7f4ca5ffb700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f4c94009ef0 0x7f4c94005b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:04:56.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:56.189+0000 7f4ca5ffb700 10 monclient(hunting): picked mon.noname-a con 0x7f4c94009ef0 addr v2:172.21.15.59:3300/0 2023-09-30T02:04:56.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:56.189+0000 7f4ca5ffb700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c9400a360 0x7f4c94008610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:04:56.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:56.189+0000 7f4ca5ffb700 10 monclient(hunting): picked mon.noname-b con 0x7f4c9400a360 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:04:56.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:56.189+0000 7f4ca5ffb700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4c9400a730 addr v1:172.21.15.59:6789/0 2023-09-30T02:04:56.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:56.189+0000 7f4ca5ffb700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ca80a75a0 con 0x7f4c9400a730 2023-09-30T02:04:56.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:56.189+0000 7f4ca5ffb700 10 monclient(hunting): start opening mon connection 2023-09-30T02:04:56.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:56.189+0000 7f4ca5ffb700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ca80a6360 con 0x7f4c94009ef0 2023-09-30T02:04:56.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:56.189+0000 7f4ca5ffb700 10 monclient(hunting): start opening mon connection 2023-09-30T02:04:56.191 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:56.189+0000 7f4ca5ffb700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca80af3f0 con 0x7f4c9400a360 2023-09-30T02:04:56.191 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:56.189+0000 7f4ca5ffb700 10 monclient(hunting): _renew_subs 2023-09-30T02:04:56.191 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:56.189+0000 7f4cacaa6700 1 -- >> v1:172.21.15.59:6789/0 conn(0x7f4c9400a730 legacy=0x7f4c94008de0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.59:6789/0 2023-09-30T02:04:56.191 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:56.189+0000 7f4ca7fff700 1 -- >> v2:172.21.15.59:3300/0 conn(0x7f4c94009ef0 msgr2=0x7f4c94005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.59:3300/0 2023-09-30T02:04:56.191 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:56.189+0000 7f4ca77fe700 1 -- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c9400a360 msgr2=0x7f4c94008610 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.59:3301/0 2023-09-30T02:04:56.191 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:56.189+0000 7f4ca7fff700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f4c94009ef0 0x7f4c94005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-30T02:04:56.191 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:56.189+0000 7f4ca77fe700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c9400a360 0x7f4c94008610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-30T02:04:56.389 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:56.390+0000 7f4cacaa6700 1 -- >> v1:172.21.15.59:6789/0 conn(0x7f4c9400a730 legacy=0x7f4c94008de0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.59:6789/0 2023-09-30T02:04:56.389 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:56.390+0000 7f4ca77fe700 1 -- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c9400a360 msgr2=0x7f4c94008610 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.59:3301/0 2023-09-30T02:04:56.389 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:56.390+0000 7f4ca7fff700 1 -- >> v2:172.21.15.59:3300/0 conn(0x7f4c94009ef0 msgr2=0x7f4c94005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.59:3300/0 2023-09-30T02:04:56.389 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:56.390+0000 7f4ca77fe700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c9400a360 0x7f4c94008610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-09-30T02:04:56.390 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:56.390+0000 7f4ca7fff700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f4c94009ef0 0x7f4c94005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-09-30T02:04:56.790 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:56.791+0000 7f4ca77fe700 1 -- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c9400a360 msgr2=0x7f4c94008610 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.59:3301/0 2023-09-30T02:04:56.790 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:56.791+0000 7f4cacaa6700 1 -- >> v1:172.21.15.59:6789/0 conn(0x7f4c9400a730 legacy=0x7f4c94008de0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.59:6789/0 2023-09-30T02:04:56.791 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:56.791+0000 7f4ca7fff700 1 -- >> v2:172.21.15.59:3300/0 conn(0x7f4c94009ef0 msgr2=0x7f4c94005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.59:3300/0 2023-09-30T02:04:56.791 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:56.791+0000 7f4ca77fe700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c9400a360 0x7f4c94008610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-09-30T02:04:56.791 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:56.791+0000 7f4ca7fff700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f4c94009ef0 0x7f4c94005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-09-30T02:04:57.591 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:57.592+0000 7f4cacaa6700 1 -- >> v1:172.21.15.59:6789/0 conn(0x7f4c9400a730 legacy=0x7f4c94008de0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.59:6789/0 2023-09-30T02:04:57.591 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:57.592+0000 7f4ca77fe700 1 -- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c9400a360 msgr2=0x7f4c94008610 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.59:3301/0 2023-09-30T02:04:57.591 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:57.592+0000 7f4ca7fff700 1 -- >> v2:172.21.15.59:3300/0 conn(0x7f4c94009ef0 msgr2=0x7f4c94005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.59:3300/0 2023-09-30T02:04:57.591 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:57.592+0000 7f4ca77fe700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c9400a360 0x7f4c94008610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-09-30T02:04:57.592 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:57.592+0000 7f4ca7fff700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f4c94009ef0 0x7f4c94005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-09-30T02:04:59.189 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:59.189+0000 7f4ca5ffb700 10 monclient(hunting): tick 2023-09-30T02:04:59.189 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:59.189+0000 7f4ca5ffb700 1 monclient(hunting): continuing hunt 2023-09-30T02:04:59.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:59.189+0000 7f4ca5ffb700 10 monclient(hunting): _reopen_session rank -1 2023-09-30T02:04:59.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:59.189+0000 7f4ca5ffb700 1 -- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c9400a360 msgr2=0x7f4c94008610 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T02:04:59.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:59.189+0000 7f4ca5ffb700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c9400a360 0x7f4c94008610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:04:59.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:59.189+0000 7f4ca5ffb700 1 -- >> v2:172.21.15.59:3300/0 conn(0x7f4c94009ef0 msgr2=0x7f4c94005b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T02:04:59.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:59.189+0000 7f4ca5ffb700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f4c94009ef0 0x7f4c94005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:04:59.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:59.189+0000 7f4ca5ffb700 1 -- >> v1:172.21.15.59:6789/0 conn(0x7f4c9400a730 legacy=0x7f4c94008de0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T02:04:59.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:59.189+0000 7f4ca5ffb700 10 monclient: _add_conns ranks=[1,4] 2023-09-30T02:04:59.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:59.189+0000 7f4ca5ffb700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4c940095e0 addr v1:172.21.15.110:6789/0 2023-09-30T02:04:59.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:59.189+0000 7f4ca5ffb700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4c940117c0 0x7f4c94013ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:04:59.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:59.189+0000 7f4ca5ffb700 10 monclient(hunting): picked mon.noname-c con 0x7f4c940117c0 addr v2:172.21.15.110:3300/0 2023-09-30T02:04:59.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:59.189+0000 7f4ca5ffb700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ca813b470 con 0x7f4c940095e0 2023-09-30T02:04:59.191 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:59.189+0000 7f4ca5ffb700 10 monclient(hunting): start opening mon connection 2023-09-30T02:04:59.191 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:59.189+0000 7f4ca5ffb700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ca80a6360 con 0x7f4c940117c0 2023-09-30T02:04:59.191 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:59.189+0000 7f4ca5ffb700 10 monclient(hunting): _renew_subs 2023-09-30T02:04:59.191 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:59.189+0000 7f4ca7fff700 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f4c940095e0 legacy=0x7f4c940100a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2023-09-30T02:04:59.191 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:59.189+0000 7f4ca77fe700 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f4c940117c0 msgr2=0x7f4c94013ba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2023-09-30T02:04:59.191 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:59.189+0000 7f4ca77fe700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4c940117c0 0x7f4c94013ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-30T02:04:59.389 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:59.390+0000 7f4ca77fe700 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f4c940117c0 msgr2=0x7f4c94013ba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2023-09-30T02:04:59.390 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:59.390+0000 7f4ca7fff700 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f4c940095e0 legacy=0x7f4c940100a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2023-09-30T02:04:59.390 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:59.390+0000 7f4ca77fe700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4c940117c0 0x7f4c94013ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-09-30T02:04:59.416 INFO:tasks.ceph.mon.b.smithi110.stderr:==00:00:00:09.263 115869== Warning: unimplemented fcntl command: 1036 2023-09-30T02:04:59.723 INFO:tasks.ceph.mon.b.smithi110.stderr:==00:00:00:09.570 115869== Warning: unimplemented fcntl command: 1036 2023-09-30T02:04:59.790 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:59.791+0000 7f4ca77fe700 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f4c940117c0 msgr2=0x7f4c94013ba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2023-09-30T02:04:59.790 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:59.791+0000 7f4ca7fff700 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f4c940095e0 legacy=0x7f4c940100a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2023-09-30T02:04:59.790 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:04:59.791+0000 7f4ca77fe700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4c940117c0 0x7f4c94013ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-09-30T02:05:00.591 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:00.592+0000 7f4ca7fff700 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f4c940095e0 legacy=0x7f4c940100a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.110:6789/0 2023-09-30T02:05:00.591 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:00.592+0000 7f4ca77fe700 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f4c940117c0 msgr2=0x7f4c94013ba0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.110:3300/0 2023-09-30T02:05:00.591 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:00.592+0000 7f4ca77fe700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4c940117c0 0x7f4c94013ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-09-30T02:05:00.661 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:00:10.527 116257== Warning: unimplemented fcntl command: 1036 2023-09-30T02:05:00.679 INFO:tasks.ceph.mon.a.smithi059.stderr:==00:00:00:10.544 116256== Warning: unimplemented fcntl command: 1036 2023-09-30T02:05:01.010 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:00:10.875 116257== Warning: unimplemented fcntl command: 1036 2023-09-30T02:05:01.027 INFO:tasks.ceph.mon.a.smithi059.stderr:==00:00:00:10.893 116256== Warning: unimplemented fcntl command: 1036 2023-09-30T02:05:02.189 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:02.189+0000 7f4ca5ffb700 10 monclient(hunting): tick 2023-09-30T02:05:02.189 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:02.189+0000 7f4ca5ffb700 1 monclient(hunting): continuing hunt 2023-09-30T02:05:02.189 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:02.189+0000 7f4ca5ffb700 10 monclient(hunting): _reopen_session rank -1 2023-09-30T02:05:02.189 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:02.189+0000 7f4ca5ffb700 1 -- >> v2:172.21.15.110:3300/0 conn(0x7f4c940117c0 msgr2=0x7f4c94013ba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T02:05:02.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:02.189+0000 7f4ca5ffb700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4c940117c0 0x7f4c94013ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:02.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:02.189+0000 7f4ca5ffb700 1 -- >> v1:172.21.15.110:6789/0 conn(0x7f4c940095e0 legacy=0x7f4c940100a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T02:05:02.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:02.189+0000 7f4ca5ffb700 10 monclient: _add_conns ranks=[0,3,1,4,2] 2023-09-30T02:05:02.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:02.189+0000 7f4cacaa6700 1 -- reap_dead start 2023-09-30T02:05:02.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:02.189+0000 7f4ca5ffb700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4c9400a730 addr v1:172.21.15.59:6789/0 2023-09-30T02:05:02.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:02.189+0000 7f4ca5ffb700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f4c940095e0 0x7f4c94005b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:02.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:02.189+0000 7f4ca5ffb700 10 monclient(hunting): picked mon.noname-a con 0x7f4c940095e0 addr v2:172.21.15.59:3300/0 2023-09-30T02:05:02.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:02.189+0000 7f4ca5ffb700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4c94009ed0 addr v1:172.21.15.110:6789/0 2023-09-30T02:05:02.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:02.189+0000 7f4ca5ffb700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ca80a75a0 con 0x7f4c9400a730 2023-09-30T02:05:02.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:02.189+0000 7f4ca5ffb700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ca813b470 con 0x7f4c94009ed0 2023-09-30T02:05:02.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:02.189+0000 7f4ca5ffb700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:02.191 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:02.189+0000 7f4ca5ffb700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ca80a6360 con 0x7f4c940095e0 2023-09-30T02:05:02.191 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:02.189+0000 7f4ca5ffb700 10 monclient(hunting): _renew_subs 2023-09-30T02:05:02.191 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:02.189+0000 7f4ca77fe700 1 -- >> v2:172.21.15.59:3300/0 conn(0x7f4c940095e0 msgr2=0x7f4c94005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.59:3300/0 2023-09-30T02:05:02.191 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:02.189+0000 7f4ca7fff700 1 -- >> v1:172.21.15.59:6789/0 conn(0x7f4c9400a730 legacy=0x7f4c94008dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.59:6789/0 2023-09-30T02:05:02.191 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:02.189+0000 7f4ca77fe700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f4c940095e0 0x7f4c94005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-30T02:05:02.269 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:02.269+0000 7f4cacaa6700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f4c94009ed0 0x7f4c9400e090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.59:46618/0 (socket says 172.21.15.59:46618) 2023-09-30T02:05:02.269 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:02.269+0000 7f4cacaa6700 1 -- 172.21.15.59:0/441643280 learned_addr learned my addr 172.21.15.59:0/441643280 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:05:02.682 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:02.683+0000 7f4ca77fe700 1 --2- 172.21.15.59:0/441643280 >> v2:172.21.15.59:3300/0 conn(0x7f4c940095e0 0x7f4c94005b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:02.706 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:02.707+0000 7f4ca77fe700 10 monclient(hunting): get_auth_request con 0x7f4c940095e0 auth_method 0 2023-09-30T02:05:02.706 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:02.707+0000 7f4ca77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:02.707 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:02.707+0000 7f4ca77fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:02.707 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:02.707+0000 7f4ca77fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:02.755 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:02.755+0000 7f4ca77fe700 1 -- 172.21.15.59:0/441643280 >> v2:172.21.15.59:3300/0 conn(0x7f4c940095e0 msgr2=0x7f4c94005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-30T02:05:02.755 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:02.755+0000 7f4ca77fe700 1 -- 172.21.15.59:0/441643280 >> v2:172.21.15.59:3300/0 conn(0x7f4c940095e0 msgr2=0x7f4c94005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T02:05:02.755 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:02.755+0000 7f4ca77fe700 1 --2- 172.21.15.59:0/441643280 >> v2:172.21.15.59:3300/0 conn(0x7f4c940095e0 0x7f4c94005b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T02:05:02.755 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:02.756+0000 7f4ca77fe700 1 --2- 172.21.15.59:0/441643280 >> v2:172.21.15.59:3300/0 conn(0x7f4c940095e0 0x7f4c94005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-09-30T02:05:03.026 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:03.027+0000 7f4ca7fff700 1 -- 172.21.15.59:0/441643280 >> v1:172.21.15.59:6789/0 conn(0x7f4c9400a730 legacy=0x7f4c94008dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2023-09-30T02:05:03.027 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:03.027+0000 7f4ca7fff700 1 -- 172.21.15.59:0/441643280 >> v1:172.21.15.59:6789/0 conn(0x7f4c9400a730 legacy=0x7f4c94008dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-09-30T02:05:03.027 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:03.027+0000 7f4ca7fff700 1 --1- 172.21.15.59:0/441643280 >> v1:172.21.15.59:6789/0 conn(0x7f4c9400a730 0x7f4c94008dc0 :-1 s=OPENED pgs=1 cs=1 l=1).handle_message read tag failed 2023-09-30T02:05:03.027 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:03.027+0000 7f4ca7fff700 1 --1- 172.21.15.59:0/441643280 >> v1:172.21.15.59:6789/0 conn(0x7f4c9400a730 0x7f4c94008dc0 :-1 s=OPENED pgs=1 cs=1 l=1).fault on lossy channel, failing 2023-09-30T02:05:03.027 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:03.027+0000 7f4ca6ffd700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.59:6789/0 2023-09-30T02:05:03.254 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:03.254+0000 7f4ca77fe700 1 --2- 172.21.15.59:0/441643280 >> v2:172.21.15.59:3300/0 conn(0x7f4c940095e0 0x7f4c94005b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:03.255 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:03.256+0000 7f4ca77fe700 10 monclient(hunting): get_auth_request con 0x7f4c940095e0 auth_method 0 2023-09-30T02:05:03.255 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:03.256+0000 7f4ca77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:03.255 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:03.256+0000 7f4ca77fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:03.255 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:03.256+0000 7f4ca77fe700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T02:05:03.257 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:03.258+0000 7f4ca77fe700 1 -- 172.21.15.59:0/441643280 >> v2:172.21.15.59:3300/0 conn(0x7f4c940095e0 msgr2=0x7f4c94005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-30T02:05:03.257 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:03.258+0000 7f4ca77fe700 1 -- 172.21.15.59:0/441643280 >> v2:172.21.15.59:3300/0 conn(0x7f4c940095e0 msgr2=0x7f4c94005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T02:05:03.257 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:03.258+0000 7f4ca77fe700 1 --2- 172.21.15.59:0/441643280 >> v2:172.21.15.59:3300/0 conn(0x7f4c940095e0 0x7f4c94005b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T02:05:03.258 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:03.258+0000 7f4ca77fe700 1 --2- 172.21.15.59:0/441643280 >> v2:172.21.15.59:3300/0 conn(0x7f4c940095e0 0x7f4c94005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-09-30T02:05:04.129 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:04.130+0000 7f4ca77fe700 1 --2- 172.21.15.59:0/441643280 >> v2:172.21.15.59:3300/0 conn(0x7f4c940095e0 0x7f4c94005b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:04.152 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:04.153+0000 7f4ca77fe700 10 monclient(hunting): get_auth_request con 0x7f4c940095e0 auth_method 0 2023-09-30T02:05:04.153 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:04.153+0000 7f4ca77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:04.153 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:04.153+0000 7f4ca77fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:04.153 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:04.153+0000 7f4ca77fe700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T02:05:04.154 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:04.154+0000 7f4ca77fe700 1 -- 172.21.15.59:0/441643280 >> v2:172.21.15.59:3300/0 conn(0x7f4c940095e0 msgr2=0x7f4c94005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-30T02:05:04.154 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:04.154+0000 7f4ca77fe700 1 -- 172.21.15.59:0/441643280 >> v2:172.21.15.59:3300/0 conn(0x7f4c940095e0 msgr2=0x7f4c94005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T02:05:04.154 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:04.154+0000 7f4ca77fe700 1 --2- 172.21.15.59:0/441643280 >> v2:172.21.15.59:3300/0 conn(0x7f4c940095e0 0x7f4c94005b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T02:05:04.154 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:04.154+0000 7f4ca77fe700 1 --2- 172.21.15.59:0/441643280 >> v2:172.21.15.59:3300/0 conn(0x7f4c940095e0 0x7f4c94005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-09-30T02:05:05.189 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:05.190+0000 7f4ca5ffb700 10 monclient(hunting): tick 2023-09-30T02:05:05.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:05.190+0000 7f4ca5ffb700 1 monclient(hunting): continuing hunt 2023-09-30T02:05:05.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:05.190+0000 7f4ca5ffb700 10 monclient(hunting): _reopen_session rank -1 2023-09-30T02:05:05.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:05.190+0000 7f4ca5ffb700 1 -- 172.21.15.59:0/441643280 >> v2:172.21.15.59:3300/0 conn(0x7f4c940095e0 msgr2=0x7f4c94005b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T02:05:05.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:05.190+0000 7f4ca5ffb700 1 --2- 172.21.15.59:0/441643280 >> v2:172.21.15.59:3300/0 conn(0x7f4c940095e0 0x7f4c94005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:05.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:05.190+0000 7f4ca5ffb700 1 -- 172.21.15.59:0/441643280 >> v1:172.21.15.110:6789/0 conn(0x7f4c94009ed0 legacy=0x7f4c9400e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:05.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:05.190+0000 7f4ca5ffb700 1 -- 172.21.15.59:0/441643280 >> v1:172.21.15.59:6789/0 conn(0x7f4c9400a730 legacy=0x7f4c94008dc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-09-30T02:05:05.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:05.190+0000 7f4ca5ffb700 10 monclient: _add_conns ranks=[2,4] 2023-09-30T02:05:05.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:05.190+0000 7f4ca5ffb700 1 --2- 172.21.15.59:0/441643280 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c9400f7b0 0x7f4c9400fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:05.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:05.190+0000 7f4ca5ffb700 10 monclient(hunting): picked mon.noname-b con 0x7f4c9400f7b0 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:05:05.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:05.190+0000 7f4ca5ffb700 1 --2- 172.21.15.59:0/441643280 >> v2:172.21.15.110:3300/0 conn(0x7f4c940100c0 0x7f4c940104c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:05.191 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:05.190+0000 7f4ca5ffb700 10 monclient(hunting): picked mon.noname-c con 0x7f4c940100c0 addr v2:172.21.15.110:3300/0 2023-09-30T02:05:05.191 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:05.190+0000 7f4ca5ffb700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:05.191 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:05.190+0000 7f4ca5ffb700 1 -- 172.21.15.59:0/441643280 --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ca80a6360 con 0x7f4c940100c0 2023-09-30T02:05:05.191 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:05.190+0000 7f4ca5ffb700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:05.191 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:05.190+0000 7f4ca5ffb700 1 -- 172.21.15.59:0/441643280 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca80af3f0 con 0x7f4c9400f7b0 2023-09-30T02:05:05.191 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:05.190+0000 7f4ca5ffb700 10 monclient(hunting): _renew_subs 2023-09-30T02:05:05.192 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:05.193+0000 7f4ca77fe700 1 --2- 172.21.15.59:0/441643280 >> v2:172.21.15.110:3300/0 conn(0x7f4c940100c0 0x7f4c940104c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:05.194 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:05.195+0000 7f4ca77fe700 10 monclient(hunting): get_auth_request con 0x7f4c940100c0 auth_method 0 2023-09-30T02:05:05.194 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:05.195+0000 7f4ca77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:05.194 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:05.195+0000 7f4ca77fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:05.194 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:05.195+0000 7f4ca77fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:05.195 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:05.195+0000 7f4ca7fff700 1 --2- 172.21.15.59:0/441643280 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c9400f7b0 0x7f4c9400fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:05.195 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:05.195+0000 7f4ca77fe700 1 -- 172.21.15.59:0/441643280 >> v2:172.21.15.110:3300/0 conn(0x7f4c940100c0 msgr2=0x7f4c940104c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-09-30T02:05:05.195 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:05.195+0000 7f4ca77fe700 1 -- 172.21.15.59:0/441643280 >> v2:172.21.15.110:3300/0 conn(0x7f4c940100c0 msgr2=0x7f4c940104c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T02:05:05.195 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:05.195+0000 7f4ca77fe700 1 --2- 172.21.15.59:0/441643280 >> v2:172.21.15.110:3300/0 conn(0x7f4c940100c0 0x7f4c940104c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T02:05:05.195 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:05.195+0000 7f4ca77fe700 1 --2- 172.21.15.59:0/441643280 >> v2:172.21.15.110:3300/0 conn(0x7f4c940100c0 0x7f4c940104c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-30T02:05:05.196 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:05.196+0000 7f4ca7fff700 10 monclient(hunting): get_auth_request con 0x7f4c9400f7b0 auth_method 0 2023-09-30T02:05:05.196 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:05.196+0000 7f4ca7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:05.196 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:05.196+0000 7f4ca7fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:05.196 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:05.196+0000 7f4ca7fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:05.196 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:05.197+0000 7f4ca7fff700 1 -- 172.21.15.59:0/441643280 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c9400f7b0 msgr2=0x7f4c9400fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-30T02:05:05.196 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:05.197+0000 7f4ca7fff700 1 -- 172.21.15.59:0/441643280 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c9400f7b0 msgr2=0x7f4c9400fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T02:05:05.197 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:05.197+0000 7f4ca7fff700 1 --2- 172.21.15.59:0/441643280 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c9400f7b0 0x7f4c9400fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T02:05:05.197 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:05.197+0000 7f4ca7fff700 1 --2- 172.21.15.59:0/441643280 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c9400f7b0 0x7f4c9400fb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-30T02:05:05.395 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:05.395+0000 7f4ca77fe700 1 --2- 172.21.15.59:0/441643280 >> v2:172.21.15.110:3300/0 conn(0x7f4c940100c0 0x7f4c940104c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:05.397 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:05.396+0000 7f4ca77fe700 10 monclient(hunting): get_auth_request con 0x7f4c940100c0 auth_method 0 2023-09-30T02:05:05.397 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:05.396+0000 7f4ca77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:05.397 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:05.396+0000 7f4ca77fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:05.397 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:05.396+0000 7f4ca77fe700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T02:05:05.397 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:05.397+0000 7f4ca77fe700 1 -- 172.21.15.59:0/441643280 >> v2:172.21.15.110:3300/0 conn(0x7f4c940100c0 msgr2=0x7f4c940104c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-30T02:05:05.397 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:05.397+0000 7f4ca77fe700 1 -- 172.21.15.59:0/441643280 >> v2:172.21.15.110:3300/0 conn(0x7f4c940100c0 msgr2=0x7f4c940104c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T02:05:05.398 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:05.397+0000 7f4ca77fe700 1 --2- 172.21.15.59:0/441643280 >> v2:172.21.15.110:3300/0 conn(0x7f4c940100c0 0x7f4c940104c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T02:05:05.398 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:05.397+0000 7f4ca77fe700 1 --2- 172.21.15.59:0/441643280 >> v2:172.21.15.110:3300/0 conn(0x7f4c940100c0 0x7f4c940104c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-09-30T02:05:05.399 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:05.399+0000 7f4ca7fff700 1 --2- 172.21.15.59:0/441643280 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c9400f7b0 0x7f4c9400fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:05.416 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:05.415+0000 7f4ca7fff700 10 monclient(hunting): get_auth_request con 0x7f4c9400f7b0 auth_method 0 2023-09-30T02:05:05.416 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:05.415+0000 7f4ca7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:05.416 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:05.415+0000 7f4ca7fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:05.416 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:05.415+0000 7f4ca7fff700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T02:05:05.422 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:05.422+0000 7f4ca7fff700 1 -- 172.21.15.59:0/441643280 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c9400f7b0 msgr2=0x7f4c9400fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-09-30T02:05:05.422 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:05.422+0000 7f4ca7fff700 1 -- 172.21.15.59:0/441643280 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c9400f7b0 msgr2=0x7f4c9400fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T02:05:05.422 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:05.422+0000 7f4ca7fff700 1 --2- 172.21.15.59:0/441643280 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c9400f7b0 0x7f4c9400fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T02:05:05.422 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:05.422+0000 7f4ca7fff700 1 --2- 172.21.15.59:0/441643280 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c9400f7b0 0x7f4c9400fb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-09-30T02:05:05.799 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:05.798+0000 7f4ca77fe700 1 --2- 172.21.15.59:0/441643280 >> v2:172.21.15.110:3300/0 conn(0x7f4c940100c0 0x7f4c940104c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:05.799 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:05.798+0000 7f4ca77fe700 10 monclient(hunting): get_auth_request con 0x7f4c940100c0 auth_method 0 2023-09-30T02:05:05.799 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:05.798+0000 7f4ca77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:05.799 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:05.798+0000 7f4ca77fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:05.799 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:05.798+0000 7f4ca77fe700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T02:05:05.800 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:05.800+0000 7f4ca77fe700 1 -- 172.21.15.59:0/441643280 >> v2:172.21.15.110:3300/0 conn(0x7f4c940100c0 msgr2=0x7f4c940104c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-30T02:05:05.800 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:05.800+0000 7f4ca77fe700 1 -- 172.21.15.59:0/441643280 >> v2:172.21.15.110:3300/0 conn(0x7f4c940100c0 msgr2=0x7f4c940104c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T02:05:05.800 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:05.800+0000 7f4ca77fe700 1 --2- 172.21.15.59:0/441643280 >> v2:172.21.15.110:3300/0 conn(0x7f4c940100c0 0x7f4c940104c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T02:05:05.801 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:05.800+0000 7f4ca77fe700 1 --2- 172.21.15.59:0/441643280 >> v2:172.21.15.110:3300/0 conn(0x7f4c940100c0 0x7f4c940104c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-09-30T02:05:05.824 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:05.823+0000 7f4ca7fff700 1 --2- 172.21.15.59:0/441643280 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c9400f7b0 0x7f4c9400fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:05.824 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:05.824+0000 7f4ca7fff700 10 monclient(hunting): get_auth_request con 0x7f4c9400f7b0 auth_method 0 2023-09-30T02:05:05.824 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:05.824+0000 7f4ca7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:05.824 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:05.824+0000 7f4ca7fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:05.825 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:05.824+0000 7f4ca7fff700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T02:05:05.825 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:05.825+0000 7f4ca7fff700 1 -- 172.21.15.59:0/441643280 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c9400f7b0 msgr2=0x7f4c9400fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-30T02:05:05.826 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:05.825+0000 7f4ca7fff700 1 -- 172.21.15.59:0/441643280 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c9400f7b0 msgr2=0x7f4c9400fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T02:05:05.826 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:05.825+0000 7f4ca7fff700 1 --2- 172.21.15.59:0/441643280 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c9400f7b0 0x7f4c9400fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T02:05:05.826 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:05.825+0000 7f4ca7fff700 1 --2- 172.21.15.59:0/441643280 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c9400f7b0 0x7f4c9400fb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-09-30T02:05:06.602 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:06.602+0000 7f4ca77fe700 1 --2- 172.21.15.59:0/441643280 >> v2:172.21.15.110:3300/0 conn(0x7f4c940100c0 0x7f4c940104c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:06.603 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:06.602+0000 7f4ca77fe700 10 monclient(hunting): get_auth_request con 0x7f4c940100c0 auth_method 0 2023-09-30T02:05:06.603 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:06.602+0000 7f4ca77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:06.603 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:06.602+0000 7f4ca77fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:06.603 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:06.602+0000 7f4ca77fe700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T02:05:06.603 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:06.603+0000 7f4ca77fe700 1 -- 172.21.15.59:0/441643280 >> v2:172.21.15.110:3300/0 conn(0x7f4c940100c0 msgr2=0x7f4c940104c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-30T02:05:06.603 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:06.603+0000 7f4ca77fe700 1 -- 172.21.15.59:0/441643280 >> v2:172.21.15.110:3300/0 conn(0x7f4c940100c0 msgr2=0x7f4c940104c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T02:05:06.604 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:06.603+0000 7f4ca77fe700 1 --2- 172.21.15.59:0/441643280 >> v2:172.21.15.110:3300/0 conn(0x7f4c940100c0 0x7f4c940104c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T02:05:06.604 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:06.603+0000 7f4ca77fe700 1 --2- 172.21.15.59:0/441643280 >> v2:172.21.15.110:3300/0 conn(0x7f4c940100c0 0x7f4c940104c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-09-30T02:05:06.627 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:06.627+0000 7f4ca7fff700 1 --2- 172.21.15.59:0/441643280 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c9400f7b0 0x7f4c9400fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:06.628 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:06.627+0000 7f4ca7fff700 10 monclient(hunting): get_auth_request con 0x7f4c9400f7b0 auth_method 0 2023-09-30T02:05:06.628 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:06.627+0000 7f4ca7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:06.628 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:06.627+0000 7f4ca7fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:06.628 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:06.627+0000 7f4ca7fff700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T02:05:06.629 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:06.628+0000 7f4ca7fff700 1 -- 172.21.15.59:0/441643280 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c9400f7b0 msgr2=0x7f4c9400fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-30T02:05:06.629 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:06.628+0000 7f4ca7fff700 1 -- 172.21.15.59:0/441643280 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c9400f7b0 msgr2=0x7f4c9400fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T02:05:06.629 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:06.628+0000 7f4ca7fff700 1 --2- 172.21.15.59:0/441643280 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c9400f7b0 0x7f4c9400fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T02:05:06.629 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:06.628+0000 7f4ca7fff700 1 --2- 172.21.15.59:0/441643280 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c9400f7b0 0x7f4c9400fb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-09-30T02:05:08.189 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:08.189+0000 7f4ca5ffb700 10 monclient(hunting): tick 2023-09-30T02:05:08.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:08.189+0000 7f4ca5ffb700 1 monclient(hunting): continuing hunt 2023-09-30T02:05:08.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:08.189+0000 7f4ca5ffb700 10 monclient(hunting): _reopen_session rank -1 2023-09-30T02:05:08.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:08.189+0000 7f4ca5ffb700 1 -- 172.21.15.59:0/441643280 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c9400f7b0 msgr2=0x7f4c9400fb80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T02:05:08.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:08.189+0000 7f4ca5ffb700 1 --2- 172.21.15.59:0/441643280 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c9400f7b0 0x7f4c9400fb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:08.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:08.189+0000 7f4ca5ffb700 1 -- 172.21.15.59:0/441643280 >> v2:172.21.15.110:3300/0 conn(0x7f4c940100c0 msgr2=0x7f4c940104c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T02:05:08.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:08.189+0000 7f4ca5ffb700 1 --2- 172.21.15.59:0/441643280 >> v2:172.21.15.110:3300/0 conn(0x7f4c940100c0 0x7f4c940104c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:08.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:08.189+0000 7f4ca5ffb700 10 monclient: _add_conns ranks=[4,1,2,3,0] 2023-09-30T02:05:08.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:08.189+0000 7f4ca5ffb700 1 --2- 172.21.15.59:0/441643280 >> v2:172.21.15.110:3300/0 conn(0x7f4c94010a00 0x7f4c94010dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:08.191 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:08.189+0000 7f4ca5ffb700 10 monclient(hunting): picked mon.noname-c con 0x7f4c94010a00 addr v2:172.21.15.110:3300/0 2023-09-30T02:05:08.191 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:08.189+0000 7f4cacaa6700 1 -- 172.21.15.59:0/441643280 reap_dead start 2023-09-30T02:05:08.191 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:08.189+0000 7f4ca5ffb700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4c94009ed0 addr v1:172.21.15.110:6789/0 2023-09-30T02:05:08.191 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:08.189+0000 7f4ca5ffb700 1 --2- 172.21.15.59:0/441643280 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c9400a730 0x7f4c94005b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:08.191 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:08.189+0000 7f4ca5ffb700 10 monclient(hunting): picked mon.noname-b con 0x7f4c9400a730 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:05:08.191 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:08.189+0000 7f4ca5ffb700 1 -- 172.21.15.59:0/441643280 --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c94008610 con 0x7f4c94009ed0 2023-09-30T02:05:08.191 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:08.189+0000 7f4ca5ffb700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:08.191 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:08.189+0000 7f4ca5ffb700 1 -- 172.21.15.59:0/441643280 --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ca80a6360 con 0x7f4c94010a00 2023-09-30T02:05:08.191 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:08.189+0000 7f4ca5ffb700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:08.191 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:08.189+0000 7f4ca5ffb700 1 -- 172.21.15.59:0/441643280 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca80af3f0 con 0x7f4c9400a730 2023-09-30T02:05:08.191 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:08.189+0000 7f4ca5ffb700 10 monclient(hunting): _renew_subs 2023-09-30T02:05:08.193 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:08.190+0000 7f4ca7fff700 1 --2- 172.21.15.59:0/441643280 >> v2:172.21.15.110:3300/0 conn(0x7f4c94010a00 0x7f4c94010dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:08.193 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:08.190+0000 7f4cacaa6700 1 --2- 172.21.15.59:0/441643280 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c9400a730 0x7f4c94005b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:08.193 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:08.190+0000 7f4cacaa6700 10 monclient(hunting): get_auth_request con 0x7f4c9400a730 auth_method 0 2023-09-30T02:05:08.193 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:08.190+0000 7f4cacaa6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:08.193 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:08.190+0000 7f4cacaa6700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:08.193 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:08.190+0000 7f4cacaa6700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:08.193 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:08.190+0000 7f4ca7fff700 10 monclient(hunting): get_auth_request con 0x7f4c94010a00 auth_method 0 2023-09-30T02:05:08.194 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:08.190+0000 7f4ca7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:08.194 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:08.190+0000 7f4ca7fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:08.194 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:08.190+0000 7f4ca7fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:08.194 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:08.191+0000 7f4cacaa6700 1 -- 172.21.15.59:0/441643280 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c9400a730 msgr2=0x7f4c94005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-09-30T02:05:08.194 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:08.191+0000 7f4cacaa6700 1 -- 172.21.15.59:0/441643280 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c9400a730 msgr2=0x7f4c94005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T02:05:08.194 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:08.191+0000 7f4cacaa6700 1 --2- 172.21.15.59:0/441643280 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c9400a730 0x7f4c94005b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T02:05:08.194 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:08.192+0000 7f4cacaa6700 1 --2- 172.21.15.59:0/441643280 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c9400a730 0x7f4c94005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-30T02:05:08.194 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:08.192+0000 7f4ca7fff700 1 -- 172.21.15.59:0/441643280 >> v2:172.21.15.110:3300/0 conn(0x7f4c94010a00 msgr2=0x7f4c94010dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-30T02:05:08.194 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:08.192+0000 7f4ca7fff700 1 -- 172.21.15.59:0/441643280 >> v2:172.21.15.110:3300/0 conn(0x7f4c94010a00 msgr2=0x7f4c94010dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T02:05:08.194 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:08.192+0000 7f4ca7fff700 1 --2- 172.21.15.59:0/441643280 >> v2:172.21.15.110:3300/0 conn(0x7f4c94010a00 0x7f4c94010dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T02:05:08.194 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:08.192+0000 7f4ca7fff700 1 --2- 172.21.15.59:0/441643280 >> v2:172.21.15.110:3300/0 conn(0x7f4c94010a00 0x7f4c94010dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-30T02:05:08.394 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:08.393+0000 7f4cacaa6700 1 --2- 172.21.15.59:0/441643280 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c9400a730 0x7f4c94005b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:08.394 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:08.393+0000 7f4ca7fff700 1 --2- 172.21.15.59:0/441643280 >> v2:172.21.15.110:3300/0 conn(0x7f4c94010a00 0x7f4c94010dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:08.394 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:08.394+0000 7f4cacaa6700 10 monclient(hunting): get_auth_request con 0x7f4c9400a730 auth_method 0 2023-09-30T02:05:08.395 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:08.394+0000 7f4cacaa6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:08.395 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:08.394+0000 7f4cacaa6700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:08.395 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:08.394+0000 7f4cacaa6700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T02:05:08.395 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:08.394+0000 7f4ca7fff700 10 monclient(hunting): get_auth_request con 0x7f4c94010a00 auth_method 0 2023-09-30T02:05:08.395 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:08.394+0000 7f4ca7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:08.395 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:08.394+0000 7f4ca7fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:08.395 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:08.394+0000 7f4ca7fff700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T02:05:08.396 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:08.395+0000 7f4ca7fff700 1 -- 172.21.15.59:0/441643280 >> v2:172.21.15.110:3300/0 conn(0x7f4c94010a00 msgr2=0x7f4c94010dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-09-30T02:05:08.396 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:08.395+0000 7f4cacaa6700 1 -- 172.21.15.59:0/441643280 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c9400a730 msgr2=0x7f4c94005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-30T02:05:08.396 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:08.395+0000 7f4ca7fff700 1 -- 172.21.15.59:0/441643280 >> v2:172.21.15.110:3300/0 conn(0x7f4c94010a00 msgr2=0x7f4c94010dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T02:05:08.396 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:08.395+0000 7f4cacaa6700 1 -- 172.21.15.59:0/441643280 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c9400a730 msgr2=0x7f4c94005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T02:05:08.396 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:08.395+0000 7f4cacaa6700 1 --2- 172.21.15.59:0/441643280 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c9400a730 0x7f4c94005b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T02:05:08.396 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:08.395+0000 7f4ca7fff700 1 --2- 172.21.15.59:0/441643280 >> v2:172.21.15.110:3300/0 conn(0x7f4c94010a00 0x7f4c94010dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T02:05:08.397 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:08.395+0000 7f4ca7fff700 1 --2- 172.21.15.59:0/441643280 >> v2:172.21.15.110:3300/0 conn(0x7f4c94010a00 0x7f4c94010dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-09-30T02:05:08.397 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:08.395+0000 7f4cacaa6700 1 --2- 172.21.15.59:0/441643280 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c9400a730 0x7f4c94005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-09-30T02:05:08.797 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:08.796+0000 7f4ca7fff700 1 --2- 172.21.15.59:0/441643280 >> v2:172.21.15.110:3300/0 conn(0x7f4c94010a00 0x7f4c94010dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:08.797 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:08.796+0000 7f4cacaa6700 1 --2- 172.21.15.59:0/441643280 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c9400a730 0x7f4c94005b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:08.797 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:08.796+0000 7f4ca7fff700 10 monclient(hunting): get_auth_request con 0x7f4c94010a00 auth_method 0 2023-09-30T02:05:08.797 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:08.796+0000 7f4ca7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:08.797 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:08.796+0000 7f4ca7fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:08.797 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:08.796+0000 7f4ca7fff700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T02:05:08.797 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:08.796+0000 7f4cacaa6700 10 monclient(hunting): get_auth_request con 0x7f4c9400a730 auth_method 0 2023-09-30T02:05:08.797 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:08.796+0000 7f4cacaa6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:08.797 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:08.796+0000 7f4cacaa6700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:08.798 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:08.796+0000 7f4cacaa6700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T02:05:08.798 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:08.797+0000 7f4ca7fff700 1 -- 172.21.15.59:0/441643280 >> v2:172.21.15.110:3300/0 conn(0x7f4c94010a00 msgr2=0x7f4c94010dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-09-30T02:05:08.798 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:08.797+0000 7f4ca7fff700 1 -- 172.21.15.59:0/441643280 >> v2:172.21.15.110:3300/0 conn(0x7f4c94010a00 msgr2=0x7f4c94010dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T02:05:08.798 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:08.797+0000 7f4ca7fff700 1 --2- 172.21.15.59:0/441643280 >> v2:172.21.15.110:3300/0 conn(0x7f4c94010a00 0x7f4c94010dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T02:05:08.799 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:08.797+0000 7f4cacaa6700 1 -- 172.21.15.59:0/441643280 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c9400a730 msgr2=0x7f4c94005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-30T02:05:08.799 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:08.797+0000 7f4cacaa6700 1 -- 172.21.15.59:0/441643280 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c9400a730 msgr2=0x7f4c94005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T02:05:08.799 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:08.797+0000 7f4cacaa6700 1 --2- 172.21.15.59:0/441643280 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c9400a730 0x7f4c94005b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T02:05:08.799 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:08.797+0000 7f4ca7fff700 1 --2- 172.21.15.59:0/441643280 >> v2:172.21.15.110:3300/0 conn(0x7f4c94010a00 0x7f4c94010dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-09-30T02:05:08.799 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:08.797+0000 7f4cacaa6700 1 --2- 172.21.15.59:0/441643280 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c9400a730 0x7f4c94005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-09-30T02:05:09.601 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:09.599+0000 7f4ca7fff700 1 --2- 172.21.15.59:0/441643280 >> v2:172.21.15.110:3300/0 conn(0x7f4c94010a00 0x7f4c94010dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:09.601 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:09.599+0000 7f4cacaa6700 1 --2- 172.21.15.59:0/441643280 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c9400a730 0x7f4c94005b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:09.601 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:09.599+0000 7f4ca7fff700 10 monclient(hunting): get_auth_request con 0x7f4c94010a00 auth_method 0 2023-09-30T02:05:09.601 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:09.599+0000 7f4ca7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:09.601 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:09.599+0000 7f4ca7fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:09.601 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:09.599+0000 7f4ca7fff700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T02:05:09.602 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:09.600+0000 7f4cacaa6700 10 monclient(hunting): get_auth_request con 0x7f4c9400a730 auth_method 0 2023-09-30T02:05:09.602 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:09.600+0000 7f4cacaa6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:09.602 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:09.600+0000 7f4cacaa6700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:09.602 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:09.600+0000 7f4cacaa6700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T02:05:09.602 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:09.600+0000 7f4ca7fff700 1 -- 172.21.15.59:0/441643280 >> v2:172.21.15.110:3300/0 conn(0x7f4c94010a00 msgr2=0x7f4c94010dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-30T02:05:09.602 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:09.600+0000 7f4ca7fff700 1 -- 172.21.15.59:0/441643280 >> v2:172.21.15.110:3300/0 conn(0x7f4c94010a00 msgr2=0x7f4c94010dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T02:05:09.602 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:09.600+0000 7f4ca7fff700 1 --2- 172.21.15.59:0/441643280 >> v2:172.21.15.110:3300/0 conn(0x7f4c94010a00 0x7f4c94010dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T02:05:09.602 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:09.600+0000 7f4ca7fff700 1 --2- 172.21.15.59:0/441643280 >> v2:172.21.15.110:3300/0 conn(0x7f4c94010a00 0x7f4c94010dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-09-30T02:05:09.603 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:09.602+0000 7f4cacaa6700 1 -- 172.21.15.59:0/441643280 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c9400a730 msgr2=0x7f4c94005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-09-30T02:05:09.603 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:09.602+0000 7f4cacaa6700 1 -- 172.21.15.59:0/441643280 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c9400a730 msgr2=0x7f4c94005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T02:05:09.603 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:09.602+0000 7f4cacaa6700 1 --2- 172.21.15.59:0/441643280 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c9400a730 0x7f4c94005b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T02:05:09.603 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:09.602+0000 7f4cacaa6700 1 --2- 172.21.15.59:0/441643280 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c9400a730 0x7f4c94005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-09-30T02:05:11.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:11.190+0000 7f4ca5ffb700 10 monclient(hunting): tick 2023-09-30T02:05:11.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:11.190+0000 7f4ca5ffb700 1 monclient(hunting): continuing hunt 2023-09-30T02:05:11.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:11.190+0000 7f4ca5ffb700 10 monclient(hunting): _reopen_session rank -1 2023-09-30T02:05:11.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:11.190+0000 7f4ca5ffb700 1 -- 172.21.15.59:0/441643280 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c9400a730 msgr2=0x7f4c94005b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T02:05:11.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:11.190+0000 7f4ca5ffb700 1 --2- 172.21.15.59:0/441643280 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c9400a730 0x7f4c94005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:11.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:11.190+0000 7f4ca5ffb700 1 -- 172.21.15.59:0/441643280 >> v2:172.21.15.110:3300/0 conn(0x7f4c94010a00 msgr2=0x7f4c94010dd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T02:05:11.191 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:11.190+0000 7f4ca5ffb700 1 --2- 172.21.15.59:0/441643280 >> v2:172.21.15.110:3300/0 conn(0x7f4c94010a00 0x7f4c94010dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:11.191 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:11.190+0000 7f4ca5ffb700 1 -- 172.21.15.59:0/441643280 >> v1:172.21.15.110:6789/0 conn(0x7f4c94009ed0 legacy=0x7f4c9400e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:11.191 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:11.190+0000 7f4ca5ffb700 10 monclient: _add_conns ranks=[3,0] 2023-09-30T02:05:11.191 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:11.190+0000 7f4ca5ffb700 1 --2- 172.21.15.59:0/441643280 >> v2:172.21.15.59:3300/0 conn(0x7f4c94011310 0x7f4c940104c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:11.191 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:11.190+0000 7f4ca5ffb700 10 monclient(hunting): picked mon.noname-a con 0x7f4c94011310 addr v2:172.21.15.59:3300/0 2023-09-30T02:05:11.191 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:11.190+0000 7f4ca5ffb700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4c94008dc0 addr v1:172.21.15.59:6789/0 2023-09-30T02:05:11.191 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:11.190+0000 7f4ca5ffb700 1 -- 172.21.15.59:0/441643280 --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c940096b0 con 0x7f4c94008dc0 2023-09-30T02:05:11.191 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:11.190+0000 7f4ca5ffb700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:11.191 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:11.190+0000 7f4ca5ffb700 1 -- 172.21.15.59:0/441643280 --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ca80a6360 con 0x7f4c94011310 2023-09-30T02:05:11.191 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:11.190+0000 7f4ca5ffb700 10 monclient(hunting): _renew_subs 2023-09-30T02:05:11.192 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:11.190+0000 7f4ca7fff700 1 --2- 172.21.15.59:0/441643280 >> v2:172.21.15.59:3300/0 conn(0x7f4c94011310 0x7f4c940104c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:11.192 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:11.191+0000 7f4ca7fff700 10 monclient(hunting): get_auth_request con 0x7f4c94011310 auth_method 0 2023-09-30T02:05:11.192 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:11.191+0000 7f4ca7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:11.192 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:11.191+0000 7f4ca7fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:11.192 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:11.191+0000 7f4ca7fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:11.194 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:11.193+0000 7f4ca7fff700 1 -- 172.21.15.59:0/441643280 >> v2:172.21.15.59:3300/0 conn(0x7f4c94011310 msgr2=0x7f4c940104c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-30T02:05:11.194 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:11.193+0000 7f4ca7fff700 1 -- 172.21.15.59:0/441643280 >> v2:172.21.15.59:3300/0 conn(0x7f4c94011310 msgr2=0x7f4c940104c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T02:05:11.194 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:11.193+0000 7f4ca7fff700 1 --2- 172.21.15.59:0/441643280 >> v2:172.21.15.59:3300/0 conn(0x7f4c94011310 0x7f4c940104c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T02:05:11.194 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:11.193+0000 7f4ca7fff700 1 --2- 172.21.15.59:0/441643280 >> v2:172.21.15.59:3300/0 conn(0x7f4c94011310 0x7f4c940104c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-30T02:05:11.396 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:11.395+0000 7f4ca7fff700 1 --2- 172.21.15.59:0/441643280 >> v2:172.21.15.59:3300/0 conn(0x7f4c94011310 0x7f4c940104c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:11.396 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:11.396+0000 7f4ca7fff700 10 monclient(hunting): get_auth_request con 0x7f4c94011310 auth_method 0 2023-09-30T02:05:11.396 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:11.396+0000 7f4ca7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:11.396 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:11.396+0000 7f4ca7fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:11.396 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:11.396+0000 7f4ca7fff700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T02:05:11.397 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:11.397+0000 7f4ca7fff700 1 -- 172.21.15.59:0/441643280 >> v2:172.21.15.59:3300/0 conn(0x7f4c94011310 msgr2=0x7f4c940104c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-30T02:05:11.397 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:11.397+0000 7f4ca7fff700 1 -- 172.21.15.59:0/441643280 >> v2:172.21.15.59:3300/0 conn(0x7f4c94011310 msgr2=0x7f4c940104c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T02:05:11.397 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:11.397+0000 7f4ca7fff700 1 --2- 172.21.15.59:0/441643280 >> v2:172.21.15.59:3300/0 conn(0x7f4c94011310 0x7f4c940104c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T02:05:11.397 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:11.397+0000 7f4ca7fff700 1 --2- 172.21.15.59:0/441643280 >> v2:172.21.15.59:3300/0 conn(0x7f4c94011310 0x7f4c940104c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-09-30T02:05:11.869 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:11.868+0000 7f4ca7fff700 1 --2- 172.21.15.59:0/441643280 >> v2:172.21.15.59:3300/0 conn(0x7f4c94011310 0x7f4c940104c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:11.869 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:11.869+0000 7f4ca7fff700 10 monclient(hunting): get_auth_request con 0x7f4c94011310 auth_method 0 2023-09-30T02:05:11.869 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:11.869+0000 7f4ca7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:11.869 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:11.869+0000 7f4ca7fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:11.869 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:11.869+0000 7f4ca7fff700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T02:05:11.870 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:11.870+0000 7f4ca7fff700 1 -- 172.21.15.59:0/441643280 >> v2:172.21.15.59:3300/0 conn(0x7f4c94011310 msgr2=0x7f4c940104c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-30T02:05:11.870 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:11.870+0000 7f4ca7fff700 1 -- 172.21.15.59:0/441643280 >> v2:172.21.15.59:3300/0 conn(0x7f4c94011310 msgr2=0x7f4c940104c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T02:05:11.870 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:11.870+0000 7f4ca7fff700 1 --2- 172.21.15.59:0/441643280 >> v2:172.21.15.59:3300/0 conn(0x7f4c94011310 0x7f4c940104c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T02:05:11.870 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:11.870+0000 7f4ca7fff700 1 --2- 172.21.15.59:0/441643280 >> v2:172.21.15.59:3300/0 conn(0x7f4c94011310 0x7f4c940104c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-09-30T02:05:12.703 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:12.703+0000 7f4ca7fff700 1 --2- 172.21.15.59:0/441643280 >> v2:172.21.15.59:3300/0 conn(0x7f4c94011310 0x7f4c940104c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:12.714 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:12.714+0000 7f4ca7fff700 10 monclient(hunting): get_auth_request con 0x7f4c94011310 auth_method 0 2023-09-30T02:05:12.714 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:12.714+0000 7f4ca7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:12.715 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:12.714+0000 7f4ca7fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:12.715 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:12.714+0000 7f4ca7fff700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T02:05:12.718 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:12.718+0000 7f4ca7fff700 1 -- 172.21.15.59:0/441643280 >> v2:172.21.15.59:3300/0 conn(0x7f4c94011310 msgr2=0x7f4c940104c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-30T02:05:12.718 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:12.718+0000 7f4ca7fff700 1 -- 172.21.15.59:0/441643280 >> v2:172.21.15.59:3300/0 conn(0x7f4c94011310 msgr2=0x7f4c940104c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T02:05:12.719 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:12.718+0000 7f4ca7fff700 1 --2- 172.21.15.59:0/441643280 >> v2:172.21.15.59:3300/0 conn(0x7f4c94011310 0x7f4c940104c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T02:05:12.719 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:12.718+0000 7f4ca7fff700 1 --2- 172.21.15.59:0/441643280 >> v2:172.21.15.59:3300/0 conn(0x7f4c94011310 0x7f4c940104c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-09-30T02:05:12.820 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:12.820+0000 7f4ca6ffd700 1 -- 172.21.15.59:0/441643280 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2196728401 0 0) 0x7f4c940096b0 con 0x7f4c94008dc0 2023-09-30T02:05:12.821 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:12.820+0000 7f4ca6ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:12.821 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:12.820+0000 7f4ca6ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:12.821 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:12.820+0000 7f4ca6ffd700 10 monclient(hunting): my global_id is 4098 2023-09-30T02:05:12.821 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:12.820+0000 7f4ca6ffd700 1 -- 172.21.15.59:0/441643280 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c90003570 con 0x7f4c94008dc0 2023-09-30T02:05:12.891 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:12.889+0000 7f4ca6ffd700 1 -- 172.21.15.59:0/441643280 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3923568465 0 0) 0x7f4c90003570 con 0x7f4c94008dc0 2023-09-30T02:05:12.891 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:12.889+0000 7f4ca6ffd700 1 -- 172.21.15.59:0/441643280 >> v2:172.21.15.59:3300/0 conn(0x7f4c94011310 msgr2=0x7f4c940104c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T02:05:12.891 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:12.889+0000 7f4ca6ffd700 1 --2- 172.21.15.59:0/441643280 >> v2:172.21.15.59:3300/0 conn(0x7f4c94011310 0x7f4c940104c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:12.891 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:12.889+0000 7f4ca6ffd700 10 monclient: _finish_hunting 0 2023-09-30T02:05:12.891 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:12.889+0000 7f4ca6ffd700 1 monclient: found mon.noname-a-legacy 2023-09-30T02:05:12.892 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:12.889+0000 7f4ca6ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:05:12.892 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:12.889+0000 7f4ca6ffd700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.59:6789/0 2023-09-30T02:05:12.892 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:12.889+0000 7f4ca6ffd700 1 -- 172.21.15.59:0/441643280 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c940091c0 con 0x7f4c94008dc0 2023-09-30T02:05:12.892 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:12.889+0000 7f4ca6ffd700 10 monclient: _finish_auth 0 2023-09-30T02:05:12.892 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:12.889+0000 7f4ca6ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:05:12.892 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:12.889+0000 7f4ca6ffd700 1 -- 172.21.15.59:0/441643280 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2174283827 0 0) 0x7f4c9c007460 con 0x7f4c94008dc0 2023-09-30T02:05:12.892 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:12.889+0000 7f4ca6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:05:12.892 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:12.889+0000 7f4ca6ffd700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-09-30T02:05:12.892 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:12.889+0000 7f4ca6ffd700 10 monclient: dump: 2023-09-30T02:05:12.892 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:05:12.892 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:12.893 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:12.893 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:12.893 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:05:12.893 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:05:12.893 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:05:12.893 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:05:12.893 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:05:12.893 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:05:12.893 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:12.889+0000 7f4ca6ffd700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] but i'm connected to v1:172.21.15.59:6789/0, reconnecting 2023-09-30T02:05:12.893 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:12.889+0000 7f4ca6ffd700 10 monclient: _reopen_session rank -1 2023-09-30T02:05:12.893 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:12.889+0000 7f4ca6ffd700 1 -- 172.21.15.59:0/441643280 >> v1:172.21.15.59:6789/0 conn(0x7f4c94008dc0 legacy=0x7f4c9400f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:12.894 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:12.889+0000 7f4ca6ffd700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T02:05:12.894 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:12.889+0000 7f4cacaa6700 1 -- 172.21.15.59:0/441643280 reap_dead start 2023-09-30T02:05:12.894 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:12.889+0000 7f4ca6ffd700 1 --2- 172.21.15.59:0/441643280 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4c90003a60 0x7f4c90003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:12.894 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:12.889+0000 7f4ca6ffd700 10 monclient(hunting): picked mon.b con 0x7f4c90003a60 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:05:12.894 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:12.889+0000 7f4ca6ffd700 1 --2- 172.21.15.59:0/441643280 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c90004810 0x7f4c90004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:12.894 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:12.889+0000 7f4ca6ffd700 10 monclient(hunting): picked mon.a con 0x7f4c90004810 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:05:12.894 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:12.889+0000 7f4ca6ffd700 1 --2- 172.21.15.59:0/441643280 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c90005210 0x7f4c9000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:12.894 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:12.889+0000 7f4ca6ffd700 10 monclient(hunting): picked mon.c con 0x7f4c90005210 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:05:12.894 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:12.889+0000 7f4ca6ffd700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:12.894 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:12.889+0000 7f4ca6ffd700 1 -- 172.21.15.59:0/441643280 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca80a6360 con 0x7f4c90004810 2023-09-30T02:05:12.894 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:12.889+0000 7f4ca6ffd700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:12.894 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:12.889+0000 7f4ca6ffd700 1 -- 172.21.15.59:0/441643280 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c9000c1b0 con 0x7f4c90003a60 2023-09-30T02:05:12.895 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:12.889+0000 7f4ca6ffd700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:12.895 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:12.889+0000 7f4ca6ffd700 1 -- 172.21.15.59:0/441643280 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c9000c370 con 0x7f4c90005210 2023-09-30T02:05:12.895 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:12.889+0000 7f4ca6ffd700 10 monclient(hunting): _renew_subs 2023-09-30T02:05:12.895 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:12.889+0000 7f4ca6ffd700 10 monclient(hunting): _finish_auth 0 2023-09-30T02:05:12.895 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:12.890+0000 7f4ca7fff700 1 --2- 172.21.15.59:0/441643280 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4c90003a60 0x7f4c90003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:12.895 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:12.890+0000 7f4ca7fff700 10 monclient(hunting): get_auth_request con 0x7f4c90003a60 auth_method 0 2023-09-30T02:05:12.895 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:12.890+0000 7f4ca7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:12.895 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:12.890+0000 7f4ca7fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:12.895 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:12.890+0000 7f4ca7fff700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T02:05:12.898 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:12.898+0000 7f4ca7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:05:12.898 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:12.898+0000 7f4ca7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:05:12.898 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:12.898+0000 7f4ca7fff700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T02:05:12.966 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:12.966+0000 7f4ca7fff700 10 monclient(hunting): handle_auth_done global_id 4098 payload 293 2023-09-30T02:05:12.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:12.966+0000 7f4ca7fff700 1 -- 172.21.15.59:0/441643280 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c90005210 msgr2=0x7f4c9000ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:05:12.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:12.966+0000 7f4ca7fff700 1 --2- 172.21.15.59:0/441643280 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c90005210 0x7f4c9000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:12.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:12.966+0000 7f4ca7fff700 1 -- 172.21.15.59:0/441643280 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c90004810 msgr2=0x7f4c90004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:05:12.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:12.966+0000 7f4ca7fff700 1 --2- 172.21.15.59:0/441643280 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c90004810 0x7f4c90004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:12.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:12.966+0000 7f4ca7fff700 10 monclient: _finish_hunting 0 2023-09-30T02:05:12.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:12.966+0000 7f4ca7fff700 1 monclient: found mon.b 2023-09-30T02:05:12.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:12.966+0000 7f4ca7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:05:12.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:12.966+0000 7f4ca7fff700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:05:12.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:12.966+0000 7f4ca7fff700 1 -- 172.21.15.59:0/441643280 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c90003570 con 0x7f4c90003a60 2023-09-30T02:05:12.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:12.966+0000 7f4ca7fff700 10 monclient: _finish_auth 0 2023-09-30T02:05:12.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:12.966+0000 7f4ca7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:05:12.968 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:12.966+0000 7f4cae509700 5 monclient: authenticate success, global_id 4098 2023-09-30T02:05:12.968 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:12.966+0000 7f4cae509700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:05:12.970 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:12.970+0000 7f4ca7fff700 1 --2- 172.21.15.59:0/441643280 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4c90003a60 0x7f4c90003e50 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4c980039d0 tx=0x7f4c98005050 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ff7d2f89e9873418 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:05:13.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.008+0000 7f4ca6ffd700 1 -- 172.21.15.59:0/441643280 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c9801c070 con 0x7f4c90003a60 2023-09-30T02:05:13.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.008+0000 7f4ca6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:05:13.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.008+0000 7f4ca6ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:05:13.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.008+0000 7f4ca6ffd700 10 monclient: dump: 2023-09-30T02:05:13.009 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:05:13.009 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:13.009 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:13.009 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:13.009 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:05:13.009 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:05:13.009 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:05:13.009 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:05:13.009 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:05:13.009 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:05:13.009 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.008+0000 7f4cae509700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:05:13.010 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.008+0000 7f4ca6ffd700 1 -- 172.21.15.59:0/441643280 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c98011040 con 0x7f4c90003a60 2023-09-30T02:05:13.010 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.008+0000 7f4ca6ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:05:13.010 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.008+0000 7f4cae509700 10 monclient: get_monmap_and_config success 2023-09-30T02:05:13.010 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.008+0000 7f4cae509700 4 set_mon_vals no callback set 2023-09-30T02:05:13.010 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.008+0000 7f4ca6ffd700 1 -- 172.21.15.59:0/441643280 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c98005960 con 0x7f4c90003a60 2023-09-30T02:05:13.010 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.008+0000 7f4ca6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:05:13.010 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.008+0000 7f4ca6ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:05:13.010 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.008+0000 7f4ca6ffd700 10 monclient: dump: 2023-09-30T02:05:13.010 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:05:13.010 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:13.011 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:13.011 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:13.011 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:05:13.011 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:05:13.011 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:05:13.011 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:05:13.011 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:05:13.011 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:05:13.014 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.008+0000 7f4cae509700 10 monclient: shutdown 2023-09-30T02:05:13.014 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.008+0000 7f4cae509700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:05:13.014 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.008+0000 7f4cae509700 1 -- 172.21.15.59:0/441643280 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4c90003a60 msgr2=0x7f4c90003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:13.014 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.008+0000 7f4cae509700 1 --2- 172.21.15.59:0/441643280 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4c90003a60 0x7f4c90003e50 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4c980039d0 tx=0x7f4c98005050 comp rx=0 tx=0).stop 2023-09-30T02:05:13.014 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.008+0000 7f4cae509700 1 -- 172.21.15.59:0/441643280 shutdown_connections 2023-09-30T02:05:13.014 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.008+0000 7f4cae509700 1 --2- 172.21.15.59:0/441643280 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c90004810 0x7f4c90004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:13.014 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.008+0000 7f4cae509700 1 --2- 172.21.15.59:0/441643280 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c90005210 0x7f4c9000ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:13.014 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.008+0000 7f4cae509700 1 --2- 172.21.15.59:0/441643280 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4c90003a60 0x7f4c90003e50 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:13.014 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.008+0000 7f4cae509700 1 -- 172.21.15.59:0/441643280 >> 172.21.15.59:0/441643280 conn(0x7f4ca80631f0 msgr2=0x7f4ca80635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:05:13.014 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.008+0000 7f4cae509700 1 -- 172.21.15.59:0/441643280 shutdown_connections 2023-09-30T02:05:13.015 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.008+0000 7f4cae509700 1 -- 172.21.15.59:0/441643280 wait complete. 2023-09-30T02:05:13.015 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.009+0000 7f4cae509700 10 monclient: build_initial_monmap 2023-09-30T02:05:13.015 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.009+0000 7f4cae509700 10 monclient: monmap: 2023-09-30T02:05:13.015 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:05:13.015 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:05:13.015 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:05:13.015 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:05:13.015 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:05:13.015 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:05:13.015 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:05:13.015 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:05:13.015 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:05:13.016 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:05:13.016 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.009+0000 7f4cae509700 1 librados: starting msgr at 2023-09-30T02:05:13.016 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.009+0000 7f4cae509700 1 librados: starting objecter 2023-09-30T02:05:13.016 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.010+0000 7f4cae509700 1 Processor -- start 2023-09-30T02:05:13.016 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.010+0000 7f4cae509700 1 -- start start 2023-09-30T02:05:13.016 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.010+0000 7f4cae509700 1 librados: setting wanted keys 2023-09-30T02:05:13.016 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.010+0000 7f4cae509700 1 librados: calling monclient init 2023-09-30T02:05:13.016 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.010+0000 7f4cae509700 10 monclient: init 2023-09-30T02:05:13.016 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.010+0000 7f4cae509700 10 monclient: _reopen_session rank -1 2023-09-30T02:05:13.016 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.010+0000 7f4cae509700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T02:05:13.016 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.010+0000 7f4cae509700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c90003a60 0x7f4ca8140540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:13.016 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.010+0000 7f4cae509700 10 monclient(hunting): picked mon.noname-c con 0x7f4c90003a60 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:05:13.017 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.010+0000 7f4cae509700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4c90004810 0x7f4ca81507f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:13.017 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.010+0000 7f4cae509700 10 monclient(hunting): picked mon.noname-b con 0x7f4c90004810 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:05:13.017 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.010+0000 7f4cae509700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c90005210 0x7f4ca8150d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:13.017 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.010+0000 7f4cae509700 10 monclient(hunting): picked mon.noname-a con 0x7f4c90005210 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:05:13.017 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.010+0000 7f4cae509700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:13.017 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.010+0000 7f4cae509700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca80a9cc0 con 0x7f4c90005210 2023-09-30T02:05:13.017 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.010+0000 7f4cae509700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:13.017 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.010+0000 7f4cae509700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca814ae70 con 0x7f4c90004810 2023-09-30T02:05:13.017 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.010+0000 7f4cae509700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:13.017 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.010+0000 7f4cae509700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca8141170 con 0x7f4c90003a60 2023-09-30T02:05:13.017 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.010+0000 7f4cae509700 10 monclient(hunting): _renew_subs 2023-09-30T02:05:13.018 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.010+0000 7f4cae509700 10 monclient(hunting): authenticate will time out at 1758.313900s 2023-09-30T02:05:13.018 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.011+0000 7f4ca7fff700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c90003a60 0x7f4ca8140540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:13.018 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.012+0000 7f4ca7fff700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c90003a60 0x7f4ca8140540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:55190/0 (socket says 172.21.15.59:55190) 2023-09-30T02:05:13.018 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.012+0000 7f4ca7fff700 1 -- 172.21.15.59:0/646815126 learned_addr learned my addr 172.21.15.59:0/646815126 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:05:13.018 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.012+0000 7f4ca7fff700 10 monclient(hunting): get_auth_request con 0x7f4c90003a60 auth_method 0 2023-09-30T02:05:13.018 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.012+0000 7f4ca7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:13.018 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.012+0000 7f4ca7fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:13.018 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.012+0000 7f4ca7fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:13.018 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.013+0000 7f4ca7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:05:13.018 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.013+0000 7f4ca7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:05:13.018 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.013+0000 7f4ca7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:05:13.033 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.032+0000 7f4ca77fe700 1 --2- 172.21.15.59:0/646815126 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4c90004810 0x7f4ca81507f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:13.033 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.033+0000 7f4ca77fe700 10 monclient(hunting): get_auth_request con 0x7f4c90004810 auth_method 0 2023-09-30T02:05:13.033 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.033+0000 7f4ca77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:13.033 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.033+0000 7f4ca77fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:13.033 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.033+0000 7f4ca77fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:13.034 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.034+0000 7f4ca77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:05:13.034 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.034+0000 7f4ca77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:05:13.034 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.034+0000 7f4ca77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:05:13.036 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.036+0000 7f4ca7fff700 10 monclient(hunting): handle_auth_done global_id 4100 payload 824 2023-09-30T02:05:13.036 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.036+0000 7f4ca7fff700 1 -- 172.21.15.59:0/646815126 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4c90004810 msgr2=0x7f4ca81507f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:13.037 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.036+0000 7f4ca7fff700 1 --2- 172.21.15.59:0/646815126 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4c90004810 0x7f4ca81507f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:13.037 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.036+0000 7f4ca7fff700 1 -- 172.21.15.59:0/646815126 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c90005210 msgr2=0x7f4ca8150d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:13.037 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.036+0000 7f4ca7fff700 1 --2- 172.21.15.59:0/646815126 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c90005210 0x7f4ca8150d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:13.037 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.036+0000 7f4ca7fff700 10 monclient: _finish_hunting 0 2023-09-30T02:05:13.037 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.036+0000 7f4ca7fff700 1 monclient: found mon.noname-c 2023-09-30T02:05:13.037 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.036+0000 7f4ca7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:05:13.037 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.036+0000 7f4ca7fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.59:3301/0 2023-09-30T02:05:13.037 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.036+0000 7f4ca7fff700 1 -- 172.21.15.59:0/646815126 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ca80a8e70 con 0x7f4c90003a60 2023-09-30T02:05:13.041 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.041+0000 7f4ca7fff700 1 --2- 172.21.15.59:0/646815126 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c90003a60 0x7f4ca8140540 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4c9000c340 tx=0x7f4c9801e000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:05:13.042 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.041+0000 7f4c8ffff700 10 client.?.objecter ms_handle_connect 0x7f4c90003a60 2023-09-30T02:05:13.042 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.041+0000 7f4c8ffff700 10 client.?.objecter resend_mon_ops 2023-09-30T02:05:13.092 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.091+0000 7f4c8ffff700 1 -- 172.21.15.59:0/646815126 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c98027070 con 0x7f4c90003a60 2023-09-30T02:05:13.092 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.091+0000 7f4c8ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:05:13.092 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.091+0000 7f4c8ffff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:05:13.092 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.091+0000 7f4c8ffff700 10 monclient: dump: 2023-09-30T02:05:13.092 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:05:13.092 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:13.092 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:13.092 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:13.092 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:05:13.093 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:05:13.093 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:05:13.093 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:05:13.093 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:05:13.093 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:05:13.093 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.091+0000 7f4c8ffff700 10 monclient: _finish_auth 0 2023-09-30T02:05:13.093 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.091+0000 7f4c8ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:05:13.093 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.091+0000 7f4c8ffff700 1 -- 172.21.15.59:0/646815126 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c98005960 con 0x7f4c90003a60 2023-09-30T02:05:13.093 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.091+0000 7f4c8ffff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:05:13.093 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.091+0000 7f4c8ffff700 1 -- 172.21.15.59:0/646815126 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c98017660 con 0x7f4c90003a60 2023-09-30T02:05:13.093 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.091+0000 7f4cae509700 5 monclient: authenticate success, global_id 4100 2023-09-30T02:05:13.093 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.091+0000 7f4c8ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:05:13.094 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.091+0000 7f4c8ffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:05:13.094 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.091+0000 7f4c8ffff700 10 monclient: dump: 2023-09-30T02:05:13.094 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:05:13.094 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:13.094 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:13.094 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:13.094 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:05:13.094 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:05:13.094 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:05:13.094 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:05:13.094 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:05:13.095 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:05:13.095 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.091+0000 7f4cae509700 10 monclient: _renew_subs 2023-09-30T02:05:13.095 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.091+0000 7f4ca67fc700 4 set_mon_vals no callback set 2023-09-30T02:05:13.095 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.091+0000 7f4cae509700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:05:13.095 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.091+0000 7f4cae509700 1 -- 172.21.15.59:0/646815126 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ca80aa150 con 0x7f4c90003a60 2023-09-30T02:05:13.095 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.091+0000 7f4cae509700 10 client.4100.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:05:13.095 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.091+0000 7f4cae509700 10 monclient: _renew_subs 2023-09-30T02:05:13.095 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.091+0000 7f4cae509700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:05:13.095 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.091+0000 7f4cae509700 1 -- 172.21.15.59:0/646815126 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ca80a9860 con 0x7f4c90003a60 2023-09-30T02:05:13.095 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.091+0000 7f4cae509700 1 librados: init done 2023-09-30T02:05:13.096 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.092+0000 7f4cae509700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:05:13.096 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.092+0000 7f4cae509700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:05:13.096 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.092+0000 7f4cae509700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:05:13.096 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.092+0000 7f4cae509700 1 -- 172.21.15.59:0/646815126 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ca80a9fa0 con 0x7f4c90003a60 2023-09-30T02:05:13.226 INFO:tasks.ceph.mgr.y.smithi059.stderr:2023-09-30T02:05:13.225+0000 7fc8624c21c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-09-30T02:05:13.351 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.350+0000 7f4c8ffff700 1 -- 172.21.15.59:0/646815126 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f4c98017cd0 con 0x7f4c90003a60 2023-09-30T02:05:13.351 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.350+0000 7f4c8ffff700 1 -- 172.21.15.59:0/646815126 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f4c9802c910 con 0x7f4c90003a60 2023-09-30T02:05:13.351 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.350+0000 7f4c8ffff700 10 client.4100.objecter ms_dispatch 0x7f4ca8001550 osd_map(1..1 src has 1..1) v4 2023-09-30T02:05:13.351 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.350+0000 7f4c8ffff700 3 client.4100.objecter handle_osd_map got epochs [1,1] > 0 2023-09-30T02:05:13.352 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.350+0000 7f4c8ffff700 3 client.4100.objecter handle_osd_map decoding full epoch 1 2023-09-30T02:05:13.352 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.351+0000 7f4c8ffff700 20 client.4100.objecter dump_active .. 0 homeless 2023-09-30T02:05:13.379 INFO:tasks.ceph.mgr.x.smithi110.stderr:2023-09-30T02:05:13.379+0000 7fae141da1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-09-30T02:05:13.423 INFO:tasks.ceph.mgr.y.smithi059.stderr:2023-09-30T02:05:13.423+0000 7fc8624c21c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-09-30T02:05:13.432 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.431+0000 7f4c8ffff700 1 -- 172.21.15.59:0/646815126 <== mon.2 v2:172.21.15.59:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f4c98031060 con 0x7f4c90003a60 2023-09-30T02:05:13.432 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.431+0000 7f4c8ffff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:05:13.432 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.431+0000 7f4c8ffff700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:05:13.517 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.517+0000 7f4cae509700 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2023-09-30T02:05:13.518 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.517+0000 7f4cae509700 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2023-09-30T02:05:13.518 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.517+0000 7f4cae509700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:05:13.518 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.517+0000 7f4cae509700 1 -- 172.21.15.59:0/646815126 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f4ca8064a00 con 0x7f4c90003a60 2023-09-30T02:05:13.545 INFO:tasks.ceph.mgr.x.smithi110.stderr:2023-09-30T02:05:13.545+0000 7fae141da1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-09-30T02:05:13.821 INFO:tasks.ceph.mgr.y.smithi059.stderr:2023-09-30T02:05:13.821+0000 7fc8624c21c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-09-30T02:05:13.860 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.860+0000 7f4c8ffff700 1 -- 172.21.15.59:0/646815126 <== mon.2 v2:172.21.15.59:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f4c9800a410 con 0x7f4c90003a60 2023-09-30T02:05:13.860 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.860+0000 7f4c8ffff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2023-09-30T02:05:13.861 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.860+0000 7f4c8ffff700 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2023-09-30T02:05:13.862 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.861+0000 7f4cae509700 10 librados: watch_flush enter 2023-09-30T02:05:13.862 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.861+0000 7f4cae509700 10 librados: watch_flush exit 2023-09-30T02:05:13.862 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.861+0000 7f4cae509700 20 client.4100.objecter shutdown clearing up homeless session... 2023-09-30T02:05:13.862 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.861+0000 7f4cae509700 10 client.4100.objecter successfully canceled tick 2023-09-30T02:05:13.862 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.861+0000 7f4cae509700 10 monclient: shutdown 2023-09-30T02:05:13.862 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.861+0000 7f4cae509700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:05:13.862 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.861+0000 7f4cae509700 1 -- 172.21.15.59:0/646815126 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c90003a60 msgr2=0x7f4ca8140540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:13.862 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.861+0000 7f4cae509700 1 --2- 172.21.15.59:0/646815126 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c90003a60 0x7f4ca8140540 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4c9000c340 tx=0x7f4c9801e000 comp rx=0 tx=0).stop 2023-09-30T02:05:13.862 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.861+0000 7f4cae509700 1 -- 172.21.15.59:0/646815126 shutdown_connections 2023-09-30T02:05:13.863 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.861+0000 7f4cae509700 1 --2- 172.21.15.59:0/646815126 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c90005210 0x7f4ca8150d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:13.863 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.861+0000 7f4cae509700 1 --2- 172.21.15.59:0/646815126 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c90003a60 0x7f4ca8140540 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:13.863 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.861+0000 7f4cae509700 1 --2- 172.21.15.59:0/646815126 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4c90004810 0x7f4ca81507f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:13.863 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.861+0000 7f4cae509700 1 -- 172.21.15.59:0/646815126 >> 172.21.15.59:0/646815126 conn(0x7f4ca80631f0 msgr2=0x7f4ca8141b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:05:13.863 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.861+0000 7f4cae509700 1 -- 172.21.15.59:0/646815126 shutdown_connections 2023-09-30T02:05:13.863 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.861+0000 7f4cae509700 1 -- 172.21.15.59:0/646815126 wait complete. 2023-09-30T02:05:13.863 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:13.861+0000 7f4cae509700 1 librados: shutdown 2023-09-30T02:05:13.863 INFO:teuthology.orchestra.run.smithi059.stderr:adjusted tunables profile to default 2023-09-30T02:05:13.872 INFO:tasks.ceph.mgr.x.smithi110.stderr:2023-09-30T02:05:13.873+0000 7fae141da1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-09-30T02:05:13.877 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-09-30T02:05:13.877 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2023-09-30T02:05:13.877 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-09-30T02:05:13.909 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2023-09-30T02:05:13.909 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-09-30T02:05:13.982 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2023-09-30T02:05:13.982 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-09-30T02:05:14.055 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2023-09-30T02:05:14.055 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-09-30T02:05:14.125 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-09-30T02:05:14.125 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-09-30T02:05:14.153 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-09-30T02:05:14.153 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-09-30T02:05:14.221 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-09-30T02:05:14.221 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-09-30T02:05:14.287 DEBUG:teuthology.orchestra.run.smithi110:> set -ex 2023-09-30T02:05:14.287 DEBUG:teuthology.orchestra.run.smithi110:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-09-30T02:05:14.354 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph --cluster ceph osd new efdab44b-be57-4b81-8567-ff2deec1d3e3 0 2023-09-30T02:05:14.378 INFO:tasks.ceph.mgr.x.smithi110.stderr:2023-09-30T02:05:14.378+0000 7fae141da1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-09-30T02:05:14.421 INFO:tasks.ceph.mgr.y.smithi059.stderr:2023-09-30T02:05:14.420+0000 7fc8624c21c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-09-30T02:05:14.495 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.495+0000 7f974e7f9700 10 monclient: get_monmap_and_config 2023-09-30T02:05:14.495 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.495+0000 7f974e7f9700 10 monclient: build_initial_monmap 2023-09-30T02:05:14.495 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.495+0000 7f974e7f9700 10 monclient: monmap: 2023-09-30T02:05:14.495 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2023-09-30T02:05:14.495 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:14.495 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2023-09-30T02:05:14.496350+0000 2023-09-30T02:05:14.496 INFO:teuthology.orchestra.run.smithi110.stderr:created 2023-09-30T02:05:14.496350+0000 2023-09-30T02:05:14.496 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2023-09-30T02:05:14.496 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 3 2023-09-30T02:05:14.496 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:05:14.496 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:05:14.496 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:05:14.496 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:05:14.496 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:05:14.496 INFO:teuthology.orchestra.run.smithi110.stderr: 2023-09-30T02:05:14.498 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.496+0000 7f974e7f9700 1 Processor -- start 2023-09-30T02:05:14.498 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.496+0000 7f974e7f9700 1 -- start start 2023-09-30T02:05:14.498 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.496+0000 7f974e7f9700 10 monclient: init 2023-09-30T02:05:14.498 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.496+0000 7f974e7f9700 10 monclient: _reopen_session rank -1 2023-09-30T02:05:14.498 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.496+0000 7f974e7f9700 10 monclient: _add_conns ranks=[4,0,1,2,3] 2023-09-30T02:05:14.499 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.496+0000 7f974e7f9700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f97480a5b70 0x7f97480b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:14.499 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.496+0000 7f974e7f9700 10 monclient(hunting): picked mon.noname-c con 0x7f97480a5b70 addr v2:172.21.15.110:3300/0 2023-09-30T02:05:14.499 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.496+0000 7f974e7f9700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f97480a5f40 addr v1:172.21.15.59:6789/0 2023-09-30T02:05:14.499 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.496+0000 7f974e7f9700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9748142290 addr v1:172.21.15.110:6789/0 2023-09-30T02:05:14.499 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.496+0000 7f974e7f9700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97480a75a0 con 0x7f97480a5f40 2023-09-30T02:05:14.499 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.496+0000 7f974e7f9700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97480af3f0 con 0x7f9748142290 2023-09-30T02:05:14.499 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.496+0000 7f974e7f9700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:14.499 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.496+0000 7f974e7f9700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97480a63e0 con 0x7f97480a5b70 2023-09-30T02:05:14.499 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.496+0000 7f974e7f9700 10 monclient(hunting): _renew_subs 2023-09-30T02:05:14.500 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.496+0000 7f974e7f9700 10 monclient(hunting): authenticate will time out at 1763.956904s 2023-09-30T02:05:14.500 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.497+0000 7f9747fff700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f97480a5b70 0x7f97480b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:14.500 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.497+0000 7f97477fe700 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f97480a5f40 0x7f9748141b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.110:49526/0 (socket says 172.21.15.110:49526) 2023-09-30T02:05:14.500 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.497+0000 7f97477fe700 1 -- 172.21.15.110:0/2042851609 learned_addr learned my addr 172.21.15.110:0/2042851609 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-09-30T02:05:14.500 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.498+0000 7f9747fff700 10 monclient(hunting): get_auth_request con 0x7f97480a5b70 auth_method 0 2023-09-30T02:05:14.500 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.498+0000 7f9747fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:14.500 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.498+0000 7f9747fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:14.500 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.498+0000 7f9747fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:14.500 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.498+0000 7f9747fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:05:14.500 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.498+0000 7f9747fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:05:14.500 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.498+0000 7f9747fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:05:14.501 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.500+0000 7f9747fff700 10 monclient(hunting): handle_auth_done global_id 4111 payload 274 2023-09-30T02:05:14.501 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.500+0000 7f9747fff700 1 -- 172.21.15.110:0/2042851609 >> v1:172.21.15.110:6789/0 conn(0x7f9748142290 legacy=0x7f974813af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:05:14.501 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.500+0000 7f9747fff700 1 -- 172.21.15.110:0/2042851609 >> v1:172.21.15.59:6789/0 conn(0x7f97480a5f40 legacy=0x7f9748141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:14.501 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.500+0000 7f9747fff700 10 monclient: _finish_hunting 0 2023-09-30T02:05:14.501 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.500+0000 7f9747fff700 1 monclient: found mon.noname-c 2023-09-30T02:05:14.501 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.500+0000 7f9747fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:05:14.502 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.500+0000 7f9747fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3300/0 2023-09-30T02:05:14.502 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.500+0000 7f9747fff700 1 -- 172.21.15.110:0/2042851609 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f974813b640 con 0x7f97480a5b70 2023-09-30T02:05:14.502 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.502+0000 7f9747fff700 1 --2- 172.21.15.110:0/2042851609 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f97480a5b70 0x7f97480b1500 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f97380098e0 tx=0x7f97380091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=790843c937157924 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:05:14.512 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.511+0000 7f9746ffd700 1 -- 172.21.15.110:0/2042851609 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9738009e80 con 0x7f97480a5b70 2023-09-30T02:05:14.512 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.511+0000 7f9746ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:05:14.512 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.511+0000 7f9746ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:05:14.513 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.511+0000 7f9746ffd700 10 monclient: dump: 2023-09-30T02:05:14.513 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2023-09-30T02:05:14.513 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:14.513 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:14.513 INFO:teuthology.orchestra.run.smithi110.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:14.513 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2023-09-30T02:05:14.513 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 3 2023-09-30T02:05:14.513 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:05:14.513 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:05:14.513 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:05:14.513 INFO:teuthology.orchestra.run.smithi110.stderr: 2023-09-30T02:05:14.514 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.511+0000 7f9746ffd700 10 monclient: _finish_auth 0 2023-09-30T02:05:14.514 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.511+0000 7f9746ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:05:14.514 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.511+0000 7f9746ffd700 1 -- 172.21.15.110:0/2042851609 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f973800b820 con 0x7f97480a5b70 2023-09-30T02:05:14.514 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.511+0000 7f974e7f9700 5 monclient: authenticate success, global_id 4111 2023-09-30T02:05:14.514 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.511+0000 7f974e7f9700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:05:14.514 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.511+0000 7f9746ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:05:14.514 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.511+0000 7f9746ffd700 1 -- 172.21.15.110:0/2042851609 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f973800bba0 con 0x7f97480a5b70 2023-09-30T02:05:14.514 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.511+0000 7f9746ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:05:14.514 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.511+0000 7f9746ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:05:14.514 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.511+0000 7f9746ffd700 10 monclient: dump: 2023-09-30T02:05:14.514 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2023-09-30T02:05:14.515 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:14.515 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:14.515 INFO:teuthology.orchestra.run.smithi110.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:14.515 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2023-09-30T02:05:14.515 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 3 2023-09-30T02:05:14.515 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:05:14.515 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:05:14.515 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:05:14.515 INFO:teuthology.orchestra.run.smithi110.stderr: 2023-09-30T02:05:14.515 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.511+0000 7f974e7f9700 10 monclient: get_monmap_and_config success 2023-09-30T02:05:14.515 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.511+0000 7f974e7f9700 4 set_mon_vals no callback set 2023-09-30T02:05:14.515 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.512+0000 7f974e7f9700 10 monclient: shutdown 2023-09-30T02:05:14.516 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.512+0000 7f974e7f9700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:05:14.516 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.512+0000 7f974e7f9700 1 -- 172.21.15.110:0/2042851609 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f97480a5b70 msgr2=0x7f97480b1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:14.516 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.512+0000 7f974e7f9700 1 --2- 172.21.15.110:0/2042851609 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f97480a5b70 0x7f97480b1500 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f97380098e0 tx=0x7f97380091a0 comp rx=0 tx=0).stop 2023-09-30T02:05:14.516 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.512+0000 7f974e7f9700 1 -- 172.21.15.110:0/2042851609 shutdown_connections 2023-09-30T02:05:14.516 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.512+0000 7f974e7f9700 1 --2- 172.21.15.110:0/2042851609 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f97480a5b70 0x7f97480b1500 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:14.516 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.512+0000 7f974e7f9700 1 -- 172.21.15.110:0/2042851609 >> 172.21.15.110:0/2042851609 conn(0x7f97480631f0 msgr2=0x7f97480635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:05:14.516 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.512+0000 7f974e7f9700 1 -- 172.21.15.110:0/2042851609 shutdown_connections 2023-09-30T02:05:14.516 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.512+0000 7f974e7f9700 1 -- 172.21.15.110:0/2042851609 wait complete. 2023-09-30T02:05:14.520 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.513+0000 7f974e7f9700 10 monclient: build_initial_monmap 2023-09-30T02:05:14.520 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.513+0000 7f974e7f9700 10 monclient: monmap: 2023-09-30T02:05:14.520 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2023-09-30T02:05:14.520 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:05:14.520 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2023-09-30T02:05:14.520 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2023-09-30T02:05:14.520 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2023-09-30T02:05:14.520 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2023-09-30T02:05:14.520 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:05:14.520 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:05:14.521 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:05:14.521 INFO:teuthology.orchestra.run.smithi110.stderr: 2023-09-30T02:05:14.521 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.513+0000 7f974e7f9700 1 librados: starting msgr at 2023-09-30T02:05:14.521 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.513+0000 7f974e7f9700 1 librados: starting objecter 2023-09-30T02:05:14.521 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.514+0000 7f974e7f9700 1 Processor -- start 2023-09-30T02:05:14.521 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.514+0000 7f974e7f9700 1 -- start start 2023-09-30T02:05:14.521 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.514+0000 7f974e7f9700 1 librados: setting wanted keys 2023-09-30T02:05:14.521 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.514+0000 7f974e7f9700 1 librados: calling monclient init 2023-09-30T02:05:14.521 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.514+0000 7f974e7f9700 10 monclient: init 2023-09-30T02:05:14.521 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.514+0000 7f974e7f9700 10 monclient: _reopen_session rank -1 2023-09-30T02:05:14.521 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.514+0000 7f974e7f9700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T02:05:14.521 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.514+0000 7f974e7f9700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97480a5b70 0x7f9748151560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:14.522 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.514+0000 7f974e7f9700 10 monclient(hunting): picked mon.noname-a con 0x7f97480a5b70 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:05:14.522 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.514+0000 7f974e7f9700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f97480a5f40 0x7f9748151aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:14.522 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.514+0000 7f974e7f9700 10 monclient(hunting): picked mon.noname-b con 0x7f97480a5f40 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:05:14.522 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.514+0000 7f974e7f9700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9748142290 0x7f974813ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:14.522 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.515+0000 7f974e7f9700 10 monclient(hunting): picked mon.noname-c con 0x7f9748142290 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:05:14.522 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.515+0000 7f974e7f9700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:14.522 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.515+0000 7f974e7f9700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f974813c820 con 0x7f97480a5b70 2023-09-30T02:05:14.522 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.515+0000 7f974e7f9700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:14.522 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.515+0000 7f974e7f9700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97480a9970 con 0x7f97480a5f40 2023-09-30T02:05:14.522 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.515+0000 7f974e7f9700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:14.522 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.515+0000 7f974e7f9700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97480a9be0 con 0x7f9748142290 2023-09-30T02:05:14.522 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.515+0000 7f974e7f9700 10 monclient(hunting): _renew_subs 2023-09-30T02:05:14.523 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.515+0000 7f974e7f9700 10 monclient(hunting): authenticate will time out at 1763.975043s 2023-09-30T02:05:14.523 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.515+0000 7f97477fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f97480a5f40 0x7f9748151aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:14.523 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.515+0000 7f9747fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97480a5b70 0x7f9748151560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:14.523 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.515+0000 7f974cd96700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9748142290 0x7f974813ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:14.523 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.515+0000 7f97477fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f97480a5f40 0x7f9748151aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:51576/0 (socket says 172.21.15.110:51576) 2023-09-30T02:05:14.523 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.515+0000 7f97477fe700 1 -- 172.21.15.110:0/1511563063 learned_addr learned my addr 172.21.15.110:0/1511563063 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-09-30T02:05:14.523 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.515+0000 7f97477fe700 10 monclient(hunting): get_auth_request con 0x7f97480a5f40 auth_method 0 2023-09-30T02:05:14.523 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.515+0000 7f97477fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:14.523 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.515+0000 7f97477fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:14.523 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.515+0000 7f97477fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:14.523 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.516+0000 7f9747fff700 10 monclient(hunting): get_auth_request con 0x7f97480a5b70 auth_method 0 2023-09-30T02:05:14.523 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.516+0000 7f9747fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:14.524 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.516+0000 7f9747fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:14.524 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.516+0000 7f9747fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:14.524 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.516+0000 7f974cd96700 10 monclient(hunting): get_auth_request con 0x7f9748142290 auth_method 0 2023-09-30T02:05:14.524 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.516+0000 7f974cd96700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:14.524 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.516+0000 7f974cd96700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:14.524 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.516+0000 7f974cd96700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:14.524 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.516+0000 7f97477fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:05:14.524 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.516+0000 7f97477fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:05:14.524 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.516+0000 7f97477fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:05:14.524 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.516+0000 7f974cd96700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:05:14.524 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.516+0000 7f974cd96700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:05:14.524 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.516+0000 7f974cd96700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:05:14.525 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.516+0000 7f9747fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:05:14.525 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.516+0000 7f9747fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:05:14.525 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.516+0000 7f9747fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:05:14.525 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.519+0000 7f97477fe700 10 monclient(hunting): handle_auth_done global_id 4114 payload 824 2023-09-30T02:05:14.525 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.519+0000 7f97477fe700 1 -- 172.21.15.110:0/1511563063 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9748142290 msgr2=0x7f974813ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:14.525 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.519+0000 7f97477fe700 1 --2- 172.21.15.110:0/1511563063 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9748142290 0x7f974813ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:14.525 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.519+0000 7f97477fe700 1 -- 172.21.15.110:0/1511563063 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97480a5b70 msgr2=0x7f9748151560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:14.525 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.519+0000 7f97477fe700 1 --2- 172.21.15.110:0/1511563063 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97480a5b70 0x7f9748151560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:14.525 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.519+0000 7f97477fe700 10 monclient: _finish_hunting 0 2023-09-30T02:05:14.525 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.519+0000 7f97477fe700 1 monclient: found mon.noname-b 2023-09-30T02:05:14.525 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.519+0000 7f97477fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:05:14.525 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.519+0000 7f97477fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:05:14.526 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.519+0000 7f97477fe700 1 -- 172.21.15.110:0/1511563063 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9738009420 con 0x7f97480a5f40 2023-09-30T02:05:14.526 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.521+0000 7f97477fe700 1 --2- 172.21.15.110:0/1511563063 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f97480a5f40 0x7f9748151aa0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f973c00c7c0 tx=0x7f973c007f30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:05:14.527 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.521+0000 7f972ffff700 10 client.?.objecter ms_handle_connect 0x7f97480a5f40 2023-09-30T02:05:14.527 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.521+0000 7f972ffff700 10 client.?.objecter resend_mon_ops 2023-09-30T02:05:14.527 INFO:tasks.ceph.mgr.x.smithi110.stderr:2023-09-30T02:05:14.523+0000 7fae141da1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-09-30T02:05:14.528 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.525+0000 7f972ffff700 1 -- 172.21.15.110:0/1511563063 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f973c005b30 con 0x7f97480a5f40 2023-09-30T02:05:14.529 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.525+0000 7f972ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:05:14.529 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.525+0000 7f972ffff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:05:14.529 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.525+0000 7f972ffff700 10 monclient: dump: 2023-09-30T02:05:14.529 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2023-09-30T02:05:14.529 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:14.529 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:14.529 INFO:teuthology.orchestra.run.smithi110.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:14.529 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2023-09-30T02:05:14.529 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 3 2023-09-30T02:05:14.529 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:05:14.529 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:05:14.529 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:05:14.530 INFO:teuthology.orchestra.run.smithi110.stderr: 2023-09-30T02:05:14.530 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.525+0000 7f972ffff700 10 monclient: _finish_auth 0 2023-09-30T02:05:14.530 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.525+0000 7f972ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:05:14.530 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.525+0000 7f974e7f9700 5 monclient: authenticate success, global_id 4114 2023-09-30T02:05:14.530 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.525+0000 7f972ffff700 1 -- 172.21.15.110:0/1511563063 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f973c005c90 con 0x7f97480a5f40 2023-09-30T02:05:14.530 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.525+0000 7f974e7f9700 10 monclient: _renew_subs 2023-09-30T02:05:14.530 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.525+0000 7f974e7f9700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:05:14.530 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.525+0000 7f974e7f9700 1 -- 172.21.15.110:0/1511563063 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9748151fe0 con 0x7f97480a5f40 2023-09-30T02:05:14.530 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.525+0000 7f972ffff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:05:14.530 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.525+0000 7f972ffff700 1 -- 172.21.15.110:0/1511563063 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f973c007320 con 0x7f97480a5f40 2023-09-30T02:05:14.530 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.525+0000 7f972ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:05:14.531 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.525+0000 7f972ffff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:05:14.531 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.525+0000 7f972ffff700 10 monclient: dump: 2023-09-30T02:05:14.531 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2023-09-30T02:05:14.531 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:14.531 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:14.531 INFO:teuthology.orchestra.run.smithi110.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:14.531 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2023-09-30T02:05:14.531 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 3 2023-09-30T02:05:14.532 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:05:14.532 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:05:14.532 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:05:14.532 INFO:teuthology.orchestra.run.smithi110.stderr: 2023-09-30T02:05:14.532 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.525+0000 7f97467fc700 4 set_mon_vals no callback set 2023-09-30T02:05:14.532 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.525+0000 7f974e7f9700 10 client.4114.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:05:14.532 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.525+0000 7f974e7f9700 10 monclient: _renew_subs 2023-09-30T02:05:14.532 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.525+0000 7f974e7f9700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:05:14.532 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.525+0000 7f974e7f9700 1 -- 172.21.15.110:0/1511563063 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f974813d3b0 con 0x7f97480a5f40 2023-09-30T02:05:14.532 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.525+0000 7f974e7f9700 1 librados: init done 2023-09-30T02:05:14.533 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.526+0000 7f974e7f9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:05:14.533 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.526+0000 7f974e7f9700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:05:14.533 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.526+0000 7f974e7f9700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:05:14.533 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.526+0000 7f974e7f9700 1 -- 172.21.15.110:0/1511563063 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9748064a00 con 0x7f97480a5f40 2023-09-30T02:05:14.579 INFO:tasks.ceph.mgr.y.smithi059.stderr:2023-09-30T02:05:14.578+0000 7fc8624c21c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-09-30T02:05:14.725 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.724+0000 7f972ffff700 1 -- 172.21.15.110:0/1511563063 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f973c01c440 con 0x7f97480a5f40 2023-09-30T02:05:14.725 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.724+0000 7f972ffff700 1 -- 172.21.15.110:0/1511563063 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 721+0+0 (secure 0 0 0) 0x7f973c012750 con 0x7f97480a5f40 2023-09-30T02:05:14.725 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.724+0000 7f972ffff700 10 client.4114.objecter ms_dispatch 0x7f9748001550 osd_map(2..2 src has 1..2) v4 2023-09-30T02:05:14.725 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.724+0000 7f972ffff700 3 client.4114.objecter handle_osd_map got epochs [2,2] > 0 2023-09-30T02:05:14.725 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.724+0000 7f972ffff700 3 client.4114.objecter handle_osd_map decoding full epoch 2 2023-09-30T02:05:14.725 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.724+0000 7f972ffff700 20 client.4114.objecter dump_active .. 0 homeless 2023-09-30T02:05:14.725 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.724+0000 7f972ffff700 1 -- 172.21.15.110:0/1511563063 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f973c017070 con 0x7f97480a5f40 2023-09-30T02:05:14.725 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.724+0000 7f972ffff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:05:14.725 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.724+0000 7f972ffff700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:05:14.806 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.806+0000 7f974e7f9700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "efdab44b-be57-4b81-8567-ff2deec1d3e3", "id": 0}] 2023-09-30T02:05:14.806 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.806+0000 7f974e7f9700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "efdab44b-be57-4b81-8567-ff2deec1d3e3", "id": 0}] 2023-09-30T02:05:14.806 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.806+0000 7f974e7f9700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:05:14.806 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:14.806+0000 7f974e7f9700 1 -- 172.21.15.110:0/1511563063 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd new", "uuid": "efdab44b-be57-4b81-8567-ff2deec1d3e3", "id": 0} v 0) v1 -- 0x7f974806a720 con 0x7f97480a5f40 2023-09-30T02:05:14.887 INFO:tasks.ceph.mgr.x.smithi110.stderr:2023-09-30T02:05:14.888+0000 7fae141da1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-09-30T02:05:14.963 INFO:tasks.ceph.mgr.y.smithi059.stderr:2023-09-30T02:05:14.962+0000 7fc8624c21c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-09-30T02:05:15.053 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.053+0000 7f972ffff700 1 -- 172.21.15.110:0/1511563063 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "efdab44b-be57-4b81-8567-ff2deec1d3e3", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f973c005e00 con 0x7f97480a5f40 2023-09-30T02:05:15.053 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.053+0000 7f972ffff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "efdab44b-be57-4b81-8567-ff2deec1d3e3", "id": 0}] 2023-09-30T02:05:15.054 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.053+0000 7f972ffff700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:05:15.054 INFO:teuthology.orchestra.run.smithi110.stdout:0 2023-09-30T02:05:15.055 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.053+0000 7f974e7f9700 10 librados: watch_flush enter 2023-09-30T02:05:15.055 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.053+0000 7f974e7f9700 10 librados: watch_flush exit 2023-09-30T02:05:15.056 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.054+0000 7f974e7f9700 20 client.4114.objecter shutdown clearing up homeless session... 2023-09-30T02:05:15.056 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.054+0000 7f974e7f9700 10 client.4114.objecter successfully canceled tick 2023-09-30T02:05:15.056 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.054+0000 7f974e7f9700 10 monclient: shutdown 2023-09-30T02:05:15.056 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.054+0000 7f974e7f9700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:05:15.056 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.054+0000 7f974e7f9700 1 -- 172.21.15.110:0/1511563063 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f97480a5f40 msgr2=0x7f9748151aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:15.056 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.054+0000 7f974e7f9700 1 --2- 172.21.15.110:0/1511563063 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f97480a5f40 0x7f9748151aa0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f973c00c7c0 tx=0x7f973c007f30 comp rx=0 tx=0).stop 2023-09-30T02:05:15.056 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.055+0000 7f974e7f9700 1 -- 172.21.15.110:0/1511563063 shutdown_connections 2023-09-30T02:05:15.056 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.055+0000 7f974e7f9700 1 --2- 172.21.15.110:0/1511563063 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9748142290 0x7f974813ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:15.056 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.055+0000 7f974e7f9700 1 --2- 172.21.15.110:0/1511563063 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f97480a5b70 0x7f9748151560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:15.056 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.055+0000 7f974e7f9700 1 --2- 172.21.15.110:0/1511563063 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f97480a5f40 0x7f9748151aa0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:15.056 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.055+0000 7f974e7f9700 1 -- 172.21.15.110:0/1511563063 >> 172.21.15.110:0/1511563063 conn(0x7f97480631f0 msgr2=0x7f9748070f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:05:15.056 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.055+0000 7f974e7f9700 1 -- 172.21.15.110:0/1511563063 shutdown_connections 2023-09-30T02:05:15.057 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.055+0000 7f974e7f9700 1 -- 172.21.15.110:0/1511563063 wait complete. 2023-09-30T02:05:15.057 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.055+0000 7f974e7f9700 1 librados: shutdown 2023-09-30T02:05:15.066 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph --cluster ceph osd new eff9786d-8c39-4044-91d2-a31f33e3b16e 1 2023-09-30T02:05:15.186 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.185+0000 7f3e44485700 10 monclient: get_monmap_and_config 2023-09-30T02:05:15.186 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.185+0000 7f3e44485700 10 monclient: build_initial_monmap 2023-09-30T02:05:15.186 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.185+0000 7f3e44485700 10 monclient: monmap: 2023-09-30T02:05:15.186 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2023-09-30T02:05:15.186 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:15.186 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2023-09-30T02:05:15.186486+0000 2023-09-30T02:05:15.186 INFO:teuthology.orchestra.run.smithi110.stderr:created 2023-09-30T02:05:15.186486+0000 2023-09-30T02:05:15.186 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2023-09-30T02:05:15.187 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 3 2023-09-30T02:05:15.187 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:05:15.187 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:05:15.187 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:05:15.187 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:05:15.187 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:05:15.187 INFO:teuthology.orchestra.run.smithi110.stderr: 2023-09-30T02:05:15.187 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.186+0000 7f3e44485700 1 Processor -- start 2023-09-30T02:05:15.187 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.186+0000 7f3e44485700 1 -- start start 2023-09-30T02:05:15.187 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.186+0000 7f3e44485700 10 monclient: init 2023-09-30T02:05:15.189 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.186+0000 7f3e44485700 10 monclient: _reopen_session rank -1 2023-09-30T02:05:15.189 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.186+0000 7f3e44485700 10 monclient: _add_conns ranks=[4,0,1,2,3] 2023-09-30T02:05:15.189 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.186+0000 7f3e44485700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f3e3c0a5b70 0x7f3e3c0b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:15.190 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.186+0000 7f3e44485700 10 monclient(hunting): picked mon.noname-c con 0x7f3e3c0a5b70 addr v2:172.21.15.110:3300/0 2023-09-30T02:05:15.190 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.186+0000 7f3e44485700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3e3c0a5f40 addr v1:172.21.15.59:6789/0 2023-09-30T02:05:15.190 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.186+0000 7f3e44485700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3e3c142290 addr v1:172.21.15.110:6789/0 2023-09-30T02:05:15.190 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.187+0000 7f3e44485700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e3c0a75a0 con 0x7f3e3c0a5f40 2023-09-30T02:05:15.190 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.187+0000 7f3e44485700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e3c0af3f0 con 0x7f3e3c142290 2023-09-30T02:05:15.190 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.187+0000 7f3e44485700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:15.190 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.187+0000 7f3e44485700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e3c0a63e0 con 0x7f3e3c0a5b70 2023-09-30T02:05:15.190 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.187+0000 7f3e44485700 10 monclient(hunting): _renew_subs 2023-09-30T02:05:15.190 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.187+0000 7f3e44485700 10 monclient(hunting): authenticate will time out at 1764.647037s 2023-09-30T02:05:15.190 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.187+0000 7f3e42221700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f3e3c0a5b70 0x7f3e3c0b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:15.191 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.187+0000 7f3e41a20700 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f3e3c0a5f40 0x7f3e3c141b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.110:49540/0 (socket says 172.21.15.110:49540) 2023-09-30T02:05:15.191 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.187+0000 7f3e41a20700 1 -- 172.21.15.110:0/80704498 learned_addr learned my addr 172.21.15.110:0/80704498 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-09-30T02:05:15.191 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.188+0000 7f3e42221700 10 monclient(hunting): get_auth_request con 0x7f3e3c0a5b70 auth_method 0 2023-09-30T02:05:15.191 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.188+0000 7f3e42221700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:15.191 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.188+0000 7f3e42221700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:15.191 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.188+0000 7f3e42221700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:15.191 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.189+0000 7f3e42221700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:05:15.191 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.189+0000 7f3e42221700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:05:15.191 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.189+0000 7f3e42221700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:05:15.192 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.191+0000 7f3e4121f700 1 -- 172.21.15.110:0/80704498 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2516766718 0 0) 0x7f3e3c0a75a0 con 0x7f3e3c0a5f40 2023-09-30T02:05:15.192 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.191+0000 7f3e4121f700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:15.192 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.191+0000 7f3e4121f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:15.192 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.191+0000 7f3e4121f700 10 monclient(hunting): my global_id is 4116 2023-09-30T02:05:15.192 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.191+0000 7f3e4121f700 1 -- 172.21.15.110:0/80704498 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e28003570 con 0x7f3e3c0a5f40 2023-09-30T02:05:15.194 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.194+0000 7f3e42221700 10 monclient(hunting): handle_auth_done global_id 4117 payload 274 2023-09-30T02:05:15.194 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.194+0000 7f3e42221700 1 -- 172.21.15.110:0/80704498 >> v1:172.21.15.110:6789/0 conn(0x7f3e3c142290 legacy=0x7f3e3c13af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:15.194 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.194+0000 7f3e42221700 1 -- 172.21.15.110:0/80704498 >> v1:172.21.15.59:6789/0 conn(0x7f3e3c0a5f40 legacy=0x7f3e3c141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:15.194 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.194+0000 7f3e42221700 10 monclient: _finish_hunting 0 2023-09-30T02:05:15.195 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.194+0000 7f3e42221700 1 monclient: found mon.noname-c 2023-09-30T02:05:15.195 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.194+0000 7f3e42221700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:05:15.195 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.194+0000 7f3e42221700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3300/0 2023-09-30T02:05:15.195 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.194+0000 7f3e42221700 1 -- 172.21.15.110:0/80704498 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e3c13b640 con 0x7f3e3c0a5b70 2023-09-30T02:05:15.198 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.198+0000 7f3e42221700 1 --2- 172.21.15.110:0/80704498 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3e3c0a5b70 0x7f3e3c0b1500 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3e380098e0 tx=0x7f3e380091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9d588963c9e84fd9 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:05:15.204 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.204+0000 7f3e4121f700 1 -- 172.21.15.110:0/80704498 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e3800b640 con 0x7f3e3c0a5b70 2023-09-30T02:05:15.204 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.204+0000 7f3e4121f700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:05:15.205 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.204+0000 7f3e4121f700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:05:15.205 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.204+0000 7f3e4121f700 10 monclient: dump: 2023-09-30T02:05:15.205 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2023-09-30T02:05:15.205 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:15.205 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:15.205 INFO:teuthology.orchestra.run.smithi110.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:15.205 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2023-09-30T02:05:15.205 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 3 2023-09-30T02:05:15.205 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:05:15.205 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:05:15.206 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:05:15.206 INFO:teuthology.orchestra.run.smithi110.stderr: 2023-09-30T02:05:15.206 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.204+0000 7f3e4121f700 10 monclient: _finish_auth 0 2023-09-30T02:05:15.206 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.204+0000 7f3e4121f700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:05:15.206 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.204+0000 7f3e4121f700 1 -- 172.21.15.110:0/80704498 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e3800b7a0 con 0x7f3e3c0a5b70 2023-09-30T02:05:15.206 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.204+0000 7f3e4121f700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:05:15.206 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.204+0000 7f3e4121f700 1 -- 172.21.15.110:0/80704498 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e3800bb40 con 0x7f3e3c0a5b70 2023-09-30T02:05:15.206 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.204+0000 7f3e4121f700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:05:15.206 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.204+0000 7f3e4121f700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:05:15.206 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.204+0000 7f3e4121f700 10 monclient: dump: 2023-09-30T02:05:15.207 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2023-09-30T02:05:15.207 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:15.207 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:15.207 INFO:teuthology.orchestra.run.smithi110.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:15.207 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2023-09-30T02:05:15.207 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 3 2023-09-30T02:05:15.207 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:05:15.207 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:05:15.207 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:05:15.207 INFO:teuthology.orchestra.run.smithi110.stderr: 2023-09-30T02:05:15.207 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.204+0000 7f3e44485700 5 monclient: authenticate success, global_id 4117 2023-09-30T02:05:15.208 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.204+0000 7f3e44485700 10 monclient: get_monmap_and_config success 2023-09-30T02:05:15.208 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.204+0000 7f3e44485700 4 set_mon_vals no callback set 2023-09-30T02:05:15.212 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.205+0000 7f3e44485700 10 monclient: shutdown 2023-09-30T02:05:15.212 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.205+0000 7f3e44485700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:05:15.212 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.205+0000 7f3e44485700 1 -- 172.21.15.110:0/80704498 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3e3c0a5b70 msgr2=0x7f3e3c0b1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:15.212 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.205+0000 7f3e44485700 1 --2- 172.21.15.110:0/80704498 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3e3c0a5b70 0x7f3e3c0b1500 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3e380098e0 tx=0x7f3e380091a0 comp rx=0 tx=0).stop 2023-09-30T02:05:15.212 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.205+0000 7f3e44485700 1 -- 172.21.15.110:0/80704498 shutdown_connections 2023-09-30T02:05:15.212 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.205+0000 7f3e44485700 1 --2- 172.21.15.110:0/80704498 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3e3c0a5b70 0x7f3e3c0b1500 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:15.213 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.205+0000 7f3e44485700 1 -- 172.21.15.110:0/80704498 >> 172.21.15.110:0/80704498 conn(0x7f3e3c0631f0 msgr2=0x7f3e3c0635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:05:15.213 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.205+0000 7f3e44485700 1 -- 172.21.15.110:0/80704498 shutdown_connections 2023-09-30T02:05:15.213 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.205+0000 7f3e44485700 1 -- 172.21.15.110:0/80704498 wait complete. 2023-09-30T02:05:15.213 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.206+0000 7f3e44485700 10 monclient: build_initial_monmap 2023-09-30T02:05:15.213 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.206+0000 7f3e44485700 10 monclient: monmap: 2023-09-30T02:05:15.213 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2023-09-30T02:05:15.213 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:05:15.213 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2023-09-30T02:05:15.213 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2023-09-30T02:05:15.213 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2023-09-30T02:05:15.214 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2023-09-30T02:05:15.214 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:05:15.214 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:05:15.214 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:05:15.214 INFO:teuthology.orchestra.run.smithi110.stderr: 2023-09-30T02:05:15.214 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.206+0000 7f3e44485700 1 librados: starting msgr at 2023-09-30T02:05:15.214 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.206+0000 7f3e44485700 1 librados: starting objecter 2023-09-30T02:05:15.214 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.206+0000 7f3e44485700 1 Processor -- start 2023-09-30T02:05:15.214 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.206+0000 7f3e44485700 1 -- start start 2023-09-30T02:05:15.214 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.206+0000 7f3e44485700 1 librados: setting wanted keys 2023-09-30T02:05:15.215 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.206+0000 7f3e44485700 1 librados: calling monclient init 2023-09-30T02:05:15.215 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.206+0000 7f3e44485700 10 monclient: init 2023-09-30T02:05:15.215 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.207+0000 7f3e44485700 10 monclient: _reopen_session rank -1 2023-09-30T02:05:15.215 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.207+0000 7f3e44485700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T02:05:15.215 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.207+0000 7f3e44485700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3e3c0a5b70 0x7f3e3c13e710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:15.215 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.207+0000 7f3e44485700 10 monclient(hunting): picked mon.noname-c con 0x7f3e3c0a5b70 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:05:15.215 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.207+0000 7f3e44485700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3e3c0a5f40 0x7f3e3c140c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:15.215 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.207+0000 7f3e44485700 10 monclient(hunting): picked mon.noname-b con 0x7f3e3c0a5f40 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:05:15.215 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.207+0000 7f3e44485700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3e3c142290 0x7f3e3c13e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:15.215 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.207+0000 7f3e44485700 10 monclient(hunting): picked mon.noname-a con 0x7f3e3c142290 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:05:15.216 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.207+0000 7f3e44485700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:15.216 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.207+0000 7f3e44485700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e3c13ed10 con 0x7f3e3c142290 2023-09-30T02:05:15.216 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.207+0000 7f3e44485700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:15.216 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.207+0000 7f3e44485700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e3c150430 con 0x7f3e3c0a5f40 2023-09-30T02:05:15.216 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.207+0000 7f3e44485700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:15.216 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.207+0000 7f3e44485700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e3c1505f0 con 0x7f3e3c0a5b70 2023-09-30T02:05:15.216 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.207+0000 7f3e44485700 10 monclient(hunting): _renew_subs 2023-09-30T02:05:15.216 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.207+0000 7f3e44485700 10 monclient(hunting): authenticate will time out at 1764.667307s 2023-09-30T02:05:15.216 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.207+0000 7f3e41a20700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3e3c0a5f40 0x7f3e3c140c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:15.216 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.208+0000 7f3e42221700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3e3c0a5b70 0x7f3e3c13e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:15.216 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.208+0000 7f3e41a20700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3e3c0a5f40 0x7f3e3c140c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:51582/0 (socket says 172.21.15.110:51582) 2023-09-30T02:05:15.217 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.208+0000 7f3e41a20700 1 -- 172.21.15.110:0/984133974 learned_addr learned my addr 172.21.15.110:0/984133974 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-09-30T02:05:15.217 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.208+0000 7f3e41a20700 10 monclient(hunting): get_auth_request con 0x7f3e3c0a5f40 auth_method 0 2023-09-30T02:05:15.217 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.208+0000 7f3e41a20700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:15.217 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.208+0000 7f3e41a20700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:15.217 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.208+0000 7f3e41a20700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:15.217 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.208+0000 7f3e42a22700 1 --2- 172.21.15.110:0/984133974 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3e3c142290 0x7f3e3c13e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:15.217 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.208+0000 7f3e42a22700 10 monclient(hunting): get_auth_request con 0x7f3e3c142290 auth_method 0 2023-09-30T02:05:15.217 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.208+0000 7f3e42a22700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:15.217 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.208+0000 7f3e42a22700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:15.217 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.208+0000 7f3e42a22700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:15.218 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.208+0000 7f3e42221700 10 monclient(hunting): get_auth_request con 0x7f3e3c0a5b70 auth_method 0 2023-09-30T02:05:15.218 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.208+0000 7f3e42221700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:15.218 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.208+0000 7f3e42221700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:15.218 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.208+0000 7f3e42221700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:15.218 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.208+0000 7f3e41a20700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:05:15.218 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.208+0000 7f3e41a20700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:05:15.218 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.208+0000 7f3e41a20700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:05:15.218 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.209+0000 7f3e42a22700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:05:15.218 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.209+0000 7f3e42a22700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:05:15.218 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.209+0000 7f3e42a22700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:05:15.219 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.209+0000 7f3e42221700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:05:15.219 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.209+0000 7f3e42221700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:05:15.219 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.209+0000 7f3e42221700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:05:15.219 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.211+0000 7f3e41a20700 10 monclient(hunting): handle_auth_done global_id 4123 payload 824 2023-09-30T02:05:15.219 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.211+0000 7f3e41a20700 1 -- 172.21.15.110:0/984133974 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3e3c0a5b70 msgr2=0x7f3e3c13e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:15.219 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.211+0000 7f3e41a20700 1 --2- 172.21.15.110:0/984133974 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3e3c0a5b70 0x7f3e3c13e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:15.219 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.211+0000 7f3e41a20700 1 -- 172.21.15.110:0/984133974 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3e3c142290 msgr2=0x7f3e3c13e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:15.219 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.211+0000 7f3e41a20700 1 --2- 172.21.15.110:0/984133974 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3e3c142290 0x7f3e3c13e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:15.219 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.211+0000 7f3e41a20700 10 monclient: _finish_hunting 0 2023-09-30T02:05:15.219 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.211+0000 7f3e41a20700 1 monclient: found mon.noname-b 2023-09-30T02:05:15.219 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.211+0000 7f3e41a20700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:05:15.220 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.211+0000 7f3e41a20700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:05:15.220 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.211+0000 7f3e41a20700 1 -- 172.21.15.110:0/984133974 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e38009420 con 0x7f3e3c0a5f40 2023-09-30T02:05:15.222 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.214+0000 7f3e41a20700 1 --2- 172.21.15.110:0/984133974 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3e3c0a5f40 0x7f3e3c140c40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c00c7c0 tx=0x7f3e2c007e00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:05:15.222 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.214+0000 7f3e327fc700 10 client.?.objecter ms_handle_connect 0x7f3e3c0a5f40 2023-09-30T02:05:15.222 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.214+0000 7f3e327fc700 10 client.?.objecter resend_mon_ops 2023-09-30T02:05:15.222 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.219+0000 7f3e327fc700 1 -- 172.21.15.110:0/984133974 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e2c005cd0 con 0x7f3e3c0a5f40 2023-09-30T02:05:15.222 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.219+0000 7f3e327fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:05:15.222 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.219+0000 7f3e327fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:05:15.222 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.219+0000 7f3e327fc700 10 monclient: dump: 2023-09-30T02:05:15.223 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2023-09-30T02:05:15.223 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:15.223 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:15.223 INFO:teuthology.orchestra.run.smithi110.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:15.223 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2023-09-30T02:05:15.223 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 3 2023-09-30T02:05:15.223 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:05:15.223 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:05:15.223 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:05:15.224 INFO:teuthology.orchestra.run.smithi110.stderr: 2023-09-30T02:05:15.224 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.219+0000 7f3e327fc700 10 monclient: _finish_auth 0 2023-09-30T02:05:15.224 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.219+0000 7f3e327fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:05:15.224 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.219+0000 7f3e327fc700 1 -- 172.21.15.110:0/984133974 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e2c005e30 con 0x7f3e3c0a5f40 2023-09-30T02:05:15.224 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.219+0000 7f3e44485700 5 monclient: authenticate success, global_id 4123 2023-09-30T02:05:15.224 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.219+0000 7f3e44485700 10 monclient: _renew_subs 2023-09-30T02:05:15.224 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.219+0000 7f3e44485700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:05:15.224 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.219+0000 7f3e44485700 1 -- 172.21.15.110:0/984133974 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e3c1508d0 con 0x7f3e3c0a5f40 2023-09-30T02:05:15.224 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.219+0000 7f3e327fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:05:15.224 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.219+0000 7f3e327fc700 1 -- 172.21.15.110:0/984133974 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e2c007320 con 0x7f3e3c0a5f40 2023-09-30T02:05:15.224 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.219+0000 7f3e327fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:05:15.225 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.219+0000 7f3e327fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:05:15.225 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.219+0000 7f3e327fc700 10 monclient: dump: 2023-09-30T02:05:15.225 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2023-09-30T02:05:15.225 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:15.225 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:15.225 INFO:teuthology.orchestra.run.smithi110.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:15.225 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2023-09-30T02:05:15.225 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 3 2023-09-30T02:05:15.225 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:05:15.225 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:05:15.226 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:05:15.226 INFO:teuthology.orchestra.run.smithi110.stderr: 2023-09-30T02:05:15.226 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.219+0000 7f3e40a1e700 4 set_mon_vals no callback set 2023-09-30T02:05:15.226 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.219+0000 7f3e44485700 10 client.4123.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:05:15.226 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.219+0000 7f3e44485700 10 monclient: _renew_subs 2023-09-30T02:05:15.226 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.219+0000 7f3e44485700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:05:15.226 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.219+0000 7f3e44485700 1 -- 172.21.15.110:0/984133974 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e3c150e40 con 0x7f3e3c0a5f40 2023-09-30T02:05:15.226 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.219+0000 7f3e44485700 1 librados: init done 2023-09-30T02:05:15.226 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.220+0000 7f3e44485700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:05:15.226 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.220+0000 7f3e44485700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:05:15.227 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.220+0000 7f3e44485700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:05:15.227 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.220+0000 7f3e44485700 1 -- 172.21.15.110:0/984133974 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e3c151060 con 0x7f3e3c0a5f40 2023-09-30T02:05:15.334 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.334+0000 7f3e327fc700 1 -- 172.21.15.110:0/984133974 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f3e2c01c6c0 con 0x7f3e3c0a5f40 2023-09-30T02:05:15.334 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.334+0000 7f3e327fc700 1 -- 172.21.15.110:0/984133974 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 836+0+0 (secure 0 0 0) 0x7f3e2c0127c0 con 0x7f3e3c0a5f40 2023-09-30T02:05:15.334 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.334+0000 7f3e327fc700 10 client.4123.objecter ms_dispatch 0x7f3e3c001550 osd_map(3..3 src has 1..3) v4 2023-09-30T02:05:15.334 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.334+0000 7f3e327fc700 3 client.4123.objecter handle_osd_map got epochs [3,3] > 0 2023-09-30T02:05:15.335 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.334+0000 7f3e327fc700 3 client.4123.objecter handle_osd_map decoding full epoch 3 2023-09-30T02:05:15.335 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.334+0000 7f3e327fc700 20 client.4123.objecter dump_active .. 0 homeless 2023-09-30T02:05:15.335 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.334+0000 7f3e327fc700 1 -- 172.21.15.110:0/984133974 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f3e2c017070 con 0x7f3e3c0a5f40 2023-09-30T02:05:15.335 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.334+0000 7f3e327fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:05:15.335 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.334+0000 7f3e327fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:05:15.416 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.416+0000 7f3e44485700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "eff9786d-8c39-4044-91d2-a31f33e3b16e", "id": 1}] 2023-09-30T02:05:15.416 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.416+0000 7f3e44485700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "eff9786d-8c39-4044-91d2-a31f33e3b16e", "id": 1}] 2023-09-30T02:05:15.416 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.416+0000 7f3e44485700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:05:15.416 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.416+0000 7f3e44485700 1 -- 172.21.15.110:0/984133974 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd new", "uuid": "eff9786d-8c39-4044-91d2-a31f33e3b16e", "id": 1} v 0) v1 -- 0x7f3e3c064a00 con 0x7f3e3c0a5f40 2023-09-30T02:05:15.519 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.520+0000 7f3e327fc700 1 -- 172.21.15.110:0/984133974 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "eff9786d-8c39-4044-91d2-a31f33e3b16e", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f3e2c007790 con 0x7f3e3c0a5f40 2023-09-30T02:05:15.520 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.520+0000 7f3e327fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "eff9786d-8c39-4044-91d2-a31f33e3b16e", "id": 1}] 2023-09-30T02:05:15.520 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.520+0000 7f3e327fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:05:15.520 INFO:teuthology.orchestra.run.smithi110.stdout:1 2023-09-30T02:05:15.521 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.520+0000 7f3e44485700 10 librados: watch_flush enter 2023-09-30T02:05:15.522 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.521+0000 7f3e44485700 10 librados: watch_flush exit 2023-09-30T02:05:15.522 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.521+0000 7f3e44485700 20 client.4123.objecter shutdown clearing up homeless session... 2023-09-30T02:05:15.522 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.521+0000 7f3e44485700 10 client.4123.objecter successfully canceled tick 2023-09-30T02:05:15.522 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.521+0000 7f3e44485700 10 monclient: shutdown 2023-09-30T02:05:15.522 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.521+0000 7f3e44485700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:05:15.522 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.521+0000 7f3e44485700 1 -- 172.21.15.110:0/984133974 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3e3c0a5f40 msgr2=0x7f3e3c140c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:15.522 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.521+0000 7f3e44485700 1 --2- 172.21.15.110:0/984133974 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3e3c0a5f40 0x7f3e3c140c40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c00c7c0 tx=0x7f3e2c007e00 comp rx=0 tx=0).stop 2023-09-30T02:05:15.522 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.521+0000 7f3e44485700 1 -- 172.21.15.110:0/984133974 shutdown_connections 2023-09-30T02:05:15.522 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.521+0000 7f3e44485700 1 --2- 172.21.15.110:0/984133974 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3e3c142290 0x7f3e3c13e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:15.522 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.521+0000 7f3e44485700 1 --2- 172.21.15.110:0/984133974 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3e3c0a5b70 0x7f3e3c13e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:15.523 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.521+0000 7f3e44485700 1 --2- 172.21.15.110:0/984133974 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3e3c0a5f40 0x7f3e3c140c40 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:15.523 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.521+0000 7f3e44485700 1 -- 172.21.15.110:0/984133974 >> 172.21.15.110:0/984133974 conn(0x7f3e3c0631f0 msgr2=0x7f3e3c0b4a60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:05:15.523 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.521+0000 7f3e44485700 1 -- 172.21.15.110:0/984133974 shutdown_connections 2023-09-30T02:05:15.523 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.521+0000 7f3e44485700 1 -- 172.21.15.110:0/984133974 wait complete. 2023-09-30T02:05:15.523 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.521+0000 7f3e44485700 1 librados: shutdown 2023-09-30T02:05:15.534 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph --cluster ceph osd new b276ed4b-317b-4ddf-96ab-7ad71b8e3ff7 2 2023-09-30T02:05:15.652 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.652+0000 7f925f371700 10 monclient: get_monmap_and_config 2023-09-30T02:05:15.652 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.652+0000 7f925f371700 10 monclient: build_initial_monmap 2023-09-30T02:05:15.652 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.652+0000 7f925f371700 10 monclient: monmap: 2023-09-30T02:05:15.652 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2023-09-30T02:05:15.652 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:15.652 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2023-09-30T02:05:15.653168+0000 2023-09-30T02:05:15.652 INFO:teuthology.orchestra.run.smithi110.stderr:created 2023-09-30T02:05:15.653168+0000 2023-09-30T02:05:15.652 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2023-09-30T02:05:15.652 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 3 2023-09-30T02:05:15.653 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:05:15.653 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:05:15.653 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:05:15.653 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:05:15.653 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:05:15.653 INFO:teuthology.orchestra.run.smithi110.stderr: 2023-09-30T02:05:15.655 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.652+0000 7f925f371700 1 Processor -- start 2023-09-30T02:05:15.655 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.653+0000 7f925f371700 1 -- start start 2023-09-30T02:05:15.655 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.653+0000 7f925f371700 10 monclient: init 2023-09-30T02:05:15.655 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.653+0000 7f925f371700 10 monclient: _reopen_session rank -1 2023-09-30T02:05:15.656 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.653+0000 7f925f371700 10 monclient: _add_conns ranks=[2,4,1,3,0] 2023-09-30T02:05:15.656 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.653+0000 7f925f371700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f92580a5b70 0x7f92580b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:15.656 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.653+0000 7f925f371700 10 monclient(hunting): picked mon.noname-b con 0x7f92580a5b70 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:05:15.656 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.653+0000 7f925f371700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f92580a5f40 0x7f9258141b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:15.656 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.653+0000 7f925f371700 10 monclient(hunting): picked mon.noname-c con 0x7f92580a5f40 addr v2:172.21.15.110:3300/0 2023-09-30T02:05:15.656 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.653+0000 7f925f371700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f92581420c0 addr v1:172.21.15.110:6789/0 2023-09-30T02:05:15.656 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.653+0000 7f925f371700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92580a63b0 con 0x7f92581420c0 2023-09-30T02:05:15.656 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.653+0000 7f925f371700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:15.656 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.653+0000 7f925f371700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92580a75a0 con 0x7f92580a5f40 2023-09-30T02:05:15.657 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.653+0000 7f925f371700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:15.657 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.653+0000 7f925f371700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92580af3f0 con 0x7f92580a5b70 2023-09-30T02:05:15.657 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.653+0000 7f925f371700 10 monclient(hunting): _renew_subs 2023-09-30T02:05:15.657 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.653+0000 7f925f371700 10 monclient(hunting): authenticate will time out at 1765.113692s 2023-09-30T02:05:15.657 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.654+0000 7f925c90c700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f92580a5f40 0x7f9258141b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:15.657 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.654+0000 7f925d10d700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f92580a5b70 0x7f92580b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:15.657 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.654+0000 7f925c90c700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f92580a5f40 0x7f9258141b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:51592/0 (socket says 172.21.15.110:51592) 2023-09-30T02:05:15.657 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.654+0000 7f925c90c700 1 -- 172.21.15.110:0/3926642576 learned_addr learned my addr 172.21.15.110:0/3926642576 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-09-30T02:05:15.657 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.654+0000 7f925c90c700 10 monclient(hunting): get_auth_request con 0x7f92580a5f40 auth_method 0 2023-09-30T02:05:15.657 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.654+0000 7f925c90c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:15.657 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.654+0000 7f925c90c700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:15.657 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.654+0000 7f925c90c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:15.658 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.654+0000 7f925d10d700 10 monclient(hunting): get_auth_request con 0x7f92580a5b70 auth_method 0 2023-09-30T02:05:15.658 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.654+0000 7f925d10d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:15.658 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.654+0000 7f925d10d700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:15.658 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.654+0000 7f925d10d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:15.659 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.655+0000 7f925d10d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:05:15.659 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.655+0000 7f925d10d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:05:15.659 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.655+0000 7f925d10d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:05:15.659 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.655+0000 7f925c90c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:05:15.659 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.655+0000 7f925c90c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:05:15.659 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.655+0000 7f925c90c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:05:15.660 INFO:tasks.ceph.mgr.x.smithi110.stderr:2023-09-30T02:05:15.656+0000 7fae141da1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-09-30T02:05:15.660 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.657+0000 7f925d10d700 10 monclient(hunting): handle_auth_done global_id 4109 payload 274 2023-09-30T02:05:15.660 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.657+0000 7f925d10d700 1 -- 172.21.15.110:0/3926642576 >> v2:172.21.15.110:3300/0 conn(0x7f92580a5f40 msgr2=0x7f9258141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:05:15.660 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.657+0000 7f925d10d700 1 --2- 172.21.15.110:0/3926642576 >> v2:172.21.15.110:3300/0 conn(0x7f92580a5f40 0x7f9258141b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:15.660 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.657+0000 7f925d10d700 1 -- 172.21.15.110:0/3926642576 >> v1:172.21.15.110:6789/0 conn(0x7f92581420c0 legacy=0x7f925813e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:15.661 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.657+0000 7f925d10d700 10 monclient: _finish_hunting 0 2023-09-30T02:05:15.661 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.657+0000 7f925d10d700 1 monclient: found mon.noname-b 2023-09-30T02:05:15.661 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.657+0000 7f925d10d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:05:15.661 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.657+0000 7f925d10d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.59:3301/0 2023-09-30T02:05:15.661 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.657+0000 7f925d10d700 1 -- 172.21.15.110:0/3926642576 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9258142500 con 0x7f92580a5b70 2023-09-30T02:05:15.661 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.659+0000 7f925d10d700 1 --2- 172.21.15.110:0/3926642576 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f92580a5b70 0x7f92580b1500 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f92540098e0 tx=0x7f92540091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=684ca2bddd6fb681 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:05:15.664 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.664+0000 7f924ffff700 1 -- 172.21.15.110:0/3926642576 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f925400b640 con 0x7f92580a5b70 2023-09-30T02:05:15.664 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.664+0000 7f924ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:05:15.664 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.664+0000 7f924ffff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:05:15.664 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.664+0000 7f924ffff700 10 monclient: dump: 2023-09-30T02:05:15.664 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2023-09-30T02:05:15.664 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:15.664 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:15.664 INFO:teuthology.orchestra.run.smithi110.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:15.664 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2023-09-30T02:05:15.665 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 3 2023-09-30T02:05:15.665 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:05:15.665 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:05:15.665 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:05:15.665 INFO:teuthology.orchestra.run.smithi110.stderr: 2023-09-30T02:05:15.665 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.664+0000 7f924ffff700 10 monclient: _finish_auth 0 2023-09-30T02:05:15.665 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.664+0000 7f924ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:05:15.665 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.664+0000 7f924ffff700 1 -- 172.21.15.110:0/3926642576 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f925400b7a0 con 0x7f92580a5b70 2023-09-30T02:05:15.665 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.664+0000 7f924ffff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:05:15.665 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.664+0000 7f924ffff700 1 -- 172.21.15.110:0/3926642576 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f925400bb40 con 0x7f92580a5b70 2023-09-30T02:05:15.665 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.664+0000 7f925f371700 5 monclient: authenticate success, global_id 4109 2023-09-30T02:05:15.665 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.664+0000 7f925f371700 10 monclient: get_monmap_and_config success 2023-09-30T02:05:15.665 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.664+0000 7f924ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:05:15.666 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.664+0000 7f925f371700 4 set_mon_vals no callback set 2023-09-30T02:05:15.666 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.664+0000 7f924ffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:05:15.666 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.664+0000 7f924ffff700 10 monclient: dump: 2023-09-30T02:05:15.666 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2023-09-30T02:05:15.666 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:15.666 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:15.666 INFO:teuthology.orchestra.run.smithi110.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:15.666 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2023-09-30T02:05:15.666 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 3 2023-09-30T02:05:15.666 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:05:15.666 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:05:15.666 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:05:15.667 INFO:teuthology.orchestra.run.smithi110.stderr: 2023-09-30T02:05:15.670 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.664+0000 7f925f371700 10 monclient: shutdown 2023-09-30T02:05:15.670 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.664+0000 7f925f371700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:05:15.670 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.664+0000 7f925f371700 1 -- 172.21.15.110:0/3926642576 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f92580a5b70 msgr2=0x7f92580b1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:15.670 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.664+0000 7f925f371700 1 --2- 172.21.15.110:0/3926642576 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f92580a5b70 0x7f92580b1500 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f92540098e0 tx=0x7f92540091a0 comp rx=0 tx=0).stop 2023-09-30T02:05:15.670 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.665+0000 7f925f371700 1 -- 172.21.15.110:0/3926642576 shutdown_connections 2023-09-30T02:05:15.670 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.665+0000 7f925f371700 1 --2- 172.21.15.110:0/3926642576 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f92580a5b70 0x7f92580b1500 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:15.670 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.665+0000 7f925f371700 1 --2- 172.21.15.110:0/3926642576 >> v2:172.21.15.110:3300/0 conn(0x7f92580a5f40 0x7f9258141b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:15.671 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.665+0000 7f925f371700 1 -- 172.21.15.110:0/3926642576 >> 172.21.15.110:0/3926642576 conn(0x7f92580631f0 msgr2=0x7f92580635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:05:15.671 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.665+0000 7f925f371700 1 -- 172.21.15.110:0/3926642576 shutdown_connections 2023-09-30T02:05:15.671 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.665+0000 7f925f371700 1 -- 172.21.15.110:0/3926642576 wait complete. 2023-09-30T02:05:15.671 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.665+0000 7f925f371700 10 monclient: build_initial_monmap 2023-09-30T02:05:15.671 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.665+0000 7f925f371700 10 monclient: monmap: 2023-09-30T02:05:15.671 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2023-09-30T02:05:15.671 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:05:15.671 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2023-09-30T02:05:15.671 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2023-09-30T02:05:15.671 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2023-09-30T02:05:15.671 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2023-09-30T02:05:15.671 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:05:15.672 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:05:15.672 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:05:15.672 INFO:teuthology.orchestra.run.smithi110.stderr: 2023-09-30T02:05:15.672 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.665+0000 7f925f371700 1 librados: starting msgr at 2023-09-30T02:05:15.672 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.665+0000 7f925f371700 1 librados: starting objecter 2023-09-30T02:05:15.672 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.666+0000 7f925f371700 1 Processor -- start 2023-09-30T02:05:15.672 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.666+0000 7f925f371700 1 -- start start 2023-09-30T02:05:15.672 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.666+0000 7f925f371700 1 librados: setting wanted keys 2023-09-30T02:05:15.672 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.666+0000 7f925f371700 1 librados: calling monclient init 2023-09-30T02:05:15.672 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.666+0000 7f925f371700 10 monclient: init 2023-09-30T02:05:15.672 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.666+0000 7f925f371700 10 monclient: _reopen_session rank -1 2023-09-30T02:05:15.672 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.666+0000 7f925f371700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T02:05:15.673 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.666+0000 7f925f371700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f92580a5b70 0x7f92580a7a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:15.673 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.666+0000 7f925f371700 10 monclient(hunting): picked mon.noname-c con 0x7f92580a5b70 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:05:15.673 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.666+0000 7f925f371700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f92580a5f40 0x7f92580b2950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:15.673 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.666+0000 7f925f371700 10 monclient(hunting): picked mon.noname-a con 0x7f92580a5f40 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:05:15.673 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.666+0000 7f925f371700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f92581420c0 0x7f92580b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:15.673 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.666+0000 7f925f371700 10 monclient(hunting): picked mon.noname-b con 0x7f92581420c0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:05:15.673 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.666+0000 7f925f371700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:15.673 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.666+0000 7f925f371700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92580a82f0 con 0x7f92580a5f40 2023-09-30T02:05:15.673 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.666+0000 7f925f371700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:15.673 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.666+0000 7f925f371700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9258137a20 con 0x7f92581420c0 2023-09-30T02:05:15.673 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.666+0000 7f925f371700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:15.673 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.666+0000 7f925f371700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92580a7fa0 con 0x7f92580a5b70 2023-09-30T02:05:15.673 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.666+0000 7f925f371700 10 monclient(hunting): _renew_subs 2023-09-30T02:05:15.674 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.666+0000 7f925f371700 10 monclient(hunting): authenticate will time out at 1765.126945s 2023-09-30T02:05:15.674 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.667+0000 7f925d90e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f92581420c0 0x7f92580b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:15.674 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.667+0000 7f925c90c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f92580a5f40 0x7f92580b2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:15.674 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.667+0000 7f925d90e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f92581420c0 0x7f92580b2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:51600/0 (socket says 172.21.15.110:51600) 2023-09-30T02:05:15.674 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.667+0000 7f925d90e700 1 -- 172.21.15.110:0/3733126885 learned_addr learned my addr 172.21.15.110:0/3733126885 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-09-30T02:05:15.674 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.667+0000 7f925d90e700 10 monclient(hunting): get_auth_request con 0x7f92581420c0 auth_method 0 2023-09-30T02:05:15.674 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.667+0000 7f925d90e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:15.674 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.667+0000 7f925d90e700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:15.674 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.667+0000 7f925d90e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:15.674 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.668+0000 7f925c90c700 10 monclient(hunting): get_auth_request con 0x7f92580a5f40 auth_method 0 2023-09-30T02:05:15.674 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.668+0000 7f925c90c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:15.675 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.668+0000 7f925c90c700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:15.675 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.668+0000 7f925c90c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:15.675 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.668+0000 7f925d90e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:05:15.675 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.668+0000 7f925d90e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:05:15.675 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.668+0000 7f925d90e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:05:15.675 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.668+0000 7f925c90c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:05:15.675 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.668+0000 7f925c90c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:05:15.675 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.668+0000 7f925c90c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:05:15.675 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.668+0000 7f925d10d700 1 --2- 172.21.15.110:0/3733126885 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f92580a5b70 0x7f92580a7a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:15.675 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.669+0000 7f925d10d700 10 monclient(hunting): get_auth_request con 0x7f92580a5b70 auth_method 0 2023-09-30T02:05:15.675 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.669+0000 7f925d10d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:15.675 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.669+0000 7f925d10d700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:15.675 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.669+0000 7f925d10d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:15.676 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.670+0000 7f925d10d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:05:15.676 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.670+0000 7f925d10d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:05:15.676 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.670+0000 7f925d10d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:05:15.677 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.671+0000 7f925d90e700 10 monclient(hunting): handle_auth_done global_id 4132 payload 824 2023-09-30T02:05:15.677 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.672+0000 7f925d90e700 1 -- 172.21.15.110:0/3733126885 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f92580a5b70 msgr2=0x7f92580a7a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:15.677 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.672+0000 7f925d90e700 1 --2- 172.21.15.110:0/3733126885 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f92580a5b70 0x7f92580a7a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:15.677 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.672+0000 7f925d90e700 1 -- 172.21.15.110:0/3733126885 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f92580a5f40 msgr2=0x7f92580b2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:15.677 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.672+0000 7f925d90e700 1 --2- 172.21.15.110:0/3733126885 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f92580a5f40 0x7f92580b2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:15.677 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.672+0000 7f925d90e700 10 monclient: _finish_hunting 0 2023-09-30T02:05:15.677 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.672+0000 7f925d90e700 1 monclient: found mon.noname-b 2023-09-30T02:05:15.677 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.672+0000 7f925d90e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:05:15.677 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.672+0000 7f925d90e700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:05:15.678 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.672+0000 7f925d90e700 1 -- 172.21.15.110:0/3733126885 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9254009420 con 0x7f92581420c0 2023-09-30T02:05:15.678 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.674+0000 7f925d90e700 1 --2- 172.21.15.110:0/3733126885 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f92581420c0 0x7f92580b2e90 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f925000c840 tx=0x7f9250001670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:05:15.678 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.674+0000 7f924d7fa700 10 client.?.objecter ms_handle_connect 0x7f92581420c0 2023-09-30T02:05:15.678 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.675+0000 7f924d7fa700 10 client.?.objecter resend_mon_ops 2023-09-30T02:05:15.679 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.679+0000 7f924d7fa700 1 -- 172.21.15.110:0/3733126885 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9250012630 con 0x7f92581420c0 2023-09-30T02:05:15.680 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.679+0000 7f924d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:05:15.680 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.679+0000 7f924d7fa700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:05:15.680 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.679+0000 7f924d7fa700 10 monclient: dump: 2023-09-30T02:05:15.680 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2023-09-30T02:05:15.680 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:15.680 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:15.680 INFO:teuthology.orchestra.run.smithi110.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:15.680 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2023-09-30T02:05:15.680 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 3 2023-09-30T02:05:15.680 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:05:15.680 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:05:15.680 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:05:15.681 INFO:teuthology.orchestra.run.smithi110.stderr: 2023-09-30T02:05:15.681 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.679+0000 7f924d7fa700 10 monclient: _finish_auth 0 2023-09-30T02:05:15.681 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.679+0000 7f924d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:05:15.681 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.679+0000 7f924d7fa700 1 -- 172.21.15.110:0/3733126885 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9250012790 con 0x7f92581420c0 2023-09-30T02:05:15.681 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.679+0000 7f925f371700 5 monclient: authenticate success, global_id 4132 2023-09-30T02:05:15.681 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.679+0000 7f925f371700 10 monclient: _renew_subs 2023-09-30T02:05:15.681 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.679+0000 7f925f371700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:05:15.681 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.679+0000 7f925f371700 1 -- 172.21.15.110:0/3733126885 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92580a9480 con 0x7f92581420c0 2023-09-30T02:05:15.681 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.679+0000 7f924d7fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:05:15.681 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.679+0000 7f924d7fa700 1 -- 172.21.15.110:0/3733126885 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9250012b50 con 0x7f92581420c0 2023-09-30T02:05:15.681 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.679+0000 7f924d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:05:15.681 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.679+0000 7f924d7fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:05:15.681 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.679+0000 7f924d7fa700 10 monclient: dump: 2023-09-30T02:05:15.682 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2023-09-30T02:05:15.682 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:15.682 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:15.682 INFO:teuthology.orchestra.run.smithi110.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:15.682 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2023-09-30T02:05:15.682 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 3 2023-09-30T02:05:15.682 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:05:15.682 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:05:15.682 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:05:15.682 INFO:teuthology.orchestra.run.smithi110.stderr: 2023-09-30T02:05:15.682 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.679+0000 7f924f7fe700 4 set_mon_vals no callback set 2023-09-30T02:05:15.682 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.679+0000 7f925f371700 10 client.4132.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:05:15.683 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.679+0000 7f925f371700 10 monclient: _renew_subs 2023-09-30T02:05:15.683 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.679+0000 7f925f371700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:05:15.683 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.679+0000 7f925f371700 1 -- 172.21.15.110:0/3733126885 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92580a8590 con 0x7f92581420c0 2023-09-30T02:05:15.683 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.679+0000 7f925f371700 1 librados: init done 2023-09-30T02:05:15.683 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.680+0000 7f925f371700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:05:15.683 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.680+0000 7f925f371700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:05:15.683 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.680+0000 7f925f371700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:05:15.683 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.680+0000 7f925f371700 1 -- 172.21.15.110:0/3733126885 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9258064a00 con 0x7f92581420c0 2023-09-30T02:05:15.683 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.682+0000 7f924d7fa700 1 -- 172.21.15.110:0/3733126885 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f9250003030 con 0x7f92581420c0 2023-09-30T02:05:15.734 INFO:tasks.ceph.mgr.x.smithi110.stderr:2023-09-30T02:05:15.734+0000 7fae141da1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-09-30T02:05:15.783 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.783+0000 7f924d7fa700 1 -- 172.21.15.110:0/3733126885 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 951+0+0 (secure 0 0 0) 0x7f9250024d50 con 0x7f92581420c0 2023-09-30T02:05:15.784 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.783+0000 7f924d7fa700 10 client.4132.objecter ms_dispatch 0x7f9258001550 osd_map(4..4 src has 1..4) v4 2023-09-30T02:05:15.784 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.783+0000 7f924d7fa700 3 client.4132.objecter handle_osd_map got epochs [4,4] > 0 2023-09-30T02:05:15.784 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.783+0000 7f924d7fa700 3 client.4132.objecter handle_osd_map decoding full epoch 4 2023-09-30T02:05:15.784 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.783+0000 7f924d7fa700 20 client.4132.objecter dump_active .. 0 homeless 2023-09-30T02:05:15.784 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.783+0000 7f924d7fa700 1 -- 172.21.15.110:0/3733126885 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f9250024420 con 0x7f92581420c0 2023-09-30T02:05:15.784 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.783+0000 7f924d7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:05:15.784 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.783+0000 7f924d7fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:05:15.811 INFO:tasks.ceph.mgr.y.smithi059.stderr:2023-09-30T02:05:15.810+0000 7fc8624c21c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-09-30T02:05:15.866 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.866+0000 7f925f371700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "b276ed4b-317b-4ddf-96ab-7ad71b8e3ff7", "id": 2}] 2023-09-30T02:05:15.866 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.866+0000 7f925f371700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "b276ed4b-317b-4ddf-96ab-7ad71b8e3ff7", "id": 2}] 2023-09-30T02:05:15.866 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.866+0000 7f925f371700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:05:15.866 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.866+0000 7f925f371700 1 -- 172.21.15.110:0/3733126885 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd new", "uuid": "b276ed4b-317b-4ddf-96ab-7ad71b8e3ff7", "id": 2} v 0) v1 -- 0x7f925806a720 con 0x7f92581420c0 2023-09-30T02:05:15.868 INFO:tasks.ceph.mgr.x.smithi110.stderr:2023-09-30T02:05:15.869+0000 7fae141da1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-09-30T02:05:15.892 INFO:tasks.ceph.mgr.y.smithi059.stderr:2023-09-30T02:05:15.891+0000 7fc8624c21c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-09-30T02:05:15.937 INFO:tasks.ceph.mgr.x.smithi110.stderr:2023-09-30T02:05:15.937+0000 7fae141da1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-09-30T02:05:15.994 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.995+0000 7f924d7fa700 1 -- 172.21.15.110:0/3733126885 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b276ed4b-317b-4ddf-96ab-7ad71b8e3ff7", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f9250024920 con 0x7f92581420c0 2023-09-30T02:05:15.995 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.995+0000 7f924d7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "b276ed4b-317b-4ddf-96ab-7ad71b8e3ff7", "id": 2}] 2023-09-30T02:05:15.995 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.995+0000 7f924d7fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:05:15.995 INFO:teuthology.orchestra.run.smithi110.stdout:2 2023-09-30T02:05:15.996 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.995+0000 7f925f371700 10 librados: watch_flush enter 2023-09-30T02:05:15.996 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.996+0000 7f925f371700 10 librados: watch_flush exit 2023-09-30T02:05:15.996 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.996+0000 7f925f371700 20 client.4132.objecter shutdown clearing up homeless session... 2023-09-30T02:05:15.996 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.996+0000 7f925f371700 10 client.4132.objecter successfully canceled tick 2023-09-30T02:05:15.997 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.996+0000 7f925f371700 10 monclient: shutdown 2023-09-30T02:05:15.997 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.996+0000 7f925f371700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:05:15.997 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.996+0000 7f925f371700 1 -- 172.21.15.110:0/3733126885 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f92581420c0 msgr2=0x7f92580b2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:15.997 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.996+0000 7f925f371700 1 --2- 172.21.15.110:0/3733126885 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f92581420c0 0x7f92580b2e90 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f925000c840 tx=0x7f9250001670 comp rx=0 tx=0).stop 2023-09-30T02:05:15.997 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.996+0000 7f925f371700 1 -- 172.21.15.110:0/3733126885 shutdown_connections 2023-09-30T02:05:15.997 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.996+0000 7f925f371700 1 --2- 172.21.15.110:0/3733126885 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f92581420c0 0x7f92580b2e90 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:15.997 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.996+0000 7f925f371700 1 --2- 172.21.15.110:0/3733126885 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f92580a5b70 0x7f92580a7a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:15.997 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.996+0000 7f925f371700 1 --2- 172.21.15.110:0/3733126885 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f92580a5f40 0x7f92580b2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:15.997 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.996+0000 7f925f371700 1 -- 172.21.15.110:0/3733126885 >> 172.21.15.110:0/3733126885 conn(0x7f92580631f0 msgr2=0x7f92580af560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:05:15.997 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.996+0000 7f925f371700 1 -- 172.21.15.110:0/3733126885 shutdown_connections 2023-09-30T02:05:15.997 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.996+0000 7f925f371700 1 -- 172.21.15.110:0/3733126885 wait complete. 2023-09-30T02:05:15.997 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:15.996+0000 7f925f371700 1 librados: shutdown 2023-09-30T02:05:16.007 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph --cluster ceph osd new 48bfaf43-5846-4385-b9c6-6bc424cd5acd 3 2023-09-30T02:05:16.045 INFO:tasks.ceph.mgr.y.smithi059.stderr:2023-09-30T02:05:16.045+0000 7fc8624c21c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-09-30T02:05:16.123 INFO:tasks.ceph.mgr.x.smithi110.stderr:2023-09-30T02:05:16.123+0000 7fae141da1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-09-30T02:05:16.128 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.128+0000 7f65a2399700 10 monclient: get_monmap_and_config 2023-09-30T02:05:16.128 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.128+0000 7f65a2399700 10 monclient: build_initial_monmap 2023-09-30T02:05:16.128 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.128+0000 7f65a2399700 10 monclient: monmap: 2023-09-30T02:05:16.128 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2023-09-30T02:05:16.129 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:16.129 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2023-09-30T02:05:16.129603+0000 2023-09-30T02:05:16.129 INFO:teuthology.orchestra.run.smithi110.stderr:created 2023-09-30T02:05:16.129603+0000 2023-09-30T02:05:16.129 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2023-09-30T02:05:16.129 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 3 2023-09-30T02:05:16.129 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:05:16.129 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:05:16.129 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:05:16.129 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:05:16.129 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:05:16.130 INFO:teuthology.orchestra.run.smithi110.stderr: 2023-09-30T02:05:16.131 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.129+0000 7f65a2399700 1 Processor -- start 2023-09-30T02:05:16.131 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.129+0000 7f65a2399700 1 -- start start 2023-09-30T02:05:16.132 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.129+0000 7f65a2399700 10 monclient: init 2023-09-30T02:05:16.132 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.130+0000 7f65a2399700 10 monclient: _reopen_session rank -1 2023-09-30T02:05:16.132 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.130+0000 7f65a2399700 10 monclient: _add_conns ranks=[4,0,3,2,1] 2023-09-30T02:05:16.132 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.130+0000 7f65a2399700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f659c0a5b70 0x7f659c0b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:16.132 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.130+0000 7f65a2399700 10 monclient(hunting): picked mon.noname-c con 0x7f659c0a5b70 addr v2:172.21.15.110:3300/0 2023-09-30T02:05:16.132 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.130+0000 7f65a2399700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f659c0a5f40 addr v1:172.21.15.59:6789/0 2023-09-30T02:05:16.132 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.130+0000 7f65a2399700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f659c142290 0x7f659c13af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:16.132 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.130+0000 7f65a2399700 10 monclient(hunting): picked mon.noname-a con 0x7f659c142290 addr v2:172.21.15.59:3300/0 2023-09-30T02:05:16.132 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.130+0000 7f65a2399700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f659c0a75a0 con 0x7f659c0a5f40 2023-09-30T02:05:16.132 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.130+0000 7f65a2399700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:16.133 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.130+0000 7f65a2399700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f659c0a63e0 con 0x7f659c142290 2023-09-30T02:05:16.133 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.130+0000 7f65a2399700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:16.133 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.130+0000 7f65a2399700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f659c1426b0 con 0x7f659c0a5b70 2023-09-30T02:05:16.133 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.130+0000 7f65a2399700 10 monclient(hunting): _renew_subs 2023-09-30T02:05:16.133 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.130+0000 7f65a2399700 10 monclient(hunting): authenticate will time out at 1765.590417s 2023-09-30T02:05:16.133 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.133+0000 7f659bfff700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f659c0a5b70 0x7f659c0b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:16.134 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.135+0000 7f659bfff700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f659c0a5b70 0x7f659c0b1500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:51610/0 (socket says 172.21.15.110:51610) 2023-09-30T02:05:16.135 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.135+0000 7f659bfff700 1 -- 172.21.15.110:0/3734218652 learned_addr learned my addr 172.21.15.110:0/3734218652 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-09-30T02:05:16.135 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.135+0000 7f659bfff700 10 monclient(hunting): get_auth_request con 0x7f659c0a5b70 auth_method 0 2023-09-30T02:05:16.135 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.135+0000 7f659bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:16.135 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.135+0000 7f659bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:16.135 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.135+0000 7f659bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:16.136 INFO:tasks.ceph.mgr.y.smithi059.stderr:2023-09-30T02:05:16.136+0000 7fc8624c21c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-09-30T02:05:16.136 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.137+0000 7f659bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:05:16.136 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.137+0000 7f659bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:05:16.137 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.137+0000 7f659bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:05:16.140 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.140+0000 7f65a0936700 1 --2- 172.21.15.110:0/3734218652 >> v2:172.21.15.59:3300/0 conn(0x7f659c142290 0x7f659c13af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:16.140 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.141+0000 7f65a0936700 10 monclient(hunting): get_auth_request con 0x7f659c142290 auth_method 0 2023-09-30T02:05:16.140 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.141+0000 7f65a0936700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:16.141 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.141+0000 7f65a0936700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:16.141 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.141+0000 7f65a0936700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:16.142 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.141+0000 7f659bfff700 10 monclient(hunting): handle_auth_done global_id 4135 payload 274 2023-09-30T02:05:16.142 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.141+0000 7f659bfff700 1 -- 172.21.15.110:0/3734218652 >> v2:172.21.15.59:3300/0 conn(0x7f659c142290 msgr2=0x7f659c13af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:05:16.142 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.141+0000 7f659bfff700 1 --2- 172.21.15.110:0/3734218652 >> v2:172.21.15.59:3300/0 conn(0x7f659c142290 0x7f659c13af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:16.142 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.141+0000 7f659bfff700 1 -- 172.21.15.110:0/3734218652 >> v1:172.21.15.59:6789/0 conn(0x7f659c0a5f40 legacy=0x7f659c141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:05:16.142 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.141+0000 7f659bfff700 10 monclient: _finish_hunting 0 2023-09-30T02:05:16.142 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.141+0000 7f659bfff700 1 monclient: found mon.noname-c 2023-09-30T02:05:16.142 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.141+0000 7f659bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:05:16.142 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.141+0000 7f659bfff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3300/0 2023-09-30T02:05:16.142 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.141+0000 7f659bfff700 1 -- 172.21.15.110:0/3734218652 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f659c0af3f0 con 0x7f659c0a5b70 2023-09-30T02:05:16.144 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.145+0000 7f659bfff700 1 --2- 172.21.15.110:0/3734218652 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f659c0a5b70 0x7f659c0b1500 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f659c0b49e0 tx=0x7f658c009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f1fb8486a788ecba server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:05:16.151 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.151+0000 7f659affd700 1 -- 172.21.15.110:0/3734218652 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f658c00b820 con 0x7f659c0a5b70 2023-09-30T02:05:16.152 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.151+0000 7f659affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:05:16.152 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.151+0000 7f659affd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:05:16.152 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.151+0000 7f659affd700 10 monclient: dump: 2023-09-30T02:05:16.152 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2023-09-30T02:05:16.152 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:16.152 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:16.152 INFO:teuthology.orchestra.run.smithi110.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:16.153 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2023-09-30T02:05:16.153 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 3 2023-09-30T02:05:16.153 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:05:16.153 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:05:16.153 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:05:16.153 INFO:teuthology.orchestra.run.smithi110.stderr: 2023-09-30T02:05:16.153 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.151+0000 7f659affd700 10 monclient: _finish_auth 0 2023-09-30T02:05:16.153 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.151+0000 7f659affd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:05:16.153 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.151+0000 7f659affd700 1 -- 172.21.15.110:0/3734218652 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f658c00b980 con 0x7f659c0a5b70 2023-09-30T02:05:16.154 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.151+0000 7f65a2399700 5 monclient: authenticate success, global_id 4135 2023-09-30T02:05:16.154 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.151+0000 7f65a2399700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:05:16.154 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.151+0000 7f659affd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:05:16.154 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.151+0000 7f65a2399700 10 monclient: get_monmap_and_config success 2023-09-30T02:05:16.154 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.151+0000 7f659affd700 1 -- 172.21.15.110:0/3734218652 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f658c00bd20 con 0x7f659c0a5b70 2023-09-30T02:05:16.154 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.151+0000 7f65a2399700 4 set_mon_vals no callback set 2023-09-30T02:05:16.154 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.151+0000 7f659affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:05:16.154 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.151+0000 7f659affd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:05:16.155 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.151+0000 7f659affd700 10 monclient: dump: 2023-09-30T02:05:16.155 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2023-09-30T02:05:16.155 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:16.155 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:16.155 INFO:teuthology.orchestra.run.smithi110.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:16.155 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2023-09-30T02:05:16.155 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 3 2023-09-30T02:05:16.156 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:05:16.156 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:05:16.156 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:05:16.156 INFO:teuthology.orchestra.run.smithi110.stderr: 2023-09-30T02:05:16.160 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.152+0000 7f65a2399700 10 monclient: shutdown 2023-09-30T02:05:16.161 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.152+0000 7f65a2399700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:05:16.161 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.152+0000 7f65a2399700 1 -- 172.21.15.110:0/3734218652 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f659c0a5b70 msgr2=0x7f659c0b1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:16.161 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.152+0000 7f65a2399700 1 --2- 172.21.15.110:0/3734218652 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f659c0a5b70 0x7f659c0b1500 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f659c0b49e0 tx=0x7f658c009250 comp rx=0 tx=0).stop 2023-09-30T02:05:16.161 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.152+0000 7f65a2399700 1 -- 172.21.15.110:0/3734218652 shutdown_connections 2023-09-30T02:05:16.161 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.152+0000 7f65a2399700 1 --2- 172.21.15.110:0/3734218652 >> v2:172.21.15.59:3300/0 conn(0x7f659c142290 0x7f659c13af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:16.161 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.152+0000 7f65a2399700 1 --2- 172.21.15.110:0/3734218652 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f659c0a5b70 0x7f659c0b1500 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:16.161 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.152+0000 7f65a2399700 1 -- 172.21.15.110:0/3734218652 >> 172.21.15.110:0/3734218652 conn(0x7f659c0631f0 msgr2=0x7f659c0635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:05:16.161 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.152+0000 7f65a2399700 1 -- 172.21.15.110:0/3734218652 shutdown_connections 2023-09-30T02:05:16.161 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.152+0000 7f65a2399700 1 -- 172.21.15.110:0/3734218652 wait complete. 2023-09-30T02:05:16.161 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.153+0000 7f65a2399700 10 monclient: build_initial_monmap 2023-09-30T02:05:16.162 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.153+0000 7f65a2399700 10 monclient: monmap: 2023-09-30T02:05:16.162 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2023-09-30T02:05:16.162 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:05:16.162 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2023-09-30T02:05:16.162 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2023-09-30T02:05:16.162 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2023-09-30T02:05:16.162 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2023-09-30T02:05:16.162 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:05:16.162 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:05:16.162 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:05:16.163 INFO:teuthology.orchestra.run.smithi110.stderr: 2023-09-30T02:05:16.163 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.153+0000 7f65a2399700 1 librados: starting msgr at 2023-09-30T02:05:16.163 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.153+0000 7f65a2399700 1 librados: starting objecter 2023-09-30T02:05:16.163 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.153+0000 7f65a2399700 1 Processor -- start 2023-09-30T02:05:16.163 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.153+0000 7f65a2399700 1 -- start start 2023-09-30T02:05:16.163 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.153+0000 7f65a2399700 1 librados: setting wanted keys 2023-09-30T02:05:16.163 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.153+0000 7f65a2399700 1 librados: calling monclient init 2023-09-30T02:05:16.163 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.153+0000 7f65a2399700 10 monclient: init 2023-09-30T02:05:16.163 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.154+0000 7f65a2399700 10 monclient: _reopen_session rank -1 2023-09-30T02:05:16.164 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.154+0000 7f65a2399700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T02:05:16.164 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.154+0000 7f65a2399700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f659c0a5b70 0x7f659c151340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:16.164 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.154+0000 7f65a2399700 10 monclient(hunting): picked mon.noname-c con 0x7f659c0a5b70 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:05:16.164 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.154+0000 7f65a2399700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f659c0a5f40 0x7f659c151880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:16.164 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.154+0000 7f65a2399700 10 monclient(hunting): picked mon.noname-b con 0x7f659c0a5f40 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:05:16.164 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.154+0000 7f65a2399700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f659c142290 0x7f659c1401a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:16.164 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.154+0000 7f65a2399700 10 monclient(hunting): picked mon.noname-a con 0x7f659c142290 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:05:16.164 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.154+0000 7f65a2399700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:16.164 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.154+0000 7f65a2399700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f659c13fd10 con 0x7f659c142290 2023-09-30T02:05:16.164 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.154+0000 7f65a2399700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:16.165 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.154+0000 7f65a2399700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f659c13e360 con 0x7f659c0a5f40 2023-09-30T02:05:16.165 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.154+0000 7f65a2399700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:16.165 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.154+0000 7f65a2399700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f659c13e520 con 0x7f659c0a5b70 2023-09-30T02:05:16.165 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.154+0000 7f65a2399700 10 monclient(hunting): _renew_subs 2023-09-30T02:05:16.165 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.154+0000 7f65a2399700 10 monclient(hunting): authenticate will time out at 1765.614483s 2023-09-30T02:05:16.165 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.155+0000 7f659b7fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f659c0a5f40 0x7f659c151880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:16.165 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.155+0000 7f659bfff700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f659c0a5b70 0x7f659c151340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:16.165 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.155+0000 7f659b7fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f659c0a5f40 0x7f659c151880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:51624/0 (socket says 172.21.15.110:51624) 2023-09-30T02:05:16.165 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.155+0000 7f659b7fe700 1 -- 172.21.15.110:0/836796759 learned_addr learned my addr 172.21.15.110:0/836796759 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-09-30T02:05:16.165 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.155+0000 7f659b7fe700 10 monclient(hunting): get_auth_request con 0x7f659c0a5f40 auth_method 0 2023-09-30T02:05:16.165 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.155+0000 7f659b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:16.166 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.155+0000 7f659b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:16.166 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.155+0000 7f659b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:16.166 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.156+0000 7f659bfff700 10 monclient(hunting): get_auth_request con 0x7f659c0a5b70 auth_method 0 2023-09-30T02:05:16.166 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.156+0000 7f659bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:16.166 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.156+0000 7f659bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:16.166 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.156+0000 7f659bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:16.166 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.156+0000 7f65a0936700 1 --2- 172.21.15.110:0/836796759 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f659c142290 0x7f659c1401a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:16.166 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.156+0000 7f659b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:05:16.166 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.156+0000 7f659b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:05:16.166 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.156+0000 7f659b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:05:16.167 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.156+0000 7f65a0936700 10 monclient(hunting): get_auth_request con 0x7f659c142290 auth_method 0 2023-09-30T02:05:16.167 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.156+0000 7f65a0936700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:16.167 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.156+0000 7f65a0936700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:16.167 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.156+0000 7f65a0936700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:16.167 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.157+0000 7f659bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:05:16.167 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.157+0000 7f659bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:05:16.167 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.157+0000 7f659bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:05:16.167 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.157+0000 7f65a0936700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:05:16.167 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.157+0000 7f65a0936700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:05:16.168 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.157+0000 7f65a0936700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:05:16.168 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.161+0000 7f65a0936700 10 monclient(hunting): handle_auth_done global_id 4128 payload 824 2023-09-30T02:05:16.168 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.161+0000 7f65a0936700 1 -- 172.21.15.110:0/836796759 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f659c0a5b70 msgr2=0x7f659c151340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:16.168 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.161+0000 7f65a0936700 1 --2- 172.21.15.110:0/836796759 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f659c0a5b70 0x7f659c151340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:16.168 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.161+0000 7f65a0936700 1 -- 172.21.15.110:0/836796759 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f659c0a5f40 msgr2=0x7f659c151880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:16.168 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.161+0000 7f65a0936700 1 --2- 172.21.15.110:0/836796759 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f659c0a5f40 0x7f659c151880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:16.168 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.161+0000 7f65a0936700 10 monclient: _finish_hunting 0 2023-09-30T02:05:16.168 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.161+0000 7f65a0936700 1 monclient: found mon.noname-a 2023-09-30T02:05:16.168 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.161+0000 7f65a0936700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:05:16.168 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.161+0000 7f65a0936700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.59:3300/0 2023-09-30T02:05:16.169 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.161+0000 7f65a0936700 1 -- 172.21.15.110:0/836796759 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f658c0094f0 con 0x7f659c142290 2023-09-30T02:05:16.171 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.163+0000 7f65a0936700 1 --2- 172.21.15.110:0/836796759 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f659c142290 0x7f659c1401a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f6594007ea0 tx=0x7f65940097b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:05:16.171 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.163+0000 7f6583fff700 10 client.?.objecter ms_handle_connect 0x7f659c142290 2023-09-30T02:05:16.171 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.163+0000 7f6583fff700 10 client.?.objecter resend_mon_ops 2023-09-30T02:05:16.171 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.168+0000 7f6583fff700 1 -- 172.21.15.110:0/836796759 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f659400eac0 con 0x7f659c142290 2023-09-30T02:05:16.171 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.168+0000 7f6583fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:05:16.171 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.168+0000 7f6583fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T02:05:16.171 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.168+0000 7f6583fff700 10 monclient: dump: 2023-09-30T02:05:16.172 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2023-09-30T02:05:16.172 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:16.172 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:16.172 INFO:teuthology.orchestra.run.smithi110.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:16.172 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2023-09-30T02:05:16.172 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 3 2023-09-30T02:05:16.172 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:05:16.172 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:05:16.172 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:05:16.172 INFO:teuthology.orchestra.run.smithi110.stderr: 2023-09-30T02:05:16.173 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.168+0000 7f6583fff700 10 monclient: _finish_auth 0 2023-09-30T02:05:16.173 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.168+0000 7f6583fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:05:16.173 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.168+0000 7f6583fff700 1 -- 172.21.15.110:0/836796759 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f659400ec20 con 0x7f659c142290 2023-09-30T02:05:16.173 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.168+0000 7f65a2399700 5 monclient: authenticate success, global_id 4128 2023-09-30T02:05:16.173 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.168+0000 7f65a2399700 10 monclient: _renew_subs 2023-09-30T02:05:16.173 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.168+0000 7f65a2399700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:05:16.173 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.168+0000 7f65a2399700 1 -- 172.21.15.110:0/836796759 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f659c151dc0 con 0x7f659c142290 2023-09-30T02:05:16.173 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.168+0000 7f6583fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:05:16.173 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.168+0000 7f6583fff700 1 -- 172.21.15.110:0/836796759 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6594014420 con 0x7f659c142290 2023-09-30T02:05:16.173 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.168+0000 7f6583fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:05:16.173 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.168+0000 7f6583fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T02:05:16.174 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.168+0000 7f6583fff700 10 monclient: dump: 2023-09-30T02:05:16.174 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2023-09-30T02:05:16.174 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:16.174 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:16.174 INFO:teuthology.orchestra.run.smithi110.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:16.174 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2023-09-30T02:05:16.174 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 3 2023-09-30T02:05:16.174 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:05:16.174 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:05:16.174 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:05:16.175 INFO:teuthology.orchestra.run.smithi110.stderr: 2023-09-30T02:05:16.175 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.168+0000 7f659a7fc700 4 set_mon_vals no callback set 2023-09-30T02:05:16.175 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.168+0000 7f65a2399700 10 client.4128.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:05:16.175 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.168+0000 7f65a2399700 10 monclient: _renew_subs 2023-09-30T02:05:16.175 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.168+0000 7f65a2399700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:05:16.175 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.168+0000 7f65a2399700 1 -- 172.21.15.110:0/836796759 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f659c140920 con 0x7f659c142290 2023-09-30T02:05:16.175 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.168+0000 7f65a2399700 1 librados: init done 2023-09-30T02:05:16.175 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.169+0000 7f65a2399700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:05:16.175 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.169+0000 7f65a2399700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:05:16.175 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.169+0000 7f65a2399700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:05:16.175 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.169+0000 7f65a2399700 1 -- 172.21.15.110:0/836796759 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f659c13fec0 con 0x7f659c142290 2023-09-30T02:05:16.176 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.171+0000 7f6583fff700 1 -- 172.21.15.110:0/836796759 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f6594003020 con 0x7f659c142290 2023-09-30T02:05:16.210 INFO:tasks.ceph.mgr.x.smithi110.stderr:2023-09-30T02:05:16.210+0000 7fae141da1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-09-30T02:05:16.285 INFO:tasks.ceph.mgr.x.smithi110.stderr:2023-09-30T02:05:16.285+0000 7fae141da1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-09-30T02:05:16.308 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.308+0000 7f6583fff700 1 -- 172.21.15.110:0/836796759 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1066+0+0 (secure 0 0 0) 0x7f6594026380 con 0x7f659c142290 2023-09-30T02:05:16.309 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.308+0000 7f6583fff700 10 client.4128.objecter ms_dispatch 0x7f659c001550 osd_map(5..5 src has 1..5) v4 2023-09-30T02:05:16.309 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.308+0000 7f6583fff700 3 client.4128.objecter handle_osd_map got epochs [5,5] > 0 2023-09-30T02:05:16.309 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.308+0000 7f6583fff700 3 client.4128.objecter handle_osd_map decoding full epoch 5 2023-09-30T02:05:16.309 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.308+0000 7f6583fff700 20 client.4128.objecter dump_active .. 0 homeless 2023-09-30T02:05:16.309 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.308+0000 7f6583fff700 1 -- 172.21.15.110:0/836796759 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f659c13fec0 con 0x7f659c142290 2023-09-30T02:05:16.309 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.308+0000 7f6583fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:05:16.309 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.308+0000 7f6583fff700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:05:16.352 INFO:tasks.ceph.mgr.y.smithi059.stderr:2023-09-30T02:05:16.352+0000 7fc8624c21c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-09-30T02:05:16.380 INFO:tasks.ceph.mgr.x.smithi110.stderr:2023-09-30T02:05:16.380+0000 7fae141da1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-09-30T02:05:16.392 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.392+0000 7f65a2399700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "48bfaf43-5846-4385-b9c6-6bc424cd5acd", "id": 3}] 2023-09-30T02:05:16.392 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.392+0000 7f65a2399700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "48bfaf43-5846-4385-b9c6-6bc424cd5acd", "id": 3}] 2023-09-30T02:05:16.392 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.392+0000 7f65a2399700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:05:16.392 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.392+0000 7f65a2399700 1 -- 172.21.15.110:0/836796759 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd new", "uuid": "48bfaf43-5846-4385-b9c6-6bc424cd5acd", "id": 3} v 0) v1 -- 0x7f659c064a00 con 0x7f659c142290 2023-09-30T02:05:16.440 INFO:tasks.ceph.mgr.y.smithi059.stderr:2023-09-30T02:05:16.440+0000 7fc8624c21c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-09-30T02:05:16.450 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.451+0000 7f6583fff700 1 -- 172.21.15.110:0/836796759 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "48bfaf43-5846-4385-b9c6-6bc424cd5acd", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f6594014a90 con 0x7f659c142290 2023-09-30T02:05:16.451 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.451+0000 7f6583fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "48bfaf43-5846-4385-b9c6-6bc424cd5acd", "id": 3}] 2023-09-30T02:05:16.451 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.451+0000 7f6583fff700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:05:16.451 INFO:teuthology.orchestra.run.smithi110.stdout:3 2023-09-30T02:05:16.452 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.452+0000 7f65a2399700 10 librados: watch_flush enter 2023-09-30T02:05:16.452 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.452+0000 7f65a2399700 10 librados: watch_flush exit 2023-09-30T02:05:16.452 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.452+0000 7f65a2399700 20 client.4128.objecter shutdown clearing up homeless session... 2023-09-30T02:05:16.453 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.452+0000 7f65a2399700 10 client.4128.objecter successfully canceled tick 2023-09-30T02:05:16.453 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.452+0000 7f65a2399700 10 monclient: shutdown 2023-09-30T02:05:16.453 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.452+0000 7f65a2399700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:05:16.453 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.452+0000 7f65a2399700 1 -- 172.21.15.110:0/836796759 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f659c142290 msgr2=0x7f659c1401a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:16.453 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.452+0000 7f65a2399700 1 --2- 172.21.15.110:0/836796759 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f659c142290 0x7f659c1401a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f6594007ea0 tx=0x7f65940097b0 comp rx=0 tx=0).stop 2023-09-30T02:05:16.453 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.452+0000 7f65a2399700 1 -- 172.21.15.110:0/836796759 shutdown_connections 2023-09-30T02:05:16.453 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.452+0000 7f65a2399700 1 --2- 172.21.15.110:0/836796759 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f659c142290 0x7f659c1401a0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:16.453 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.452+0000 7f65a2399700 1 --2- 172.21.15.110:0/836796759 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f659c0a5b70 0x7f659c151340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:16.453 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.452+0000 7f65a2399700 1 --2- 172.21.15.110:0/836796759 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f659c0a5f40 0x7f659c151880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:16.454 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.452+0000 7f65a2399700 1 -- 172.21.15.110:0/836796759 >> 172.21.15.110:0/836796759 conn(0x7f659c0631f0 msgr2=0x7f659c070f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:05:16.454 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.452+0000 7f65a2399700 1 -- 172.21.15.110:0/836796759 shutdown_connections 2023-09-30T02:05:16.454 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.452+0000 7f65a2399700 1 -- 172.21.15.110:0/836796759 wait complete. 2023-09-30T02:05:16.454 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.452+0000 7f65a2399700 1 librados: shutdown 2023-09-30T02:05:16.464 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph --cluster ceph osd new 76fa5443-1571-4e72-9d22-6ab19e59839a 4 2023-09-30T02:05:16.529 INFO:tasks.ceph.mgr.y.smithi059.stderr:2023-09-30T02:05:16.529+0000 7fc8624c21c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-09-30T02:05:16.573 INFO:tasks.ceph.mgr.x.smithi110.stderr:2023-09-30T02:05:16.574+0000 7fae141da1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-09-30T02:05:16.593 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.593+0000 7f7477864700 10 monclient: get_monmap_and_config 2023-09-30T02:05:16.593 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.593+0000 7f7477864700 10 monclient: build_initial_monmap 2023-09-30T02:05:16.593 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.593+0000 7f7477864700 10 monclient: monmap: 2023-09-30T02:05:16.593 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2023-09-30T02:05:16.593 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:16.593 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2023-09-30T02:05:16.594167+0000 2023-09-30T02:05:16.593 INFO:teuthology.orchestra.run.smithi110.stderr:created 2023-09-30T02:05:16.594167+0000 2023-09-30T02:05:16.594 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2023-09-30T02:05:16.594 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 3 2023-09-30T02:05:16.594 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:05:16.594 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:05:16.594 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:05:16.594 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:05:16.594 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:05:16.594 INFO:teuthology.orchestra.run.smithi110.stderr: 2023-09-30T02:05:16.596 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.594+0000 7f7477864700 1 Processor -- start 2023-09-30T02:05:16.596 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.594+0000 7f7477864700 1 -- start start 2023-09-30T02:05:16.596 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.594+0000 7f7477864700 10 monclient: init 2023-09-30T02:05:16.596 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.594+0000 7f7477864700 10 monclient: _reopen_session rank -1 2023-09-30T02:05:16.596 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.594+0000 7f7477864700 10 monclient: _add_conns ranks=[2,1,4,3,0] 2023-09-30T02:05:16.596 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.595+0000 7f7477864700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f74700a5b70 0x7f74700b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:16.597 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.595+0000 7f7477864700 10 monclient(hunting): picked mon.noname-b con 0x7f74700a5b70 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:05:16.597 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.595+0000 7f7477864700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f74700a5f40 addr v1:172.21.15.110:6789/0 2023-09-30T02:05:16.597 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.595+0000 7f7477864700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f7470142290 0x7f747013af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:16.597 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.595+0000 7f7477864700 10 monclient(hunting): picked mon.noname-c con 0x7f7470142290 addr v2:172.21.15.110:3300/0 2023-09-30T02:05:16.597 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.595+0000 7f7477864700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74700a63b0 con 0x7f74700a5f40 2023-09-30T02:05:16.597 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.595+0000 7f7477864700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:16.597 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.595+0000 7f7477864700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74701426d0 con 0x7f7470142290 2023-09-30T02:05:16.597 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.595+0000 7f7477864700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:16.597 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.595+0000 7f7477864700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74700a75a0 con 0x7f74700a5b70 2023-09-30T02:05:16.597 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.595+0000 7f7477864700 10 monclient(hunting): _renew_subs 2023-09-30T02:05:16.597 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.595+0000 7f7477864700 10 monclient(hunting): authenticate will time out at 1766.055104s 2023-09-30T02:05:16.597 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.595+0000 7f7475e01700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f7470142290 0x7f747013af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:16.597 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.595+0000 7f7475600700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f74700a5b70 0x7f74700b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:16.598 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.596+0000 7f7474dff700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f74700a5f40 0x7f7470141b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:35286/0 (socket says 172.21.15.110:35286) 2023-09-30T02:05:16.598 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.596+0000 7f7474dff700 1 -- 172.21.15.110:0/1980685349 learned_addr learned my addr 172.21.15.110:0/1980685349 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-09-30T02:05:16.598 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.596+0000 7f7475e01700 10 monclient(hunting): get_auth_request con 0x7f7470142290 auth_method 0 2023-09-30T02:05:16.598 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.596+0000 7f7475e01700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:16.598 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.596+0000 7f7475e01700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:16.598 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.596+0000 7f7475e01700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:16.598 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.596+0000 7f7475600700 10 monclient(hunting): get_auth_request con 0x7f74700a5b70 auth_method 0 2023-09-30T02:05:16.598 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.596+0000 7f7475600700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:16.598 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.596+0000 7f7475600700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:16.598 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.596+0000 7f7475600700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:16.599 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.597+0000 7f7475e01700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:05:16.599 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.597+0000 7f7475e01700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:05:16.599 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.597+0000 7f7475e01700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:05:16.599 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.597+0000 7f7475600700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:05:16.599 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.597+0000 7f7475600700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:05:16.600 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.597+0000 7f7475600700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:05:16.600 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.599+0000 7f7467fff700 1 -- 172.21.15.110:0/1980685349 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1254204218 0 0) 0x7f74700a63b0 con 0x7f74700a5f40 2023-09-30T02:05:16.600 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.599+0000 7f7467fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:16.600 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.599+0000 7f7467fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:16.600 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.599+0000 7f7467fff700 10 monclient(hunting): my global_id is 4144 2023-09-30T02:05:16.600 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.599+0000 7f7467fff700 1 -- 172.21.15.110:0/1980685349 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f745c003570 con 0x7f74700a5f40 2023-09-30T02:05:16.601 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.600+0000 7f7475600700 10 monclient(hunting): handle_auth_done global_id 4118 payload 274 2023-09-30T02:05:16.601 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.600+0000 7f7475600700 1 -- 172.21.15.110:0/1980685349 >> v2:172.21.15.110:3300/0 conn(0x7f7470142290 msgr2=0x7f747013af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:05:16.601 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.600+0000 7f7475600700 1 --2- 172.21.15.110:0/1980685349 >> v2:172.21.15.110:3300/0 conn(0x7f7470142290 0x7f747013af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:16.601 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.600+0000 7f7475600700 1 -- 172.21.15.110:0/1980685349 >> v1:172.21.15.110:6789/0 conn(0x7f74700a5f40 legacy=0x7f7470141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:16.601 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.600+0000 7f7475600700 10 monclient: _finish_hunting 0 2023-09-30T02:05:16.601 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.600+0000 7f7475600700 1 monclient: found mon.noname-b 2023-09-30T02:05:16.601 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.600+0000 7f7475600700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:05:16.601 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.600+0000 7f7475600700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.59:3301/0 2023-09-30T02:05:16.601 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.600+0000 7f7475600700 1 -- 172.21.15.110:0/1980685349 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74700af3f0 con 0x7f74700a5b70 2023-09-30T02:05:16.602 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.603+0000 7f7475600700 1 --2- 172.21.15.110:0/1980685349 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f74700a5b70 0x7f74700b1500 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f746c0098e0 tx=0x7f746c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1036c478b65ec69d server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:05:16.607 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.607+0000 7f7467fff700 1 -- 172.21.15.110:0/1980685349 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f746c011630 con 0x7f74700a5b70 2023-09-30T02:05:16.607 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.607+0000 7f7467fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:05:16.607 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.607+0000 7f7467fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:05:16.607 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.607+0000 7f7467fff700 10 monclient: dump: 2023-09-30T02:05:16.607 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2023-09-30T02:05:16.607 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:16.608 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:16.608 INFO:teuthology.orchestra.run.smithi110.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:16.608 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2023-09-30T02:05:16.608 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 3 2023-09-30T02:05:16.608 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:05:16.608 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:05:16.608 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:05:16.608 INFO:teuthology.orchestra.run.smithi110.stderr: 2023-09-30T02:05:16.608 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.607+0000 7f7467fff700 10 monclient: _finish_auth 0 2023-09-30T02:05:16.608 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.607+0000 7f7467fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:05:16.608 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.607+0000 7f7467fff700 1 -- 172.21.15.110:0/1980685349 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f746c011790 con 0x7f74700a5b70 2023-09-30T02:05:16.608 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.607+0000 7f7477864700 5 monclient: authenticate success, global_id 4118 2023-09-30T02:05:16.609 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.607+0000 7f7477864700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:05:16.609 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.607+0000 7f7467fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:05:16.609 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.607+0000 7f7467fff700 1 -- 172.21.15.110:0/1980685349 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f746c011b30 con 0x7f74700a5b70 2023-09-30T02:05:16.609 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.607+0000 7f7467fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:05:16.609 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.607+0000 7f7467fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:05:16.609 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.607+0000 7f7467fff700 10 monclient: dump: 2023-09-30T02:05:16.609 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2023-09-30T02:05:16.609 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:16.609 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:16.609 INFO:teuthology.orchestra.run.smithi110.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:16.609 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2023-09-30T02:05:16.609 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 3 2023-09-30T02:05:16.609 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:05:16.610 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:05:16.610 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:05:16.610 INFO:teuthology.orchestra.run.smithi110.stderr: 2023-09-30T02:05:16.610 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.607+0000 7f7477864700 10 monclient: get_monmap_and_config success 2023-09-30T02:05:16.610 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.607+0000 7f7477864700 4 set_mon_vals no callback set 2023-09-30T02:05:16.613 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.608+0000 7f7477864700 10 monclient: shutdown 2023-09-30T02:05:16.613 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.608+0000 7f7477864700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:05:16.614 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.608+0000 7f7477864700 1 -- 172.21.15.110:0/1980685349 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f74700a5b70 msgr2=0x7f74700b1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:16.614 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.608+0000 7f7477864700 1 --2- 172.21.15.110:0/1980685349 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f74700a5b70 0x7f74700b1500 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f746c0098e0 tx=0x7f746c0091a0 comp rx=0 tx=0).stop 2023-09-30T02:05:16.614 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.608+0000 7f7477864700 1 -- 172.21.15.110:0/1980685349 shutdown_connections 2023-09-30T02:05:16.614 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.608+0000 7f7477864700 1 --2- 172.21.15.110:0/1980685349 >> v2:172.21.15.110:3300/0 conn(0x7f7470142290 0x7f747013af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:16.614 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.608+0000 7f7477864700 1 --2- 172.21.15.110:0/1980685349 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f74700a5b70 0x7f74700b1500 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:16.614 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.608+0000 7f7477864700 1 -- 172.21.15.110:0/1980685349 >> 172.21.15.110:0/1980685349 conn(0x7f74700631f0 msgr2=0x7f74700635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:05:16.614 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.608+0000 7f7477864700 1 -- 172.21.15.110:0/1980685349 shutdown_connections 2023-09-30T02:05:16.614 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.608+0000 7f7477864700 1 -- 172.21.15.110:0/1980685349 wait complete. 2023-09-30T02:05:16.614 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.609+0000 7f7477864700 10 monclient: build_initial_monmap 2023-09-30T02:05:16.614 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.609+0000 7f7477864700 10 monclient: monmap: 2023-09-30T02:05:16.614 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2023-09-30T02:05:16.615 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:05:16.615 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2023-09-30T02:05:16.615 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2023-09-30T02:05:16.615 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2023-09-30T02:05:16.615 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2023-09-30T02:05:16.615 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:05:16.615 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:05:16.615 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:05:16.615 INFO:teuthology.orchestra.run.smithi110.stderr: 2023-09-30T02:05:16.615 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.609+0000 7f7477864700 1 librados: starting msgr at 2023-09-30T02:05:16.615 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.609+0000 7f7477864700 1 librados: starting objecter 2023-09-30T02:05:16.615 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.609+0000 7f7477864700 1 Processor -- start 2023-09-30T02:05:16.615 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.610+0000 7f7477864700 1 -- start start 2023-09-30T02:05:16.616 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.610+0000 7f7477864700 1 librados: setting wanted keys 2023-09-30T02:05:16.616 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.610+0000 7f7477864700 1 librados: calling monclient init 2023-09-30T02:05:16.616 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.610+0000 7f7477864700 10 monclient: init 2023-09-30T02:05:16.616 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.611+0000 7f7477864700 10 monclient: _reopen_session rank -1 2023-09-30T02:05:16.616 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.611+0000 7f7477864700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T02:05:16.616 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.611+0000 7f7477864700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f74700a5b70 0x7f747014e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:16.616 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.611+0000 7f7477864700 10 monclient(hunting): picked mon.noname-c con 0x7f74700a5b70 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:05:16.616 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.611+0000 7f7477864700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f74700a5f40 0x7f74700a79b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:16.616 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.611+0000 7f7477864700 10 monclient(hunting): picked mon.noname-b con 0x7f74700a5f40 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:05:16.616 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.611+0000 7f7477864700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7470142290 0x7f74700a7ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:16.616 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.611+0000 7f7477864700 10 monclient(hunting): picked mon.noname-a con 0x7f7470142290 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:05:16.616 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.611+0000 7f7477864700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:16.617 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.611+0000 7f7477864700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f747014ea80 con 0x7f7470142290 2023-09-30T02:05:16.617 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.611+0000 7f7477864700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:16.617 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.611+0000 7f7477864700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f747014e750 con 0x7f74700a5f40 2023-09-30T02:05:16.617 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.611+0000 7f7477864700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:16.617 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.611+0000 7f7477864700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74700ae950 con 0x7f74700a5b70 2023-09-30T02:05:16.617 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.611+0000 7f7477864700 10 monclient(hunting): _renew_subs 2023-09-30T02:05:16.617 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.611+0000 7f7477864700 10 monclient(hunting): authenticate will time out at 1766.071168s 2023-09-30T02:05:16.617 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.611+0000 7f7475600700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f74700a5b70 0x7f747014e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:16.617 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.611+0000 7f7475e01700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7470142290 0x7f74700a7ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:16.617 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.612+0000 7f7475600700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f74700a5b70 0x7f747014e210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.110:56698/0 (socket says 172.21.15.110:56698) 2023-09-30T02:05:16.617 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.612+0000 7f7475600700 1 -- 172.21.15.110:0/2948247726 learned_addr learned my addr 172.21.15.110:0/2948247726 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-09-30T02:05:16.617 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.612+0000 7f7475600700 10 monclient(hunting): get_auth_request con 0x7f74700a5b70 auth_method 0 2023-09-30T02:05:16.618 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.612+0000 7f7475600700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:16.618 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.612+0000 7f7475600700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:16.618 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.612+0000 7f7475600700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:16.618 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.612+0000 7f7475e01700 10 monclient(hunting): get_auth_request con 0x7f7470142290 auth_method 0 2023-09-30T02:05:16.618 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.612+0000 7f7475e01700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:16.618 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.612+0000 7f7475e01700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:16.618 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.612+0000 7f7475e01700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:16.618 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.612+0000 7f7475600700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:05:16.618 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.612+0000 7f7475600700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:05:16.618 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.612+0000 7f7475600700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:05:16.618 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.612+0000 7f7474dff700 1 --2- 172.21.15.110:0/2948247726 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f74700a5f40 0x7f74700a79b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:16.618 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.612+0000 7f7475e01700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:05:16.619 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.613+0000 7f7475e01700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:05:16.619 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.613+0000 7f7475e01700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:05:16.619 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.613+0000 7f7474dff700 10 monclient(hunting): get_auth_request con 0x7f74700a5f40 auth_method 0 2023-09-30T02:05:16.619 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.613+0000 7f7474dff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:16.619 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.613+0000 7f7474dff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:16.619 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.613+0000 7f7474dff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:16.619 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.613+0000 7f7474dff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:05:16.619 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.613+0000 7f7474dff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:05:16.619 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.613+0000 7f7474dff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:05:16.620 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.616+0000 7f7475600700 10 monclient(hunting): handle_auth_done global_id 4121 payload 824 2023-09-30T02:05:16.620 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.616+0000 7f7475600700 1 -- 172.21.15.110:0/2948247726 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f74700a5f40 msgr2=0x7f74700a79b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:16.620 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.616+0000 7f7475600700 1 --2- 172.21.15.110:0/2948247726 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f74700a5f40 0x7f74700a79b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:16.620 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.616+0000 7f7475600700 1 -- 172.21.15.110:0/2948247726 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7470142290 msgr2=0x7f74700a7ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:16.620 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.616+0000 7f7475600700 1 --2- 172.21.15.110:0/2948247726 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7470142290 0x7f74700a7ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:16.620 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.616+0000 7f7475600700 10 monclient: _finish_hunting 0 2023-09-30T02:05:16.620 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.616+0000 7f7475600700 1 monclient: found mon.noname-c 2023-09-30T02:05:16.621 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.616+0000 7f7475600700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:05:16.621 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.616+0000 7f7475600700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.59:3301/0 2023-09-30T02:05:16.621 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.616+0000 7f7475600700 1 -- 172.21.15.110:0/2948247726 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f746c009420 con 0x7f74700a5b70 2023-09-30T02:05:16.621 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.618+0000 7f7475600700 1 --2- 172.21.15.110:0/2948247726 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f74700a5b70 0x7f747014e210 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f74700a76e0 tx=0x7f746c017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:05:16.621 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.618+0000 7f74657fa700 10 client.?.objecter ms_handle_connect 0x7f74700a5b70 2023-09-30T02:05:16.621 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.618+0000 7f74657fa700 10 client.?.objecter resend_mon_ops 2023-09-30T02:05:16.623 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.622+0000 7f74657fa700 1 -- 172.21.15.110:0/2948247726 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f746c011900 con 0x7f74700a5b70 2023-09-30T02:05:16.623 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.622+0000 7f74657fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:05:16.623 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.622+0000 7f74657fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:05:16.623 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.622+0000 7f74657fa700 10 monclient: dump: 2023-09-30T02:05:16.623 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2023-09-30T02:05:16.623 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:16.623 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:16.624 INFO:teuthology.orchestra.run.smithi110.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:16.624 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2023-09-30T02:05:16.624 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 3 2023-09-30T02:05:16.624 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:05:16.624 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:05:16.624 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:05:16.624 INFO:teuthology.orchestra.run.smithi110.stderr: 2023-09-30T02:05:16.624 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.622+0000 7f74657fa700 10 monclient: _finish_auth 0 2023-09-30T02:05:16.624 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.622+0000 7f74657fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:05:16.624 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.622+0000 7f7477864700 5 monclient: authenticate success, global_id 4121 2023-09-30T02:05:16.624 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.622+0000 7f7477864700 10 monclient: _renew_subs 2023-09-30T02:05:16.624 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.622+0000 7f7477864700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:05:16.625 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.622+0000 7f7477864700 1 -- 172.21.15.110:0/2948247726 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f747013e330 con 0x7f74700a5b70 2023-09-30T02:05:16.625 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.622+0000 7f7477864700 10 client.4121.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:05:16.625 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.622+0000 7f7477864700 10 monclient: _renew_subs 2023-09-30T02:05:16.625 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.622+0000 7f7477864700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:05:16.625 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.622+0000 7f7477864700 1 -- 172.21.15.110:0/2948247726 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f747013ef70 con 0x7f74700a5b70 2023-09-30T02:05:16.625 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.622+0000 7f7477864700 1 librados: init done 2023-09-30T02:05:16.625 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.622+0000 7f74657fa700 1 -- 172.21.15.110:0/2948247726 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f746c0253f0 con 0x7f74700a5b70 2023-09-30T02:05:16.625 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.622+0000 7f74657fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:05:16.625 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.622+0000 7f74677fe700 4 set_mon_vals no callback set 2023-09-30T02:05:16.625 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.623+0000 7f74657fa700 1 -- 172.21.15.110:0/2948247726 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f746c025560 con 0x7f74700a5b70 2023-09-30T02:05:16.625 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.623+0000 7f74657fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:05:16.625 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.623+0000 7f74657fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:05:16.625 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.623+0000 7f74657fa700 10 monclient: dump: 2023-09-30T02:05:16.626 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2023-09-30T02:05:16.626 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:16.626 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:16.626 INFO:teuthology.orchestra.run.smithi110.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:16.626 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2023-09-30T02:05:16.626 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 3 2023-09-30T02:05:16.626 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:05:16.626 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:05:16.626 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:05:16.626 INFO:teuthology.orchestra.run.smithi110.stderr: 2023-09-30T02:05:16.626 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.623+0000 7f7477864700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:05:16.626 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.623+0000 7f7477864700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:05:16.627 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.623+0000 7f7477864700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:05:16.627 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.623+0000 7f7477864700 1 -- 172.21.15.110:0/2948247726 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74500050b0 con 0x7f74700a5b70 2023-09-30T02:05:16.627 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.626+0000 7f74657fa700 1 -- 172.21.15.110:0/2948247726 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f746c025b90 con 0x7f74700a5b70 2023-09-30T02:05:16.627 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.626+0000 7f74657fa700 1 -- 172.21.15.110:0/2948247726 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1181+0+0 (secure 0 0 0) 0x7f746c02ecc0 con 0x7f74700a5b70 2023-09-30T02:05:16.628 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.626+0000 7f74657fa700 10 client.4121.objecter ms_dispatch 0x7f7470001550 osd_map(6..6 src has 1..6) v4 2023-09-30T02:05:16.628 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.626+0000 7f74657fa700 3 client.4121.objecter handle_osd_map got epochs [6,6] > 0 2023-09-30T02:05:16.628 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.626+0000 7f74657fa700 3 client.4121.objecter handle_osd_map decoding full epoch 6 2023-09-30T02:05:16.628 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.626+0000 7f74657fa700 20 client.4121.objecter dump_active .. 0 homeless 2023-09-30T02:05:16.630 INFO:tasks.ceph.mgr.y.smithi059.stderr:2023-09-30T02:05:16.630+0000 7fc8624c21c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-09-30T02:05:16.658 INFO:tasks.ceph.mgr.x.smithi110.stderr:2023-09-30T02:05:16.658+0000 7fae141da1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-09-30T02:05:16.741 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.741+0000 7f74657fa700 1 -- 172.21.15.110:0/2948247726 <== mon.2 v2:172.21.15.59:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f746c033060 con 0x7f74700a5b70 2023-09-30T02:05:16.741 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.741+0000 7f74657fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:05:16.741 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.741+0000 7f74657fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:05:16.820 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.819+0000 7f7477864700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "76fa5443-1571-4e72-9d22-6ab19e59839a", "id": 4}] 2023-09-30T02:05:16.820 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.819+0000 7f7477864700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "76fa5443-1571-4e72-9d22-6ab19e59839a", "id": 4}] 2023-09-30T02:05:16.820 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.819+0000 7f7477864700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:05:16.820 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.819+0000 7f7477864700 1 -- 172.21.15.110:0/2948247726 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "osd new", "uuid": "76fa5443-1571-4e72-9d22-6ab19e59839a", "id": 4} v 0) v1 -- 0x7f7450005630 con 0x7f74700a5b70 2023-09-30T02:05:16.831 INFO:tasks.ceph.mgr.y.smithi059.stderr:2023-09-30T02:05:16.831+0000 7fc8624c21c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-09-30T02:05:16.911 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.911+0000 7f74657fa700 1 -- 172.21.15.110:0/2948247726 <== mon.2 v2:172.21.15.59:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "76fa5443-1571-4e72-9d22-6ab19e59839a", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f746c015020 con 0x7f74700a5b70 2023-09-30T02:05:16.911 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.911+0000 7f74657fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "76fa5443-1571-4e72-9d22-6ab19e59839a", "id": 4}] 2023-09-30T02:05:16.911 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.911+0000 7f74657fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:05:16.911 INFO:teuthology.orchestra.run.smithi110.stdout:4 2023-09-30T02:05:16.913 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.912+0000 7f7477864700 10 librados: watch_flush enter 2023-09-30T02:05:16.913 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.912+0000 7f7477864700 10 librados: watch_flush exit 2023-09-30T02:05:16.913 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.912+0000 7f7477864700 20 client.4121.objecter shutdown clearing up homeless session... 2023-09-30T02:05:16.913 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.912+0000 7f7477864700 10 client.4121.objecter successfully canceled tick 2023-09-30T02:05:16.913 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.912+0000 7f7477864700 10 monclient: shutdown 2023-09-30T02:05:16.913 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.912+0000 7f7477864700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:05:16.913 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.912+0000 7f7477864700 1 -- 172.21.15.110:0/2948247726 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f74700a5b70 msgr2=0x7f747014e210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:16.913 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.912+0000 7f7477864700 1 --2- 172.21.15.110:0/2948247726 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f74700a5b70 0x7f747014e210 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f74700a76e0 tx=0x7f746c017000 comp rx=0 tx=0).stop 2023-09-30T02:05:16.914 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.912+0000 7f7477864700 1 -- 172.21.15.110:0/2948247726 shutdown_connections 2023-09-30T02:05:16.914 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.912+0000 7f7477864700 1 --2- 172.21.15.110:0/2948247726 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7470142290 0x7f74700a7ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:16.914 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.912+0000 7f7477864700 1 --2- 172.21.15.110:0/2948247726 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f74700a5b70 0x7f747014e210 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:16.914 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.912+0000 7f7477864700 1 --2- 172.21.15.110:0/2948247726 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f74700a5f40 0x7f74700a79b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:16.914 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.912+0000 7f7477864700 1 -- 172.21.15.110:0/2948247726 >> 172.21.15.110:0/2948247726 conn(0x7f74700631f0 msgr2=0x7f74700b3220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:05:16.914 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.912+0000 7f7477864700 1 -- 172.21.15.110:0/2948247726 shutdown_connections 2023-09-30T02:05:16.914 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.912+0000 7f7477864700 1 -- 172.21.15.110:0/2948247726 wait complete. 2023-09-30T02:05:16.914 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:16.912+0000 7f7477864700 1 librados: shutdown 2023-09-30T02:05:16.924 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph --cluster ceph osd new e9436478-4140-4fd6-ba2e-04fa6f6cf367 5 2023-09-30T02:05:16.925 INFO:tasks.ceph.mgr.y.smithi059.stderr:2023-09-30T02:05:16.925+0000 7fc8624c21c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-09-30T02:05:16.943 INFO:tasks.ceph.mgr.x.smithi110.stderr:2023-09-30T02:05:16.943+0000 7fae141da1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-09-30T02:05:17.015 INFO:tasks.ceph.mgr.x.smithi110.stderr:2023-09-30T02:05:17.015+0000 7fae141da1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-09-30T02:05:17.041 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.041+0000 7f34402c3700 10 monclient: get_monmap_and_config 2023-09-30T02:05:17.041 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.041+0000 7f34402c3700 10 monclient: build_initial_monmap 2023-09-30T02:05:17.041 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.041+0000 7f34402c3700 10 monclient: monmap: 2023-09-30T02:05:17.041 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2023-09-30T02:05:17.041 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:17.042 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2023-09-30T02:05:17.042078+0000 2023-09-30T02:05:17.042 INFO:teuthology.orchestra.run.smithi110.stderr:created 2023-09-30T02:05:17.042078+0000 2023-09-30T02:05:17.042 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2023-09-30T02:05:17.042 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 3 2023-09-30T02:05:17.042 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:05:17.042 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:05:17.042 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:05:17.042 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:05:17.042 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:05:17.042 INFO:teuthology.orchestra.run.smithi110.stderr: 2023-09-30T02:05:17.042 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.041+0000 7f34402c3700 1 Processor -- start 2023-09-30T02:05:17.042 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.041+0000 7f34402c3700 1 -- start start 2023-09-30T02:05:17.043 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.041+0000 7f34402c3700 10 monclient: init 2023-09-30T02:05:17.045 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.042+0000 7f34402c3700 10 monclient: _reopen_session rank -1 2023-09-30T02:05:17.045 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.042+0000 7f34402c3700 10 monclient: _add_conns ranks=[4,1,2,3,0] 2023-09-30T02:05:17.045 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.042+0000 7f34402c3700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f34380a5b70 0x7f34380b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:17.045 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.042+0000 7f34402c3700 10 monclient(hunting): picked mon.noname-c con 0x7f34380a5b70 addr v2:172.21.15.110:3300/0 2023-09-30T02:05:17.045 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.042+0000 7f34402c3700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f34380a5f40 addr v1:172.21.15.110:6789/0 2023-09-30T02:05:17.045 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.042+0000 7f34402c3700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3438142290 0x7f343813af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:17.045 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.042+0000 7f34402c3700 10 monclient(hunting): picked mon.noname-b con 0x7f3438142290 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:05:17.045 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.042+0000 7f34402c3700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34380a75a0 con 0x7f34380a5f40 2023-09-30T02:05:17.045 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.042+0000 7f34402c3700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:17.045 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.042+0000 7f34402c3700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34381426d0 con 0x7f34380a5b70 2023-09-30T02:05:17.046 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.042+0000 7f34402c3700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:17.046 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.042+0000 7f34402c3700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34380a63e0 con 0x7f3438142290 2023-09-30T02:05:17.046 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.042+0000 7f34402c3700 10 monclient(hunting): _renew_subs 2023-09-30T02:05:17.046 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.042+0000 7f34402c3700 10 monclient(hunting): authenticate will time out at 1766.502646s 2023-09-30T02:05:17.046 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.043+0000 7f343e05f700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f34380a5b70 0x7f34380b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:17.046 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.043+0000 7f343e860700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3438142290 0x7f343813af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:17.046 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.043+0000 7f343e05f700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f34380a5b70 0x7f34380b1500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:51654/0 (socket says 172.21.15.110:51654) 2023-09-30T02:05:17.046 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.043+0000 7f343e05f700 1 -- 172.21.15.110:0/1380389393 learned_addr learned my addr 172.21.15.110:0/1380389393 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-09-30T02:05:17.046 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.043+0000 7f343e05f700 10 monclient(hunting): get_auth_request con 0x7f34380a5b70 auth_method 0 2023-09-30T02:05:17.046 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.043+0000 7f343e05f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:17.046 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.043+0000 7f343e05f700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:17.046 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.043+0000 7f343e05f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:17.047 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.043+0000 7f343e860700 10 monclient(hunting): get_auth_request con 0x7f3438142290 auth_method 0 2023-09-30T02:05:17.047 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.043+0000 7f343e860700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:17.047 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.043+0000 7f343e860700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:17.047 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.043+0000 7f343e860700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:17.047 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.044+0000 7f343e860700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:05:17.047 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.044+0000 7f343e860700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:05:17.047 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.044+0000 7f343e860700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:05:17.047 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.044+0000 7f343e05f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:05:17.047 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.044+0000 7f343e05f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:05:17.047 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.044+0000 7f343e05f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:05:17.048 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.046+0000 7f343e05f700 10 monclient(hunting): handle_auth_done global_id 4150 payload 274 2023-09-30T02:05:17.048 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.046+0000 7f343e05f700 1 -- 172.21.15.110:0/1380389393 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3438142290 msgr2=0x7f343813af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:05:17.048 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.046+0000 7f343e05f700 1 --2- 172.21.15.110:0/1380389393 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3438142290 0x7f343813af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:17.048 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.046+0000 7f343e05f700 1 -- 172.21.15.110:0/1380389393 >> v1:172.21.15.110:6789/0 conn(0x7f34380a5f40 legacy=0x7f3438141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:05:17.049 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.046+0000 7f343e05f700 10 monclient: _finish_hunting 0 2023-09-30T02:05:17.049 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.046+0000 7f343e05f700 1 monclient: found mon.noname-c 2023-09-30T02:05:17.049 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.046+0000 7f343e05f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:05:17.049 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.046+0000 7f343e05f700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3300/0 2023-09-30T02:05:17.049 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.046+0000 7f343e05f700 1 -- 172.21.15.110:0/1380389393 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34380af3f0 con 0x7f34380a5b70 2023-09-30T02:05:17.049 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.048+0000 7f343e05f700 1 --2- 172.21.15.110:0/1380389393 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f34380a5b70 0x7f34380b1500 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f34340099e0 tx=0x7f3434009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3158dc063c47a34c server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:05:17.053 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.053+0000 7f343d05d700 1 -- 172.21.15.110:0/1380389393 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f343400b7e0 con 0x7f34380a5b70 2023-09-30T02:05:17.053 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.053+0000 7f343d05d700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:05:17.053 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.053+0000 7f343d05d700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:05:17.053 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.053+0000 7f343d05d700 10 monclient: dump: 2023-09-30T02:05:17.053 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2023-09-30T02:05:17.053 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:17.053 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:17.053 INFO:teuthology.orchestra.run.smithi110.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:17.054 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2023-09-30T02:05:17.054 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 3 2023-09-30T02:05:17.054 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:05:17.054 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:05:17.054 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:05:17.054 INFO:teuthology.orchestra.run.smithi110.stderr: 2023-09-30T02:05:17.054 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.053+0000 7f343d05d700 10 monclient: _finish_auth 0 2023-09-30T02:05:17.054 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.053+0000 7f343d05d700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:05:17.054 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.053+0000 7f343d05d700 1 -- 172.21.15.110:0/1380389393 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f343400b940 con 0x7f34380a5b70 2023-09-30T02:05:17.054 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.053+0000 7f34402c3700 5 monclient: authenticate success, global_id 4150 2023-09-30T02:05:17.054 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.053+0000 7f34402c3700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:05:17.054 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.053+0000 7f343d05d700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:05:17.054 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.053+0000 7f34402c3700 10 monclient: get_monmap_and_config success 2023-09-30T02:05:17.055 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.053+0000 7f34402c3700 4 set_mon_vals no callback set 2023-09-30T02:05:17.055 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.053+0000 7f343d05d700 1 -- 172.21.15.110:0/1380389393 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f343400bce0 con 0x7f34380a5b70 2023-09-30T02:05:17.055 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.053+0000 7f343d05d700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:05:17.055 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.053+0000 7f343d05d700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:05:17.055 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.053+0000 7f343d05d700 10 monclient: dump: 2023-09-30T02:05:17.055 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2023-09-30T02:05:17.055 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:17.055 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:17.055 INFO:teuthology.orchestra.run.smithi110.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:17.055 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2023-09-30T02:05:17.055 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 3 2023-09-30T02:05:17.055 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:05:17.055 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:05:17.056 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:05:17.056 INFO:teuthology.orchestra.run.smithi110.stderr: 2023-09-30T02:05:17.059 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.054+0000 7f34402c3700 10 monclient: shutdown 2023-09-30T02:05:17.059 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.054+0000 7f34402c3700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:05:17.059 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.054+0000 7f34402c3700 1 -- 172.21.15.110:0/1380389393 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f34380a5b70 msgr2=0x7f34380b1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:17.059 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.054+0000 7f34402c3700 1 --2- 172.21.15.110:0/1380389393 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f34380a5b70 0x7f34380b1500 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f34340099e0 tx=0x7f3434009250 comp rx=0 tx=0).stop 2023-09-30T02:05:17.059 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.054+0000 7f34402c3700 1 -- 172.21.15.110:0/1380389393 shutdown_connections 2023-09-30T02:05:17.059 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.054+0000 7f34402c3700 1 --2- 172.21.15.110:0/1380389393 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3438142290 0x7f343813af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:17.059 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.054+0000 7f34402c3700 1 --2- 172.21.15.110:0/1380389393 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f34380a5b70 0x7f34380b1500 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:17.060 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.054+0000 7f34402c3700 1 -- 172.21.15.110:0/1380389393 >> 172.21.15.110:0/1380389393 conn(0x7f34380631f0 msgr2=0x7f34380635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:05:17.060 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.054+0000 7f34402c3700 1 -- 172.21.15.110:0/1380389393 shutdown_connections 2023-09-30T02:05:17.060 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.054+0000 7f34402c3700 1 -- 172.21.15.110:0/1380389393 wait complete. 2023-09-30T02:05:17.060 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.055+0000 7f34402c3700 10 monclient: build_initial_monmap 2023-09-30T02:05:17.060 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.055+0000 7f34402c3700 10 monclient: monmap: 2023-09-30T02:05:17.060 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2023-09-30T02:05:17.060 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:05:17.060 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2023-09-30T02:05:17.060 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2023-09-30T02:05:17.060 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2023-09-30T02:05:17.060 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2023-09-30T02:05:17.060 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:05:17.061 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:05:17.061 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:05:17.061 INFO:teuthology.orchestra.run.smithi110.stderr: 2023-09-30T02:05:17.061 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.055+0000 7f34402c3700 1 librados: starting msgr at 2023-09-30T02:05:17.061 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.055+0000 7f34402c3700 1 librados: starting objecter 2023-09-30T02:05:17.061 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.055+0000 7f34402c3700 1 Processor -- start 2023-09-30T02:05:17.061 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.055+0000 7f34402c3700 1 -- start start 2023-09-30T02:05:17.061 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.055+0000 7f34402c3700 1 librados: setting wanted keys 2023-09-30T02:05:17.061 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.055+0000 7f34402c3700 1 librados: calling monclient init 2023-09-30T02:05:17.061 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.055+0000 7f34402c3700 10 monclient: init 2023-09-30T02:05:17.061 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.056+0000 7f34402c3700 10 monclient: _reopen_session rank -1 2023-09-30T02:05:17.061 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.056+0000 7f34402c3700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T02:05:17.062 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.056+0000 7f34402c3700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f34380a5b70 0x7f3438140c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:17.062 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.056+0000 7f34402c3700 10 monclient(hunting): picked mon.noname-c con 0x7f34380a5b70 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:05:17.062 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.056+0000 7f34402c3700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34380a5f40 0x7f343814deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:17.062 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.056+0000 7f34402c3700 10 monclient(hunting): picked mon.noname-a con 0x7f34380a5f40 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:05:17.062 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.056+0000 7f34402c3700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3438142290 0x7f343814e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:17.062 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.056+0000 7f34402c3700 10 monclient(hunting): picked mon.noname-b con 0x7f3438142290 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:05:17.062 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.056+0000 7f34402c3700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:17.062 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.056+0000 7f34402c3700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f343814b610 con 0x7f34380a5f40 2023-09-30T02:05:17.062 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.056+0000 7f34402c3700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:17.062 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.056+0000 7f34402c3700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f343813c160 con 0x7f3438142290 2023-09-30T02:05:17.062 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.056+0000 7f34402c3700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:17.062 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.056+0000 7f34402c3700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f343813dbd0 con 0x7f34380a5b70 2023-09-30T02:05:17.063 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.056+0000 7f34402c3700 10 monclient(hunting): _renew_subs 2023-09-30T02:05:17.063 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.056+0000 7f34402c3700 10 monclient(hunting): authenticate will time out at 1766.516368s 2023-09-30T02:05:17.063 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.056+0000 7f343e860700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3438142290 0x7f343814e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:17.063 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.057+0000 7f343e860700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3438142290 0x7f343814e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:51656/0 (socket says 172.21.15.110:51656) 2023-09-30T02:05:17.063 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.057+0000 7f343e860700 1 -- 172.21.15.110:0/3261468061 learned_addr learned my addr 172.21.15.110:0/3261468061 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-09-30T02:05:17.063 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.057+0000 7f343e860700 10 monclient(hunting): get_auth_request con 0x7f3438142290 auth_method 0 2023-09-30T02:05:17.063 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.057+0000 7f343e860700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:17.063 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.057+0000 7f343e860700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:17.063 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.057+0000 7f343e860700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:17.063 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.057+0000 7f343e05f700 1 --2- 172.21.15.110:0/3261468061 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f34380a5b70 0x7f3438140c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:17.063 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.057+0000 7f343d85e700 1 --2- 172.21.15.110:0/3261468061 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34380a5f40 0x7f343814deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:17.063 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.057+0000 7f343e05f700 10 monclient(hunting): get_auth_request con 0x7f34380a5b70 auth_method 0 2023-09-30T02:05:17.064 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.057+0000 7f343e05f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:17.064 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.057+0000 7f343e05f700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:17.064 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.057+0000 7f343e05f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:17.064 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.057+0000 7f343d85e700 10 monclient(hunting): get_auth_request con 0x7f34380a5f40 auth_method 0 2023-09-30T02:05:17.064 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.057+0000 7f343d85e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:17.064 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.057+0000 7f343d85e700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:17.064 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.057+0000 7f343d85e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:17.064 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.057+0000 7f343e860700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:05:17.064 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.057+0000 7f343e860700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:05:17.064 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.057+0000 7f343e860700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:05:17.064 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.058+0000 7f343e05f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:05:17.064 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.058+0000 7f343e05f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:05:17.065 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.058+0000 7f343e05f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:05:17.065 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.058+0000 7f343d85e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:05:17.065 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.058+0000 7f343d85e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:05:17.065 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.058+0000 7f343d85e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:05:17.066 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.060+0000 7f343e860700 10 monclient(hunting): handle_auth_done global_id 4153 payload 824 2023-09-30T02:05:17.066 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.060+0000 7f343e860700 1 -- 172.21.15.110:0/3261468061 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f34380a5b70 msgr2=0x7f3438140c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:17.066 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.060+0000 7f343e860700 1 --2- 172.21.15.110:0/3261468061 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f34380a5b70 0x7f3438140c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:17.066 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.060+0000 7f343e860700 1 -- 172.21.15.110:0/3261468061 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34380a5f40 msgr2=0x7f343814deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:17.066 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.060+0000 7f343e860700 1 --2- 172.21.15.110:0/3261468061 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34380a5f40 0x7f343814deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:17.066 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.060+0000 7f343e860700 10 monclient: _finish_hunting 0 2023-09-30T02:05:17.066 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.060+0000 7f343e860700 1 monclient: found mon.noname-b 2023-09-30T02:05:17.066 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.060+0000 7f343e860700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:05:17.066 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.060+0000 7f343e860700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:05:17.066 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.060+0000 7f343e860700 1 -- 172.21.15.110:0/3261468061 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34340094f0 con 0x7f3438142290 2023-09-30T02:05:17.066 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.063+0000 7f343e860700 1 --2- 172.21.15.110:0/3261468061 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3438142290 0x7f343814e3f0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3430009df0 tx=0x7f343000b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:05:17.067 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.063+0000 7f342e7fc700 10 client.?.objecter ms_handle_connect 0x7f3438142290 2023-09-30T02:05:17.067 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.063+0000 7f342e7fc700 10 client.?.objecter resend_mon_ops 2023-09-30T02:05:17.068 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.067+0000 7f342e7fc700 1 -- 172.21.15.110:0/3261468061 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f343000caa0 con 0x7f3438142290 2023-09-30T02:05:17.068 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.067+0000 7f342e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:05:17.068 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.067+0000 7f342e7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:05:17.068 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.067+0000 7f342e7fc700 10 monclient: dump: 2023-09-30T02:05:17.068 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2023-09-30T02:05:17.068 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:17.068 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:17.068 INFO:teuthology.orchestra.run.smithi110.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:17.068 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2023-09-30T02:05:17.068 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 3 2023-09-30T02:05:17.069 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:05:17.069 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:05:17.069 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:05:17.069 INFO:teuthology.orchestra.run.smithi110.stderr: 2023-09-30T02:05:17.069 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.067+0000 7f342e7fc700 10 monclient: _finish_auth 0 2023-09-30T02:05:17.069 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.067+0000 7f342e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:05:17.069 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.067+0000 7f342e7fc700 1 -- 172.21.15.110:0/3261468061 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f343000f040 con 0x7f3438142290 2023-09-30T02:05:17.069 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.067+0000 7f34402c3700 5 monclient: authenticate success, global_id 4153 2023-09-30T02:05:17.069 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.067+0000 7f34402c3700 10 monclient: _renew_subs 2023-09-30T02:05:17.069 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.067+0000 7f34402c3700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:05:17.069 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.067+0000 7f34402c3700 1 -- 172.21.15.110:0/3261468061 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f343813c680 con 0x7f3438142290 2023-09-30T02:05:17.069 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.067+0000 7f342e7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:05:17.070 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.067+0000 7f342e7fc700 1 -- 172.21.15.110:0/3261468061 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f343000ce70 con 0x7f3438142290 2023-09-30T02:05:17.070 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.067+0000 7f342e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:05:17.070 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.067+0000 7f342e7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:05:17.070 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.067+0000 7f342e7fc700 10 monclient: dump: 2023-09-30T02:05:17.070 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2023-09-30T02:05:17.070 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:17.070 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:17.070 INFO:teuthology.orchestra.run.smithi110.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:17.070 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2023-09-30T02:05:17.070 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 3 2023-09-30T02:05:17.070 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:05:17.070 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:05:17.070 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:05:17.071 INFO:teuthology.orchestra.run.smithi110.stderr: 2023-09-30T02:05:17.071 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.067+0000 7f343c85c700 4 set_mon_vals no callback set 2023-09-30T02:05:17.071 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.067+0000 7f34402c3700 10 client.4153.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:05:17.071 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.067+0000 7f34402c3700 10 monclient: _renew_subs 2023-09-30T02:05:17.071 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.067+0000 7f34402c3700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:05:17.071 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.068+0000 7f34402c3700 1 -- 172.21.15.110:0/3261468061 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f343813cb70 con 0x7f3438142290 2023-09-30T02:05:17.071 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.068+0000 7f34402c3700 1 librados: init done 2023-09-30T02:05:17.072 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.069+0000 7f34402c3700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:05:17.072 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.069+0000 7f34402c3700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:05:17.072 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.069+0000 7f34402c3700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:05:17.072 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.069+0000 7f34402c3700 1 -- 172.21.15.110:0/3261468061 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f343813c380 con 0x7f3438142290 2023-09-30T02:05:17.175 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.175+0000 7f342e7fc700 1 -- 172.21.15.110:0/3261468061 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f3430014690 con 0x7f3438142290 2023-09-30T02:05:17.176 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.175+0000 7f342e7fc700 1 -- 172.21.15.110:0/3261468061 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1296+0+0 (secure 0 0 0) 0x7f3430026ac0 con 0x7f3438142290 2023-09-30T02:05:17.176 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.175+0000 7f342e7fc700 10 client.4153.objecter ms_dispatch 0x7f3438001550 osd_map(7..7 src has 1..7) v4 2023-09-30T02:05:17.176 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.175+0000 7f342e7fc700 3 client.4153.objecter handle_osd_map got epochs [7,7] > 0 2023-09-30T02:05:17.176 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.175+0000 7f342e7fc700 3 client.4153.objecter handle_osd_map decoding full epoch 7 2023-09-30T02:05:17.176 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.175+0000 7f342e7fc700 20 client.4153.objecter dump_active .. 0 homeless 2023-09-30T02:05:17.176 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.175+0000 7f342e7fc700 1 -- 172.21.15.110:0/3261468061 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f3430018020 con 0x7f3438142290 2023-09-30T02:05:17.176 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.175+0000 7f342e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:05:17.176 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.175+0000 7f342e7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:05:17.256 INFO:tasks.ceph.mgr.y.smithi059.stderr:2023-09-30T02:05:17.255+0000 7fc8624c21c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-09-30T02:05:17.278 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.278+0000 7f34402c3700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "e9436478-4140-4fd6-ba2e-04fa6f6cf367", "id": 5}] 2023-09-30T02:05:17.278 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.278+0000 7f34402c3700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "e9436478-4140-4fd6-ba2e-04fa6f6cf367", "id": 5}] 2023-09-30T02:05:17.278 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.278+0000 7f34402c3700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:05:17.278 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.278+0000 7f34402c3700 1 -- 172.21.15.110:0/3261468061 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd new", "uuid": "e9436478-4140-4fd6-ba2e-04fa6f6cf367", "id": 5} v 0) v1 -- 0x7f3438064a00 con 0x7f3438142290 2023-09-30T02:05:17.337 INFO:tasks.ceph.mgr.y.smithi059.stderr:2023-09-30T02:05:17.337+0000 7fc8624c21c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-09-30T02:05:17.372 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.371+0000 7f342e7fc700 1 -- 172.21.15.110:0/3261468061 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e9436478-4140-4fd6-ba2e-04fa6f6cf367", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f3430026d90 con 0x7f3438142290 2023-09-30T02:05:17.372 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.372+0000 7f342e7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "e9436478-4140-4fd6-ba2e-04fa6f6cf367", "id": 5}] 2023-09-30T02:05:17.372 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.372+0000 7f342e7fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:05:17.372 INFO:teuthology.orchestra.run.smithi110.stdout:5 2023-09-30T02:05:17.373 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.373+0000 7f34402c3700 10 librados: watch_flush enter 2023-09-30T02:05:17.373 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.373+0000 7f34402c3700 10 librados: watch_flush exit 2023-09-30T02:05:17.373 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.373+0000 7f34402c3700 20 client.4153.objecter shutdown clearing up homeless session... 2023-09-30T02:05:17.374 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.373+0000 7f34402c3700 10 client.4153.objecter successfully canceled tick 2023-09-30T02:05:17.374 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.373+0000 7f34402c3700 10 monclient: shutdown 2023-09-30T02:05:17.374 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.373+0000 7f34402c3700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:05:17.374 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.373+0000 7f34402c3700 1 -- 172.21.15.110:0/3261468061 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3438142290 msgr2=0x7f343814e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:17.374 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.373+0000 7f34402c3700 1 --2- 172.21.15.110:0/3261468061 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3438142290 0x7f343814e3f0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3430009df0 tx=0x7f343000b040 comp rx=0 tx=0).stop 2023-09-30T02:05:17.374 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.373+0000 7f34402c3700 1 -- 172.21.15.110:0/3261468061 shutdown_connections 2023-09-30T02:05:17.374 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.373+0000 7f34402c3700 1 --2- 172.21.15.110:0/3261468061 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3438142290 0x7f343814e3f0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:17.374 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.373+0000 7f34402c3700 1 --2- 172.21.15.110:0/3261468061 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f34380a5b70 0x7f3438140c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:17.374 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.373+0000 7f34402c3700 1 --2- 172.21.15.110:0/3261468061 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f34380a5f40 0x7f343814deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:17.374 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.373+0000 7f34402c3700 1 -- 172.21.15.110:0/3261468061 >> 172.21.15.110:0/3261468061 conn(0x7f34380631f0 msgr2=0x7f34380b4a60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:05:17.375 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.373+0000 7f34402c3700 1 -- 172.21.15.110:0/3261468061 shutdown_connections 2023-09-30T02:05:17.375 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.373+0000 7f34402c3700 1 -- 172.21.15.110:0/3261468061 wait complete. 2023-09-30T02:05:17.375 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.373+0000 7f34402c3700 1 librados: shutdown 2023-09-30T02:05:17.385 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph --cluster ceph osd new 76676042-2074-4cf8-9ef9-5c87bf604944 6 2023-09-30T02:05:17.545 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.545+0000 7f9ac64f9700 10 monclient: get_monmap_and_config 2023-09-30T02:05:17.545 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.545+0000 7f9ac64f9700 10 monclient: build_initial_monmap 2023-09-30T02:05:17.545 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.545+0000 7f9ac64f9700 10 monclient: monmap: 2023-09-30T02:05:17.545 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2023-09-30T02:05:17.545 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:17.545 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2023-09-30T02:05:17.546401+0000 2023-09-30T02:05:17.545 INFO:teuthology.orchestra.run.smithi110.stderr:created 2023-09-30T02:05:17.546401+0000 2023-09-30T02:05:17.546 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2023-09-30T02:05:17.546 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 3 2023-09-30T02:05:17.546 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:05:17.546 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:05:17.546 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:05:17.546 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:05:17.546 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:05:17.546 INFO:teuthology.orchestra.run.smithi110.stderr: 2023-09-30T02:05:17.548 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.546+0000 7f9ac64f9700 1 Processor -- start 2023-09-30T02:05:17.548 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.546+0000 7f9ac64f9700 1 -- start start 2023-09-30T02:05:17.548 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.546+0000 7f9ac64f9700 10 monclient: init 2023-09-30T02:05:17.548 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.546+0000 7f9ac64f9700 10 monclient: _reopen_session rank -1 2023-09-30T02:05:17.549 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.546+0000 7f9ac64f9700 10 monclient: _add_conns ranks=[1,3,4,2,0] 2023-09-30T02:05:17.549 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.546+0000 7f9ac64f9700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9ac00a5b70 addr v1:172.21.15.110:6789/0 2023-09-30T02:05:17.549 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.546+0000 7f9ac64f9700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f9ac0142290 0x7f9ac00b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:17.549 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.546+0000 7f9ac64f9700 10 monclient(hunting): picked mon.noname-a con 0x7f9ac0142290 addr v2:172.21.15.59:3300/0 2023-09-30T02:05:17.549 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.546+0000 7f9ac64f9700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f9ac00a5f40 0x7f9ac013af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:17.549 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.546+0000 7f9ac64f9700 10 monclient(hunting): picked mon.noname-c con 0x7f9ac00a5f40 addr v2:172.21.15.110:3300/0 2023-09-30T02:05:17.549 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.546+0000 7f9ac64f9700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ac00a75a0 con 0x7f9ac00a5b70 2023-09-30T02:05:17.549 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.546+0000 7f9ac64f9700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:17.549 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.546+0000 7f9ac64f9700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ac00a6360 con 0x7f9ac0142290 2023-09-30T02:05:17.549 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.546+0000 7f9ac64f9700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:17.549 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.546+0000 7f9ac64f9700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ac00af3f0 con 0x7f9ac00a5f40 2023-09-30T02:05:17.550 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.546+0000 7f9ac64f9700 10 monclient(hunting): _renew_subs 2023-09-30T02:05:17.550 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.546+0000 7f9ac64f9700 10 monclient(hunting): authenticate will time out at 1767.006987s 2023-09-30T02:05:17.550 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.547+0000 7f9ac4a96700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f9ac00a5f40 0x7f9ac013af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:17.550 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.548+0000 7f9abffff700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f9ac00a5b70 0x7f9ac0141b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:35306/0 (socket says 172.21.15.110:35306) 2023-09-30T02:05:17.550 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.548+0000 7f9abffff700 1 -- 172.21.15.110:0/3899357549 learned_addr learned my addr 172.21.15.110:0/3899357549 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-09-30T02:05:17.550 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.548+0000 7f9ac4a96700 10 monclient(hunting): get_auth_request con 0x7f9ac00a5f40 auth_method 0 2023-09-30T02:05:17.550 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.548+0000 7f9ac4a96700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:17.550 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.548+0000 7f9ac4a96700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:17.550 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.548+0000 7f9ac4a96700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:17.550 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.548+0000 7f9ac4a96700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:05:17.550 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.548+0000 7f9ac4a96700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:05:17.550 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.548+0000 7f9ac4a96700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:05:17.551 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.550+0000 7f9abf7fe700 1 --2- 172.21.15.110:0/3899357549 >> v2:172.21.15.59:3300/0 conn(0x7f9ac0142290 0x7f9ac00b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:17.551 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.551+0000 7f9abf7fe700 10 monclient(hunting): get_auth_request con 0x7f9ac0142290 auth_method 0 2023-09-30T02:05:17.552 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.551+0000 7f9abf7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:17.552 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.551+0000 7f9abf7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:17.552 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.551+0000 7f9abf7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:17.552 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.552+0000 7f9ac4a96700 10 monclient(hunting): handle_auth_done global_id 4156 payload 274 2023-09-30T02:05:17.552 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.552+0000 7f9ac4a96700 1 -- 172.21.15.110:0/3899357549 >> v2:172.21.15.59:3300/0 conn(0x7f9ac0142290 msgr2=0x7f9ac00b1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:05:17.552 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.552+0000 7f9ac4a96700 1 --2- 172.21.15.110:0/3899357549 >> v2:172.21.15.59:3300/0 conn(0x7f9ac0142290 0x7f9ac00b1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:17.552 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.552+0000 7f9ac4a96700 1 -- 172.21.15.110:0/3899357549 >> v1:172.21.15.110:6789/0 conn(0x7f9ac00a5b70 legacy=0x7f9ac0141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:17.552 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.552+0000 7f9ac4a96700 10 monclient: _finish_hunting 0 2023-09-30T02:05:17.552 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.552+0000 7f9ac4a96700 1 monclient: found mon.noname-c 2023-09-30T02:05:17.552 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.552+0000 7f9ac4a96700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:05:17.552 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.552+0000 7f9ac4a96700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3300/0 2023-09-30T02:05:17.552 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.552+0000 7f9ac4a96700 1 -- 172.21.15.110:0/3899357549 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ac013b470 con 0x7f9ac00a5f40 2023-09-30T02:05:17.554 INFO:tasks.ceph.mgr.x.smithi110.stderr:2023-09-30T02:05:17.555+0000 7fae141da1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-09-30T02:05:17.556 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.556+0000 7f9ac4a96700 1 --2- 172.21.15.110:0/3899357549 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9ac00a5f40 0x7f9ac013af30 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f9ac00b0640 tx=0x7f9ab8009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e561d73f5f7925c server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:05:17.564 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.563+0000 7f9abeffd700 1 -- 172.21.15.110:0/3899357549 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab800e070 con 0x7f9ac00a5f40 2023-09-30T02:05:17.564 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.563+0000 7f9abeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:05:17.564 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.564+0000 7f9abeffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:05:17.564 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.564+0000 7f9abeffd700 10 monclient: dump: 2023-09-30T02:05:17.564 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2023-09-30T02:05:17.564 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:17.564 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:17.564 INFO:teuthology.orchestra.run.smithi110.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:17.564 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2023-09-30T02:05:17.564 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 3 2023-09-30T02:05:17.564 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:05:17.565 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:05:17.565 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:05:17.565 INFO:teuthology.orchestra.run.smithi110.stderr: 2023-09-30T02:05:17.565 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.564+0000 7f9abeffd700 10 monclient: _finish_auth 0 2023-09-30T02:05:17.565 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.564+0000 7f9abeffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:05:17.565 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.564+0000 7f9abeffd700 1 -- 172.21.15.110:0/3899357549 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ab800bc30 con 0x7f9ac00a5f40 2023-09-30T02:05:17.565 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.564+0000 7f9ac64f9700 5 monclient: authenticate success, global_id 4156 2023-09-30T02:05:17.565 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.564+0000 7f9ac64f9700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:05:17.565 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.564+0000 7f9abeffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:05:17.565 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.564+0000 7f9ac64f9700 10 monclient: get_monmap_and_config success 2023-09-30T02:05:17.565 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.564+0000 7f9abeffd700 1 -- 172.21.15.110:0/3899357549 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab8012430 con 0x7f9ac00a5f40 2023-09-30T02:05:17.565 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.564+0000 7f9ac64f9700 4 set_mon_vals no callback set 2023-09-30T02:05:17.566 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.564+0000 7f9abeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:05:17.566 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.564+0000 7f9abeffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:05:17.566 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.564+0000 7f9abeffd700 10 monclient: dump: 2023-09-30T02:05:17.566 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2023-09-30T02:05:17.566 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:17.566 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:17.566 INFO:teuthology.orchestra.run.smithi110.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:17.566 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2023-09-30T02:05:17.566 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 3 2023-09-30T02:05:17.566 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:05:17.566 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:05:17.566 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:05:17.566 INFO:teuthology.orchestra.run.smithi110.stderr: 2023-09-30T02:05:17.569 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.564+0000 7f9ac64f9700 10 monclient: shutdown 2023-09-30T02:05:17.569 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.564+0000 7f9ac64f9700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:05:17.569 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.564+0000 7f9ac64f9700 1 -- 172.21.15.110:0/3899357549 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9ac00a5f40 msgr2=0x7f9ac013af30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:17.569 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.564+0000 7f9ac64f9700 1 --2- 172.21.15.110:0/3899357549 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9ac00a5f40 0x7f9ac013af30 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f9ac00b0640 tx=0x7f9ab8009780 comp rx=0 tx=0).stop 2023-09-30T02:05:17.569 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.564+0000 7f9ac64f9700 1 -- 172.21.15.110:0/3899357549 shutdown_connections 2023-09-30T02:05:17.569 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.564+0000 7f9ac64f9700 1 --2- 172.21.15.110:0/3899357549 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9ac00a5f40 0x7f9ac013af30 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:17.569 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.564+0000 7f9ac64f9700 1 --2- 172.21.15.110:0/3899357549 >> v2:172.21.15.59:3300/0 conn(0x7f9ac0142290 0x7f9ac00b1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:17.569 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.564+0000 7f9ac64f9700 1 -- 172.21.15.110:0/3899357549 >> 172.21.15.110:0/3899357549 conn(0x7f9ac00631f0 msgr2=0x7f9ac00635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:05:17.569 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.564+0000 7f9ac64f9700 1 -- 172.21.15.110:0/3899357549 shutdown_connections 2023-09-30T02:05:17.570 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.565+0000 7f9ac64f9700 1 -- 172.21.15.110:0/3899357549 wait complete. 2023-09-30T02:05:17.570 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.565+0000 7f9ac64f9700 10 monclient: build_initial_monmap 2023-09-30T02:05:17.570 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.565+0000 7f9ac64f9700 10 monclient: monmap: 2023-09-30T02:05:17.570 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2023-09-30T02:05:17.570 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:05:17.570 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2023-09-30T02:05:17.570 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2023-09-30T02:05:17.570 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2023-09-30T02:05:17.570 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2023-09-30T02:05:17.570 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:05:17.570 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:05:17.570 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:05:17.571 INFO:teuthology.orchestra.run.smithi110.stderr: 2023-09-30T02:05:17.571 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.566+0000 7f9ac64f9700 1 librados: starting msgr at 2023-09-30T02:05:17.571 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.566+0000 7f9ac64f9700 1 librados: starting objecter 2023-09-30T02:05:17.571 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.566+0000 7f9ac64f9700 1 Processor -- start 2023-09-30T02:05:17.571 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.566+0000 7f9ac64f9700 1 -- start start 2023-09-30T02:05:17.571 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.566+0000 7f9ac64f9700 1 librados: setting wanted keys 2023-09-30T02:05:17.571 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.566+0000 7f9ac64f9700 1 librados: calling monclient init 2023-09-30T02:05:17.571 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.566+0000 7f9ac64f9700 10 monclient: init 2023-09-30T02:05:17.571 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.566+0000 7f9ac64f9700 10 monclient: _reopen_session rank -1 2023-09-30T02:05:17.571 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.566+0000 7f9ac64f9700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T02:05:17.571 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.566+0000 7f9ac64f9700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9ac00a5b70 0x7f9ac013df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:17.571 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.566+0000 7f9ac64f9700 10 monclient(hunting): picked mon.noname-b con 0x7f9ac00a5b70 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:05:17.571 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.566+0000 7f9ac64f9700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ac00a5f40 0x7f9ac014df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:17.572 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.566+0000 7f9ac64f9700 10 monclient(hunting): picked mon.noname-a con 0x7f9ac00a5f40 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:05:17.572 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.566+0000 7f9ac64f9700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9ac0142290 0x7f9ac014e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:17.572 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.566+0000 7f9ac64f9700 10 monclient(hunting): picked mon.noname-c con 0x7f9ac0142290 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:05:17.572 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.566+0000 7f9ac64f9700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:17.572 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.566+0000 7f9ac64f9700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac013d1c0 con 0x7f9ac00a5f40 2023-09-30T02:05:17.572 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.566+0000 7f9ac64f9700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:17.572 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.566+0000 7f9ac64f9700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac00a7dd0 con 0x7f9ac00a5b70 2023-09-30T02:05:17.572 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.566+0000 7f9ac64f9700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:17.572 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.566+0000 7f9ac64f9700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac014da00 con 0x7f9ac0142290 2023-09-30T02:05:17.572 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.566+0000 7f9ac64f9700 10 monclient(hunting): _renew_subs 2023-09-30T02:05:17.572 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.566+0000 7f9ac64f9700 10 monclient(hunting): authenticate will time out at 1767.026991s 2023-09-30T02:05:17.584 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.584+0000 7f9ac4a96700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9ac0142290 0x7f9ac014e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:17.584 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.584+0000 7f9ac4a96700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9ac0142290 0x7f9ac014e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.110:56710/0 (socket says 172.21.15.110:56710) 2023-09-30T02:05:17.584 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.584+0000 7f9ac4a96700 1 -- 172.21.15.110:0/2407194930 learned_addr learned my addr 172.21.15.110:0/2407194930 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-09-30T02:05:17.584 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.584+0000 7f9ac4a96700 10 monclient(hunting): get_auth_request con 0x7f9ac0142290 auth_method 0 2023-09-30T02:05:17.584 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.584+0000 7f9ac4a96700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:17.584 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.584+0000 7f9ac4a96700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:17.584 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.584+0000 7f9ac4a96700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:17.585 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.585+0000 7f9ac4a96700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:05:17.585 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.585+0000 7f9ac4a96700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:05:17.585 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.585+0000 7f9ac4a96700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:05:17.585 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.585+0000 7f9abffff700 1 --2- 172.21.15.110:0/2407194930 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9ac00a5b70 0x7f9ac013df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:17.586 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.586+0000 7f9abffff700 10 monclient(hunting): get_auth_request con 0x7f9ac00a5b70 auth_method 0 2023-09-30T02:05:17.586 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.586+0000 7f9abffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:17.586 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.586+0000 7f9abffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:17.586 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.586+0000 7f9abffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:17.586 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.586+0000 7f9abffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:05:17.586 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.586+0000 7f9abffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:05:17.587 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.586+0000 7f9abffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:05:17.588 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.588+0000 7f9ac4a96700 10 monclient(hunting): handle_auth_done global_id 4130 payload 824 2023-09-30T02:05:17.588 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.588+0000 7f9ac4a96700 1 -- 172.21.15.110:0/2407194930 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9ac00a5b70 msgr2=0x7f9ac013df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:17.589 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.588+0000 7f9ac4a96700 1 --2- 172.21.15.110:0/2407194930 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9ac00a5b70 0x7f9ac013df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:17.589 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.588+0000 7f9ac4a96700 1 -- 172.21.15.110:0/2407194930 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ac00a5f40 msgr2=0x7f9ac014df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:17.589 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.588+0000 7f9ac4a96700 1 --2- 172.21.15.110:0/2407194930 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ac00a5f40 0x7f9ac014df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:17.589 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.588+0000 7f9ac4a96700 10 monclient: _finish_hunting 0 2023-09-30T02:05:17.589 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.588+0000 7f9ac4a96700 1 monclient: found mon.noname-c 2023-09-30T02:05:17.589 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.588+0000 7f9ac4a96700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:05:17.589 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.588+0000 7f9ac4a96700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.59:3301/0 2023-09-30T02:05:17.589 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.588+0000 7f9ac4a96700 1 -- 172.21.15.110:0/2407194930 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ab8009a20 con 0x7f9ac0142290 2023-09-30T02:05:17.591 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.591+0000 7f9ac4a96700 1 --2- 172.21.15.110:0/2407194930 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9ac0142290 0x7f9ac014e460 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9ab8013000 tx=0x7f9ab8019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:05:17.591 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.591+0000 7f9aa7fff700 10 client.?.objecter ms_handle_connect 0x7f9ac0142290 2023-09-30T02:05:17.591 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.591+0000 7f9aa7fff700 10 client.?.objecter resend_mon_ops 2023-09-30T02:05:17.597 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.596+0000 7f9aa7fff700 1 -- 172.21.15.110:0/2407194930 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab8027470 con 0x7f9ac0142290 2023-09-30T02:05:17.597 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.596+0000 7f9aa7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:05:17.597 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.596+0000 7f9aa7fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:05:17.598 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.596+0000 7f9aa7fff700 10 monclient: dump: 2023-09-30T02:05:17.598 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2023-09-30T02:05:17.598 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:17.598 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:17.598 INFO:teuthology.orchestra.run.smithi110.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:17.598 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2023-09-30T02:05:17.598 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 3 2023-09-30T02:05:17.598 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:05:17.598 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:05:17.598 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:05:17.599 INFO:teuthology.orchestra.run.smithi110.stderr: 2023-09-30T02:05:17.599 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.597+0000 7f9aa7fff700 10 monclient: _finish_auth 0 2023-09-30T02:05:17.599 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.597+0000 7f9aa7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:05:17.599 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.597+0000 7f9ac64f9700 5 monclient: authenticate success, global_id 4130 2023-09-30T02:05:17.599 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.597+0000 7f9aa7fff700 1 -- 172.21.15.110:0/2407194930 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ab8027630 con 0x7f9ac0142290 2023-09-30T02:05:17.599 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.597+0000 7f9aa7fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:05:17.599 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.597+0000 7f9aa7fff700 1 -- 172.21.15.110:0/2407194930 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab80279f0 con 0x7f9ac0142290 2023-09-30T02:05:17.599 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.597+0000 7f9aa7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:05:17.599 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.597+0000 7f9aa7fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:05:17.599 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.597+0000 7f9aa7fff700 10 monclient: dump: 2023-09-30T02:05:17.599 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2023-09-30T02:05:17.600 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:17.600 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:17.600 INFO:teuthology.orchestra.run.smithi110.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:17.600 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2023-09-30T02:05:17.600 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 3 2023-09-30T02:05:17.600 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:05:17.600 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:05:17.600 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:05:17.600 INFO:teuthology.orchestra.run.smithi110.stderr: 2023-09-30T02:05:17.600 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.597+0000 7f9abe7fc700 4 set_mon_vals no callback set 2023-09-30T02:05:17.600 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.597+0000 7f9ac64f9700 10 monclient: _renew_subs 2023-09-30T02:05:17.600 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.597+0000 7f9ac64f9700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:05:17.600 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.597+0000 7f9ac64f9700 1 -- 172.21.15.110:0/2407194930 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ac013c5a0 con 0x7f9ac0142290 2023-09-30T02:05:17.601 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.597+0000 7f9ac64f9700 10 client.4130.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:05:17.601 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.597+0000 7f9ac64f9700 10 monclient: _renew_subs 2023-09-30T02:05:17.601 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.597+0000 7f9ac64f9700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:05:17.601 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.597+0000 7f9ac64f9700 1 -- 172.21.15.110:0/2407194930 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ac0139830 con 0x7f9ac0142290 2023-09-30T02:05:17.601 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.597+0000 7f9ac64f9700 1 librados: init done 2023-09-30T02:05:17.601 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.597+0000 7f9ac64f9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:05:17.601 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.597+0000 7f9ac64f9700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:05:17.601 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.597+0000 7f9ac64f9700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:05:17.601 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.597+0000 7f9ac64f9700 1 -- 172.21.15.110:0/2407194930 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ac014dc10 con 0x7f9ac0142290 2023-09-30T02:05:17.714 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.714+0000 7f9aa7fff700 1 -- 172.21.15.110:0/2407194930 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f9ab8024070 con 0x7f9ac0142290 2023-09-30T02:05:17.718 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.718+0000 7f9aa7fff700 1 -- 172.21.15.110:0/2407194930 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1411+0+0 (secure 0 0 0) 0x7f9ab801ecf0 con 0x7f9ac0142290 2023-09-30T02:05:17.718 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.718+0000 7f9aa7fff700 10 client.4130.objecter ms_dispatch 0x7f9ac0001550 osd_map(8..8 src has 1..8) v4 2023-09-30T02:05:17.718 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.718+0000 7f9aa7fff700 3 client.4130.objecter handle_osd_map got epochs [8,8] > 0 2023-09-30T02:05:17.719 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.718+0000 7f9aa7fff700 3 client.4130.objecter handle_osd_map decoding full epoch 8 2023-09-30T02:05:17.719 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.718+0000 7f9aa7fff700 20 client.4130.objecter dump_active .. 0 homeless 2023-09-30T02:05:17.719 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.718+0000 7f9aa7fff700 1 -- 172.21.15.110:0/2407194930 <== mon.2 v2:172.21.15.59:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f9ab80476d0 con 0x7f9ac0142290 2023-09-30T02:05:17.719 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.718+0000 7f9aa7fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:05:17.719 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.718+0000 7f9aa7fff700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:05:17.801 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.801+0000 7f9ac64f9700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "76676042-2074-4cf8-9ef9-5c87bf604944", "id": 6}] 2023-09-30T02:05:17.801 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.801+0000 7f9ac64f9700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "76676042-2074-4cf8-9ef9-5c87bf604944", "id": 6}] 2023-09-30T02:05:17.801 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.801+0000 7f9ac64f9700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:05:17.801 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.801+0000 7f9ac64f9700 1 -- 172.21.15.110:0/2407194930 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "osd new", "uuid": "76676042-2074-4cf8-9ef9-5c87bf604944", "id": 6} v 0) v1 -- 0x7f9ac0064a00 con 0x7f9ac0142290 2023-09-30T02:05:17.878 INFO:tasks.ceph.mgr.x.smithi110.stderr:2023-09-30T02:05:17.878+0000 7fae141da1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-09-30T02:05:17.899 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.899+0000 7f9aa7fff700 1 -- 172.21.15.110:0/2407194930 <== mon.2 v2:172.21.15.59:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "76676042-2074-4cf8-9ef9-5c87bf604944", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f9ab8036030 con 0x7f9ac0142290 2023-09-30T02:05:17.899 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.899+0000 7f9aa7fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "76676042-2074-4cf8-9ef9-5c87bf604944", "id": 6}] 2023-09-30T02:05:17.900 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.899+0000 7f9aa7fff700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:05:17.900 INFO:teuthology.orchestra.run.smithi110.stdout:6 2023-09-30T02:05:17.901 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.900+0000 7f9ac64f9700 10 librados: watch_flush enter 2023-09-30T02:05:17.901 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.900+0000 7f9ac64f9700 10 librados: watch_flush exit 2023-09-30T02:05:17.902 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.900+0000 7f9ac64f9700 20 client.4130.objecter shutdown clearing up homeless session... 2023-09-30T02:05:17.902 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.900+0000 7f9ac64f9700 10 client.4130.objecter successfully canceled tick 2023-09-30T02:05:17.902 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.900+0000 7f9ac64f9700 10 monclient: shutdown 2023-09-30T02:05:17.902 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.900+0000 7f9ac64f9700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:05:17.902 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.900+0000 7f9ac64f9700 1 -- 172.21.15.110:0/2407194930 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9ac0142290 msgr2=0x7f9ac014e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:17.902 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.900+0000 7f9ac64f9700 1 --2- 172.21.15.110:0/2407194930 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9ac0142290 0x7f9ac014e460 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9ab8013000 tx=0x7f9ab8019000 comp rx=0 tx=0).stop 2023-09-30T02:05:17.902 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.900+0000 7f9ac64f9700 1 -- 172.21.15.110:0/2407194930 shutdown_connections 2023-09-30T02:05:17.902 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.900+0000 7f9ac64f9700 1 --2- 172.21.15.110:0/2407194930 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9ac00a5b70 0x7f9ac013df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:17.902 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.900+0000 7f9ac64f9700 1 --2- 172.21.15.110:0/2407194930 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9ac0142290 0x7f9ac014e460 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:17.902 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.900+0000 7f9ac64f9700 1 --2- 172.21.15.110:0/2407194930 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ac00a5f40 0x7f9ac014df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:17.902 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.900+0000 7f9ac64f9700 1 -- 172.21.15.110:0/2407194930 >> 172.21.15.110:0/2407194930 conn(0x7f9ac00631f0 msgr2=0x7f9ac0144fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:05:17.902 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.901+0000 7f9ac64f9700 1 -- 172.21.15.110:0/2407194930 shutdown_connections 2023-09-30T02:05:17.903 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.901+0000 7f9ac64f9700 1 -- 172.21.15.110:0/2407194930 wait complete. 2023-09-30T02:05:17.903 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:17.901+0000 7f9ac64f9700 1 librados: shutdown 2023-09-30T02:05:17.912 DEBUG:teuthology.orchestra.run.smithi110:> sudo ceph --cluster ceph osd new 045479f4-f7f7-4b95-bc38-a2c52ac31b13 7 2023-09-30T02:05:17.932 INFO:tasks.ceph.mgr.y.smithi059.stderr:2023-09-30T02:05:17.932+0000 7fc8624c21c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-09-30T02:05:18.024 INFO:tasks.ceph.mgr.x.smithi110.stderr:2023-09-30T02:05:18.024+0000 7fae141da1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-09-30T02:05:18.029 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.029+0000 7f9939f95700 10 monclient: get_monmap_and_config 2023-09-30T02:05:18.029 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.029+0000 7f9939f95700 10 monclient: build_initial_monmap 2023-09-30T02:05:18.029 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.029+0000 7f9939f95700 10 monclient: monmap: 2023-09-30T02:05:18.029 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2023-09-30T02:05:18.029 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:18.030 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2023-09-30T02:05:18.030615+0000 2023-09-30T02:05:18.030 INFO:teuthology.orchestra.run.smithi110.stderr:created 2023-09-30T02:05:18.030615+0000 2023-09-30T02:05:18.030 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2023-09-30T02:05:18.030 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 3 2023-09-30T02:05:18.030 INFO:teuthology.orchestra.run.smithi110.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:05:18.030 INFO:teuthology.orchestra.run.smithi110.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:05:18.030 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:05:18.030 INFO:teuthology.orchestra.run.smithi110.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:05:18.030 INFO:teuthology.orchestra.run.smithi110.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:05:18.030 INFO:teuthology.orchestra.run.smithi110.stderr: 2023-09-30T02:05:18.032 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.030+0000 7f9939f95700 1 Processor -- start 2023-09-30T02:05:18.032 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.030+0000 7f9939f95700 1 -- start start 2023-09-30T02:05:18.032 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.030+0000 7f9939f95700 10 monclient: init 2023-09-30T02:05:18.032 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.031+0000 7f9939f95700 10 monclient: _reopen_session rank -1 2023-09-30T02:05:18.033 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.031+0000 7f9939f95700 10 monclient: _add_conns ranks=[2,1,0,4,3] 2023-09-30T02:05:18.033 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.031+0000 7f9939f95700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f99340a5b70 0x7f99340b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:18.033 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.031+0000 7f9939f95700 10 monclient(hunting): picked mon.noname-b con 0x7f99340a5b70 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:05:18.033 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.031+0000 7f9939f95700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f99340a5f40 addr v1:172.21.15.110:6789/0 2023-09-30T02:05:18.033 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.031+0000 7f9939f95700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9934142290 addr v1:172.21.15.59:6789/0 2023-09-30T02:05:18.033 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.031+0000 7f9939f95700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99340a63b0 con 0x7f9934142290 2023-09-30T02:05:18.033 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.031+0000 7f9939f95700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99340a75a0 con 0x7f99340a5f40 2023-09-30T02:05:18.033 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.031+0000 7f9939f95700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:18.033 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.031+0000 7f9939f95700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99341426d0 con 0x7f99340a5b70 2023-09-30T02:05:18.033 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.031+0000 7f9939f95700 10 monclient(hunting): _renew_subs 2023-09-30T02:05:18.033 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.031+0000 7f9939f95700 10 monclient(hunting): authenticate will time out at 1767.491181s 2023-09-30T02:05:18.034 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.031+0000 7f992affd700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f99340a5f40 0x7f9934141b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.110:35316/0 (socket says 172.21.15.110:35316) 2023-09-30T02:05:18.034 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.031+0000 7f992affd700 1 -- 172.21.15.110:0/4049816854 learned_addr learned my addr 172.21.15.110:0/4049816854 (peer_addr_for_me v1:172.21.15.110:0/0) 2023-09-30T02:05:18.034 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.031+0000 7f99337fe700 1 --2- 172.21.15.110:0/4049816854 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f99340a5b70 0x7f99340b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:18.034 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.032+0000 7f99337fe700 10 monclient(hunting): get_auth_request con 0x7f99340a5b70 auth_method 0 2023-09-30T02:05:18.034 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.032+0000 7f99337fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:18.034 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.032+0000 7f99337fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:18.034 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.032+0000 7f99337fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:18.034 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.032+0000 7f99337fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:05:18.034 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.033+0000 7f99337fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:05:18.034 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.033+0000 7f99337fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:05:18.035 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.034+0000 7f9932ffd700 1 -- 172.21.15.110:0/4049816854 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4249650426 0 0) 0x7f99340a75a0 con 0x7f99340a5f40 2023-09-30T02:05:18.035 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.034+0000 7f9932ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:18.036 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.034+0000 7f9932ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:18.036 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.034+0000 7f9932ffd700 10 monclient(hunting): my global_id is 4162 2023-09-30T02:05:18.036 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.034+0000 7f9932ffd700 1 -- 172.21.15.110:0/4049816854 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9920003570 con 0x7f99340a5f40 2023-09-30T02:05:18.036 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.035+0000 7f9932ffd700 1 -- 172.21.15.110:0/4049816854 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1444361120 0 0) 0x7f99340a63b0 con 0x7f9934142290 2023-09-30T02:05:18.036 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.035+0000 7f9932ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:18.036 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.035+0000 7f9932ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:18.036 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.035+0000 7f9932ffd700 10 monclient(hunting): my global_id is 4140 2023-09-30T02:05:18.036 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.035+0000 7f9932ffd700 1 -- 172.21.15.110:0/4049816854 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99340a75a0 con 0x7f9934142290 2023-09-30T02:05:18.036 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.035+0000 7f99337fe700 10 monclient(hunting): handle_auth_done global_id 4133 payload 274 2023-09-30T02:05:18.036 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.035+0000 7f99337fe700 1 -- 172.21.15.110:0/4049816854 >> v1:172.21.15.110:6789/0 conn(0x7f99340a5f40 legacy=0x7f9934141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:18.036 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.035+0000 7f99337fe700 1 -- 172.21.15.110:0/4049816854 >> v1:172.21.15.59:6789/0 conn(0x7f9934142290 legacy=0x7f993413af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:18.037 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.035+0000 7f99337fe700 10 monclient: _finish_hunting 0 2023-09-30T02:05:18.037 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.035+0000 7f99337fe700 1 monclient: found mon.noname-b 2023-09-30T02:05:18.037 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.035+0000 7f99337fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:05:18.037 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.035+0000 7f99337fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.59:3301/0 2023-09-30T02:05:18.037 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.035+0000 7f99337fe700 1 -- 172.21.15.110:0/4049816854 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99340af3f0 con 0x7f99340a5b70 2023-09-30T02:05:18.037 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.037+0000 7f99337fe700 1 --2- 172.21.15.110:0/4049816854 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f99340a5b70 0x7f99340b1500 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f9924009910 tx=0x7f99240091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=75d86c79d0138a64 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:05:18.039 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.040+0000 7f9932ffd700 1 -- 172.21.15.110:0/4049816854 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9924011630 con 0x7f99340a5b70 2023-09-30T02:05:18.039 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.040+0000 7f9932ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:05:18.040 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.040+0000 7f9932ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:05:18.040 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.040+0000 7f9932ffd700 10 monclient: dump: 2023-09-30T02:05:18.040 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2023-09-30T02:05:18.040 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:18.040 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:18.040 INFO:teuthology.orchestra.run.smithi110.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:18.040 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2023-09-30T02:05:18.040 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 3 2023-09-30T02:05:18.040 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:05:18.040 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:05:18.040 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:05:18.041 INFO:teuthology.orchestra.run.smithi110.stderr: 2023-09-30T02:05:18.041 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.040+0000 7f9932ffd700 10 monclient: _finish_auth 0 2023-09-30T02:05:18.041 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.040+0000 7f9932ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:05:18.041 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.040+0000 7f9939f95700 5 monclient: authenticate success, global_id 4133 2023-09-30T02:05:18.041 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.040+0000 7f9939f95700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:05:18.043 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.041+0000 7f9932ffd700 1 -- 172.21.15.110:0/4049816854 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9924011790 con 0x7f99340a5b70 2023-09-30T02:05:18.043 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.041+0000 7f9932ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:05:18.043 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.041+0000 7f9932ffd700 1 -- 172.21.15.110:0/4049816854 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9924011b10 con 0x7f99340a5b70 2023-09-30T02:05:18.043 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.041+0000 7f9932ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:05:18.043 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.041+0000 7f9932ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:05:18.043 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.041+0000 7f9932ffd700 10 monclient: dump: 2023-09-30T02:05:18.043 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2023-09-30T02:05:18.043 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:18.043 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:18.043 INFO:teuthology.orchestra.run.smithi110.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:18.043 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2023-09-30T02:05:18.044 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 3 2023-09-30T02:05:18.044 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:05:18.044 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:05:18.044 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:05:18.044 INFO:teuthology.orchestra.run.smithi110.stderr: 2023-09-30T02:05:18.044 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.041+0000 7f9939f95700 10 monclient: get_monmap_and_config success 2023-09-30T02:05:18.044 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.041+0000 7f9939f95700 4 set_mon_vals no callback set 2023-09-30T02:05:18.044 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.041+0000 7f9939f95700 10 monclient: shutdown 2023-09-30T02:05:18.044 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.041+0000 7f9939f95700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:05:18.044 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.041+0000 7f9939f95700 1 -- 172.21.15.110:0/4049816854 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f99340a5b70 msgr2=0x7f99340b1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:18.044 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.042+0000 7f9939f95700 1 --2- 172.21.15.110:0/4049816854 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f99340a5b70 0x7f99340b1500 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f9924009910 tx=0x7f99240091a0 comp rx=0 tx=0).stop 2023-09-30T02:05:18.044 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.042+0000 7f9939f95700 1 -- 172.21.15.110:0/4049816854 shutdown_connections 2023-09-30T02:05:18.045 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.042+0000 7f9939f95700 1 --2- 172.21.15.110:0/4049816854 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f99340a5b70 0x7f99340b1500 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:18.045 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.042+0000 7f9939f95700 1 -- 172.21.15.110:0/4049816854 >> 172.21.15.110:0/4049816854 conn(0x7f99340631f0 msgr2=0x7f99340635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:05:18.045 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.042+0000 7f9939f95700 1 -- 172.21.15.110:0/4049816854 shutdown_connections 2023-09-30T02:05:18.045 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.042+0000 7f9939f95700 1 -- 172.21.15.110:0/4049816854 wait complete. 2023-09-30T02:05:18.045 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.042+0000 7f9939f95700 10 monclient: build_initial_monmap 2023-09-30T02:05:18.045 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.042+0000 7f9939f95700 10 monclient: monmap: 2023-09-30T02:05:18.045 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 0 2023-09-30T02:05:18.045 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:05:18.045 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 0.000000 2023-09-30T02:05:18.045 INFO:teuthology.orchestra.run.smithi110.stderr:created 0.000000 2023-09-30T02:05:18.045 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 0 (unknown) 2023-09-30T02:05:18.045 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 1 2023-09-30T02:05:18.046 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:05:18.046 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:05:18.046 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:05:18.046 INFO:teuthology.orchestra.run.smithi110.stderr: 2023-09-30T02:05:18.046 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.042+0000 7f9939f95700 1 librados: starting msgr at 2023-09-30T02:05:18.046 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.042+0000 7f9939f95700 1 librados: starting objecter 2023-09-30T02:05:18.046 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.043+0000 7f9939f95700 1 Processor -- start 2023-09-30T02:05:18.046 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.043+0000 7f9939f95700 1 -- start start 2023-09-30T02:05:18.046 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.043+0000 7f9939f95700 1 librados: setting wanted keys 2023-09-30T02:05:18.046 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.043+0000 7f9939f95700 1 librados: calling monclient init 2023-09-30T02:05:18.046 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.043+0000 7f9939f95700 10 monclient: init 2023-09-30T02:05:18.050 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.043+0000 7f9939f95700 10 monclient: _reopen_session rank -1 2023-09-30T02:05:18.050 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.043+0000 7f9939f95700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T02:05:18.051 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.043+0000 7f9939f95700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f99340a5b70 0x7f993413dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:18.051 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.043+0000 7f9939f95700 10 monclient(hunting): picked mon.noname-b con 0x7f99340a5b70 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:05:18.051 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.043+0000 7f9939f95700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f99340a5f40 0x7f993414d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:18.051 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.043+0000 7f9939f95700 10 monclient(hunting): picked mon.noname-c con 0x7f99340a5f40 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:05:18.051 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.043+0000 7f9939f95700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9934142290 0x7f993414dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:18.051 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.043+0000 7f9939f95700 10 monclient(hunting): picked mon.noname-a con 0x7f9934142290 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:05:18.051 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.043+0000 7f9939f95700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:18.051 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.043+0000 7f9939f95700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9934145010 con 0x7f9934142290 2023-09-30T02:05:18.051 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.043+0000 7f9939f95700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:18.051 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.043+0000 7f9939f95700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f993413eb30 con 0x7f99340a5b70 2023-09-30T02:05:18.052 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.043+0000 7f9939f95700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:18.052 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.043+0000 7f9939f95700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f993413eca0 con 0x7f99340a5f40 2023-09-30T02:05:18.052 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.043+0000 7f9939f95700 10 monclient(hunting): _renew_subs 2023-09-30T02:05:18.052 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.043+0000 7f9939f95700 10 monclient(hunting): authenticate will time out at 1767.503803s 2023-09-30T02:05:18.052 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.044+0000 7f99337fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f99340a5b70 0x7f993413dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:18.052 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.044+0000 7f992affd700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f99340a5f40 0x7f993414d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:18.052 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.044+0000 7f9933fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9934142290 0x7f993414dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:18.052 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.044+0000 7f99337fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f99340a5b70 0x7f993413dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.110:51692/0 (socket says 172.21.15.110:51692) 2023-09-30T02:05:18.052 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.044+0000 7f99337fe700 1 -- 172.21.15.110:0/3208885489 learned_addr learned my addr 172.21.15.110:0/3208885489 (peer_addr_for_me v2:172.21.15.110:0/0) 2023-09-30T02:05:18.052 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.044+0000 7f99337fe700 10 monclient(hunting): get_auth_request con 0x7f99340a5b70 auth_method 0 2023-09-30T02:05:18.052 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.044+0000 7f99337fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:18.053 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.044+0000 7f99337fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:18.053 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.044+0000 7f99337fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:18.053 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.044+0000 7f992affd700 10 monclient(hunting): get_auth_request con 0x7f99340a5f40 auth_method 0 2023-09-30T02:05:18.053 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.044+0000 7f992affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:18.053 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.044+0000 7f992affd700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:18.053 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.044+0000 7f992affd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:18.053 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.045+0000 7f9933fff700 10 monclient(hunting): get_auth_request con 0x7f9934142290 auth_method 0 2023-09-30T02:05:18.053 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.045+0000 7f9933fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:18.053 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.045+0000 7f9933fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:18.053 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.045+0000 7f9933fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:18.053 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.045+0000 7f99337fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:05:18.053 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.045+0000 7f99337fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:05:18.054 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.045+0000 7f99337fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:05:18.054 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.045+0000 7f992affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:05:18.054 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.045+0000 7f992affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:05:18.054 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.045+0000 7f992affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:05:18.054 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.045+0000 7f9933fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:05:18.054 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.045+0000 7f9933fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:05:18.054 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.045+0000 7f9933fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:05:18.054 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.048+0000 7f99337fe700 10 monclient(hunting): handle_auth_done global_id 4165 payload 824 2023-09-30T02:05:18.054 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.048+0000 7f99337fe700 1 -- 172.21.15.110:0/3208885489 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f99340a5f40 msgr2=0x7f993414d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:18.054 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.048+0000 7f99337fe700 1 --2- 172.21.15.110:0/3208885489 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f99340a5f40 0x7f993414d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:18.054 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.048+0000 7f99337fe700 1 -- 172.21.15.110:0/3208885489 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9934142290 msgr2=0x7f993414dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:18.055 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.048+0000 7f99337fe700 1 --2- 172.21.15.110:0/3208885489 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9934142290 0x7f993414dd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:18.055 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.048+0000 7f99337fe700 10 monclient: _finish_hunting 0 2023-09-30T02:05:18.055 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.048+0000 7f99337fe700 1 monclient: found mon.noname-b 2023-09-30T02:05:18.055 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.048+0000 7f99337fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:05:18.055 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.048+0000 7f99337fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:05:18.055 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.048+0000 7f99337fe700 1 -- 172.21.15.110:0/3208885489 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9924009420 con 0x7f99340a5b70 2023-09-30T02:05:18.057 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.051+0000 7f99337fe700 1 --2- 172.21.15.110:0/3208885489 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f99340a5b70 0x7f993413dfe0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f99340b1ff0 tx=0x7f9924017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:05:18.057 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.051+0000 7f992bfff700 10 client.?.objecter ms_handle_connect 0x7f99340a5b70 2023-09-30T02:05:18.057 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.051+0000 7f992bfff700 10 client.?.objecter resend_mon_ops 2023-09-30T02:05:18.057 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.055+0000 7f992bfff700 1 -- 172.21.15.110:0/3208885489 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9924011900 con 0x7f99340a5b70 2023-09-30T02:05:18.057 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.055+0000 7f992bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:05:18.057 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.055+0000 7f992bfff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:05:18.057 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.055+0000 7f992bfff700 10 monclient: dump: 2023-09-30T02:05:18.057 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2023-09-30T02:05:18.057 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:18.058 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:18.058 INFO:teuthology.orchestra.run.smithi110.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:18.058 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2023-09-30T02:05:18.058 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 3 2023-09-30T02:05:18.058 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:05:18.058 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:05:18.058 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:05:18.058 INFO:teuthology.orchestra.run.smithi110.stderr: 2023-09-30T02:05:18.058 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.055+0000 7f992bfff700 10 monclient: _finish_auth 0 2023-09-30T02:05:18.058 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.055+0000 7f992bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:05:18.058 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.055+0000 7f992bfff700 1 -- 172.21.15.110:0/3208885489 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9924011a60 con 0x7f99340a5b70 2023-09-30T02:05:18.058 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.055+0000 7f9939f95700 5 monclient: authenticate success, global_id 4165 2023-09-30T02:05:18.059 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.055+0000 7f9939f95700 10 monclient: _renew_subs 2023-09-30T02:05:18.059 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.055+0000 7f9939f95700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:05:18.059 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.055+0000 7f9939f95700 1 -- 172.21.15.110:0/3208885489 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9934144b80 con 0x7f99340a5b70 2023-09-30T02:05:18.059 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.055+0000 7f992bfff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:05:18.059 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.055+0000 7f992bfff700 1 -- 172.21.15.110:0/3208885489 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9924023620 con 0x7f99340a5b70 2023-09-30T02:05:18.059 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.055+0000 7f992bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:05:18.059 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.055+0000 7f992bfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:05:18.059 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.055+0000 7f992bfff700 10 monclient: dump: 2023-09-30T02:05:18.059 INFO:teuthology.orchestra.run.smithi110.stderr:epoch 1 2023-09-30T02:05:18.059 INFO:teuthology.orchestra.run.smithi110.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:18.059 INFO:teuthology.orchestra.run.smithi110.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:18.059 INFO:teuthology.orchestra.run.smithi110.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:18.060 INFO:teuthology.orchestra.run.smithi110.stderr:min_mon_release 17 (quincy) 2023-09-30T02:05:18.060 INFO:teuthology.orchestra.run.smithi110.stderr:election_strategy: 3 2023-09-30T02:05:18.060 INFO:teuthology.orchestra.run.smithi110.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:05:18.060 INFO:teuthology.orchestra.run.smithi110.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:05:18.060 INFO:teuthology.orchestra.run.smithi110.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:05:18.060 INFO:teuthology.orchestra.run.smithi110.stderr: 2023-09-30T02:05:18.060 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.055+0000 7f99327fc700 4 set_mon_vals no callback set 2023-09-30T02:05:18.060 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.055+0000 7f9939f95700 10 client.4165.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:05:18.060 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.055+0000 7f9939f95700 10 monclient: _renew_subs 2023-09-30T02:05:18.060 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.055+0000 7f9939f95700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:05:18.060 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.055+0000 7f9939f95700 1 -- 172.21.15.110:0/3208885489 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99341456e0 con 0x7f99340a5b70 2023-09-30T02:05:18.060 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.055+0000 7f9939f95700 1 librados: init done 2023-09-30T02:05:18.061 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.056+0000 7f9939f95700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:05:18.061 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.056+0000 7f9939f95700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:05:18.061 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.056+0000 7f9939f95700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:05:18.061 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.056+0000 7f9939f95700 1 -- 172.21.15.110:0/3208885489 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9934064a00 con 0x7f99340a5b70 2023-09-30T02:05:18.061 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.058+0000 7f992bfff700 1 -- 172.21.15.110:0/3208885489 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f9924023a10 con 0x7f99340a5b70 2023-09-30T02:05:18.159 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.159+0000 7f992bfff700 1 -- 172.21.15.110:0/3208885489 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1526+0+0 (secure 0 0 0) 0x7f9924019a20 con 0x7f99340a5b70 2023-09-30T02:05:18.159 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.159+0000 7f992bfff700 10 client.4165.objecter ms_dispatch 0x7f9934001550 osd_map(9..9 src has 1..9) v4 2023-09-30T02:05:18.159 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.159+0000 7f992bfff700 3 client.4165.objecter handle_osd_map got epochs [9,9] > 0 2023-09-30T02:05:18.159 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.159+0000 7f992bfff700 3 client.4165.objecter handle_osd_map decoding full epoch 9 2023-09-30T02:05:18.159 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.159+0000 7f992bfff700 20 client.4165.objecter dump_active .. 0 homeless 2023-09-30T02:05:18.159 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.159+0000 7f992bfff700 1 -- 172.21.15.110:0/3208885489 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f9924028070 con 0x7f99340a5b70 2023-09-30T02:05:18.159 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.159+0000 7f992bfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:05:18.159 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.159+0000 7f992bfff700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:05:18.241 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.241+0000 7f9939f95700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "045479f4-f7f7-4b95-bc38-a2c52ac31b13", "id": 7}] 2023-09-30T02:05:18.241 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.241+0000 7f9939f95700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "045479f4-f7f7-4b95-bc38-a2c52ac31b13", "id": 7}] 2023-09-30T02:05:18.241 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.241+0000 7f9939f95700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:05:18.241 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.241+0000 7f9939f95700 1 -- 172.21.15.110:0/3208885489 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd new", "uuid": "045479f4-f7f7-4b95-bc38-a2c52ac31b13", "id": 7} v 0) v1 -- 0x7f993406a720 con 0x7f99340a5b70 2023-09-30T02:05:18.290 INFO:tasks.ceph.mgr.y.smithi059.stderr:2023-09-30T02:05:18.290+0000 7fc8624c21c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-09-30T02:05:18.341 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.341+0000 7f992bfff700 1 -- 172.21.15.110:0/3208885489 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "045479f4-f7f7-4b95-bc38-a2c52ac31b13", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f9924015020 con 0x7f99340a5b70 2023-09-30T02:05:18.341 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.341+0000 7f992bfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "045479f4-f7f7-4b95-bc38-a2c52ac31b13", "id": 7}] 2023-09-30T02:05:18.341 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.341+0000 7f992bfff700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:05:18.341 INFO:teuthology.orchestra.run.smithi110.stdout:7 2023-09-30T02:05:18.342 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.342+0000 7f9939f95700 10 librados: watch_flush enter 2023-09-30T02:05:18.343 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.342+0000 7f9939f95700 10 librados: watch_flush exit 2023-09-30T02:05:18.343 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.342+0000 7f9939f95700 20 client.4165.objecter shutdown clearing up homeless session... 2023-09-30T02:05:18.343 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.342+0000 7f9939f95700 10 client.4165.objecter successfully canceled tick 2023-09-30T02:05:18.343 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.342+0000 7f9939f95700 10 monclient: shutdown 2023-09-30T02:05:18.343 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.342+0000 7f9939f95700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:05:18.343 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.342+0000 7f9939f95700 1 -- 172.21.15.110:0/3208885489 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f99340a5b70 msgr2=0x7f993413dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:18.343 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.342+0000 7f9939f95700 1 --2- 172.21.15.110:0/3208885489 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f99340a5b70 0x7f993413dfe0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f99340b1ff0 tx=0x7f9924017000 comp rx=0 tx=0).stop 2023-09-30T02:05:18.343 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.342+0000 7f9939f95700 1 -- 172.21.15.110:0/3208885489 shutdown_connections 2023-09-30T02:05:18.343 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.342+0000 7f9939f95700 1 --2- 172.21.15.110:0/3208885489 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f99340a5b70 0x7f993413dfe0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:18.343 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.342+0000 7f9939f95700 1 --2- 172.21.15.110:0/3208885489 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9934142290 0x7f993414dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:18.344 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.342+0000 7f9939f95700 1 --2- 172.21.15.110:0/3208885489 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f99340a5f40 0x7f993414d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:18.344 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.342+0000 7f9939f95700 1 -- 172.21.15.110:0/3208885489 >> 172.21.15.110:0/3208885489 conn(0x7f99340631f0 msgr2=0x7f99340b0be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:05:18.344 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.342+0000 7f9939f95700 1 -- 172.21.15.110:0/3208885489 shutdown_connections 2023-09-30T02:05:18.344 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.342+0000 7f9939f95700 1 -- 172.21.15.110:0/3208885489 wait complete. 2023-09-30T02:05:18.344 INFO:teuthology.orchestra.run.smithi110.stderr:2023-09-30T02:05:18.342+0000 7f9939f95700 1 librados: shutdown 2023-09-30T02:05:18.354 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-09-30T02:05:18.354 INFO:tasks.ceph.osd.0:Restarting daemon 2023-09-30T02:05:18.354 DEBUG:teuthology.orchestra.run.smithi059:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2023-09-30T02:05:18.356 INFO:tasks.ceph.osd.0:Started 2023-09-30T02:05:18.356 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-09-30T02:05:18.357 INFO:tasks.ceph.osd.1:Restarting daemon 2023-09-30T02:05:18.357 DEBUG:teuthology.orchestra.run.smithi059:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2023-09-30T02:05:18.359 INFO:tasks.ceph.osd.1:Started 2023-09-30T02:05:18.359 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-09-30T02:05:18.359 INFO:tasks.ceph.osd.2:Restarting daemon 2023-09-30T02:05:18.359 DEBUG:teuthology.orchestra.run.smithi059:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2023-09-30T02:05:18.361 INFO:tasks.ceph.osd.2:Started 2023-09-30T02:05:18.361 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-09-30T02:05:18.361 INFO:tasks.ceph.osd.3:Restarting daemon 2023-09-30T02:05:18.362 DEBUG:teuthology.orchestra.run.smithi059:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2023-09-30T02:05:18.364 INFO:tasks.ceph.osd.3:Started 2023-09-30T02:05:18.364 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-09-30T02:05:18.364 INFO:tasks.ceph.osd.4:Restarting daemon 2023-09-30T02:05:18.364 DEBUG:teuthology.orchestra.run.smithi110:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2023-09-30T02:05:18.366 INFO:tasks.ceph.osd.4:Started 2023-09-30T02:05:18.366 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-09-30T02:05:18.366 INFO:tasks.ceph.osd.5:Restarting daemon 2023-09-30T02:05:18.367 DEBUG:teuthology.orchestra.run.smithi110:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2023-09-30T02:05:18.368 INFO:tasks.ceph.osd.5:Started 2023-09-30T02:05:18.369 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-09-30T02:05:18.369 INFO:tasks.ceph.osd.6:Restarting daemon 2023-09-30T02:05:18.369 DEBUG:teuthology.orchestra.run.smithi110:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2023-09-30T02:05:18.371 INFO:tasks.ceph.osd.6:Started 2023-09-30T02:05:18.371 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2023-09-30T02:05:18.371 INFO:tasks.ceph.osd.7:Restarting daemon 2023-09-30T02:05:18.371 DEBUG:teuthology.orchestra.run.smithi110:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2023-09-30T02:05:18.373 INFO:tasks.ceph.osd.7:Started 2023-09-30T02:05:18.373 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-30T02:05:18.420 INFO:tasks.ceph.mgr.x.smithi110.stderr:2023-09-30T02:05:18.420+0000 7fae141da1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-09-30T02:05:18.487 INFO:tasks.ceph.mgr.y.smithi059.stderr:2023-09-30T02:05:18.486+0000 7fc8624c21c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-09-30T02:05:18.578 INFO:tasks.ceph.mgr.x.smithi110.stderr:2023-09-30T02:05:18.578+0000 7fae141da1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-09-30T02:05:18.591 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.590+0000 7f4d75591700 10 monclient: get_monmap_and_config 2023-09-30T02:05:18.591 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.590+0000 7f4d75591700 10 monclient: build_initial_monmap 2023-09-30T02:05:18.591 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.590+0000 7f4d75591700 10 monclient: monmap: 2023-09-30T02:05:18.591 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:05:18.591 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:18.591 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:05:18.592304+0000 2023-09-30T02:05:18.592 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:05:18.592304+0000 2023-09-30T02:05:18.592 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:05:18.592 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:05:18.592 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:05:18.592 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:05:18.592 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:05:18.592 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:05:18.592 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:05:18.592 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:05:18.593 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.591+0000 7f4d75591700 1 Processor -- start 2023-09-30T02:05:18.593 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.591+0000 7f4d75591700 1 -- start start 2023-09-30T02:05:18.593 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.591+0000 7f4d75591700 10 monclient: init 2023-09-30T02:05:18.593 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.592+0000 7f4d75591700 10 monclient: _reopen_session rank -1 2023-09-30T02:05:18.593 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.592+0000 7f4d75591700 10 monclient: _add_conns ranks=[2,0,1,4,3] 2023-09-30T02:05:18.594 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.592+0000 7f4d75591700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4d700a5b70 0x7f4d700b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:18.594 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.592+0000 7f4d75591700 10 monclient(hunting): picked mon.noname-b con 0x7f4d700a5b70 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:05:18.594 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.592+0000 7f4d75591700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4d700a5f40 addr v1:172.21.15.59:6789/0 2023-09-30T02:05:18.594 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.592+0000 7f4d75591700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4d70142290 addr v1:172.21.15.110:6789/0 2023-09-30T02:05:18.595 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.592+0000 7f4d75591700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d700a63b0 con 0x7f4d700a5f40 2023-09-30T02:05:18.595 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.592+0000 7f4d75591700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d700a75a0 con 0x7f4d70142290 2023-09-30T02:05:18.595 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.592+0000 7f4d75591700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:18.595 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.592+0000 7f4d75591700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d701426d0 con 0x7f4d700a5b70 2023-09-30T02:05:18.595 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.592+0000 7f4d75591700 10 monclient(hunting): _renew_subs 2023-09-30T02:05:18.595 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.593+0000 7f4d75591700 10 monclient(hunting): authenticate will time out at 1763.896009s 2023-09-30T02:05:18.595 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.594+0000 7f4d6f7fe700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f4d70142290 0x7f4d7013af30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.59:44334/0 (socket says 172.21.15.59:44334) 2023-09-30T02:05:18.595 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.594+0000 7f4d6f7fe700 1 -- 172.21.15.59:0/940617978 learned_addr learned my addr 172.21.15.59:0/940617978 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:05:18.595 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.594+0000 7f4d6effd700 1 --2- 172.21.15.59:0/940617978 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4d700a5b70 0x7f4d700b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:18.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.594+0000 7f4d6effd700 10 monclient(hunting): get_auth_request con 0x7f4d700a5b70 auth_method 0 2023-09-30T02:05:18.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.594+0000 7f4d6effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:18.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.594+0000 7f4d6effd700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:18.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.594+0000 7f4d6effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:18.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.595+0000 7f4d6effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:05:18.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.595+0000 7f4d6effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:05:18.606 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.595+0000 7f4d6effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:05:18.606 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.597+0000 7f4d6dffb700 1 -- 172.21.15.59:0/940617978 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 419802 0 0) 0x7f4d700a75a0 con 0x7f4d70142290 2023-09-30T02:05:18.606 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.597+0000 7f4d6dffb700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:18.606 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.597+0000 7f4d6dffb700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:18.606 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.597+0000 7f4d6dffb700 10 monclient(hunting): my global_id is 4168 2023-09-30T02:05:18.606 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.597+0000 7f4d6dffb700 1 -- 172.21.15.59:0/940617978 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d58003570 con 0x7f4d70142290 2023-09-30T02:05:18.606 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.598+0000 7f4d6effd700 10 monclient(hunting): handle_auth_done global_id 4139 payload 274 2023-09-30T02:05:18.606 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.598+0000 7f4d6effd700 1 -- 172.21.15.59:0/940617978 >> v1:172.21.15.110:6789/0 conn(0x7f4d70142290 legacy=0x7f4d7013af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:18.606 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.598+0000 7f4d6effd700 1 -- 172.21.15.59:0/940617978 >> v1:172.21.15.59:6789/0 conn(0x7f4d700a5f40 legacy=0x7f4d70141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:18.607 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.598+0000 7f4d6effd700 10 monclient: _finish_hunting 0 2023-09-30T02:05:18.607 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.598+0000 7f4d6effd700 1 monclient: found mon.noname-b 2023-09-30T02:05:18.607 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.598+0000 7f4d6effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:05:18.607 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.598+0000 7f4d6effd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.59:3301/0 2023-09-30T02:05:18.607 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.598+0000 7f4d6effd700 1 -- 172.21.15.59:0/940617978 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d700af3f0 con 0x7f4d700a5b70 2023-09-30T02:05:18.607 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.600+0000 7f4d6effd700 1 --2- 172.21.15.59:0/940617978 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4d700a5b70 0x7f4d700b1500 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4d60009910 tx=0x7f4d600091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a7f17b609df2c4da server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:05:18.609 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.608+0000 7f4d6dffb700 1 -- 172.21.15.59:0/940617978 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d6000b680 con 0x7f4d700a5b70 2023-09-30T02:05:18.609 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.608+0000 7f4d6dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:05:18.609 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.609+0000 7f4d6dffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:05:18.609 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.609+0000 7f4d6dffb700 10 monclient: dump: 2023-09-30T02:05:18.609 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:05:18.609 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:18.610 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:18.610 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:18.610 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:05:18.610 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:05:18.610 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:05:18.610 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:05:18.610 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:05:18.610 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:05:18.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.609+0000 7f4d6dffb700 10 monclient: _finish_auth 0 2023-09-30T02:05:18.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.609+0000 7f4d6dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:05:18.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.609+0000 7f4d6dffb700 1 -- 172.21.15.59:0/940617978 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d6000b7e0 con 0x7f4d700a5b70 2023-09-30T02:05:18.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.609+0000 7f4d6dffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:05:18.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.609+0000 7f4d6dffb700 1 -- 172.21.15.59:0/940617978 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d6000bba0 con 0x7f4d700a5b70 2023-09-30T02:05:18.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.609+0000 7f4d6dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:05:18.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.609+0000 7f4d6dffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:05:18.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.609+0000 7f4d6dffb700 10 monclient: dump: 2023-09-30T02:05:18.611 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:05:18.611 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:18.612 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:18.612 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:18.612 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:05:18.612 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:05:18.612 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:05:18.612 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:05:18.612 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:05:18.612 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:05:18.612 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.609+0000 7f4d75591700 5 monclient: authenticate success, global_id 4139 2023-09-30T02:05:18.612 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.609+0000 7f4d75591700 10 monclient: get_monmap_and_config success 2023-09-30T02:05:18.612 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.609+0000 7f4d75591700 4 set_mon_vals no callback set 2023-09-30T02:05:18.613 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.610+0000 7f4d75591700 10 monclient: shutdown 2023-09-30T02:05:18.614 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.610+0000 7f4d75591700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:05:18.614 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.610+0000 7f4d75591700 1 -- 172.21.15.59:0/940617978 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4d700a5b70 msgr2=0x7f4d700b1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:18.614 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.610+0000 7f4d75591700 1 --2- 172.21.15.59:0/940617978 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4d700a5b70 0x7f4d700b1500 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4d60009910 tx=0x7f4d600091a0 comp rx=0 tx=0).stop 2023-09-30T02:05:18.614 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.610+0000 7f4d75591700 1 -- 172.21.15.59:0/940617978 shutdown_connections 2023-09-30T02:05:18.614 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.610+0000 7f4d75591700 1 --2- 172.21.15.59:0/940617978 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4d700a5b70 0x7f4d700b1500 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:18.614 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.610+0000 7f4d75591700 1 -- 172.21.15.59:0/940617978 >> 172.21.15.59:0/940617978 conn(0x7f4d700631f0 msgr2=0x7f4d700635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:05:18.614 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.610+0000 7f4d75591700 1 -- 172.21.15.59:0/940617978 shutdown_connections 2023-09-30T02:05:18.614 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.610+0000 7f4d75591700 1 -- 172.21.15.59:0/940617978 wait complete. 2023-09-30T02:05:18.614 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.611+0000 7f4d75591700 10 monclient: build_initial_monmap 2023-09-30T02:05:18.614 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.611+0000 7f4d75591700 10 monclient: monmap: 2023-09-30T02:05:18.615 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:05:18.615 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:05:18.615 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:05:18.615 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:05:18.615 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:05:18.615 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:05:18.615 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:05:18.615 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:05:18.615 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:05:18.615 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:05:18.615 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.611+0000 7f4d75591700 1 librados: starting msgr at 2023-09-30T02:05:18.616 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.611+0000 7f4d75591700 1 librados: starting objecter 2023-09-30T02:05:18.616 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.612+0000 7f4d75591700 1 Processor -- start 2023-09-30T02:05:18.616 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.612+0000 7f4d75591700 1 -- start start 2023-09-30T02:05:18.616 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.612+0000 7f4d75591700 1 librados: setting wanted keys 2023-09-30T02:05:18.616 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.612+0000 7f4d75591700 1 librados: calling monclient init 2023-09-30T02:05:18.616 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.612+0000 7f4d75591700 10 monclient: init 2023-09-30T02:05:18.616 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.613+0000 7f4d75591700 10 monclient: _reopen_session rank -1 2023-09-30T02:05:18.616 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.613+0000 7f4d75591700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T02:05:18.616 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.613+0000 7f4d75591700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4d700a5b70 0x7f4d7006ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:18.616 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.613+0000 7f4d75591700 10 monclient(hunting): picked mon.noname-c con 0x7f4d700a5b70 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:05:18.616 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.613+0000 7f4d75591700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4d700a5f40 0x7f4d7013ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:18.617 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.613+0000 7f4d75591700 10 monclient(hunting): picked mon.noname-b con 0x7f4d700a5f40 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:05:18.617 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.613+0000 7f4d75591700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d70142290 0x7f4d7015aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:18.617 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.613+0000 7f4d75591700 10 monclient(hunting): picked mon.noname-a con 0x7f4d70142290 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:05:18.617 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.613+0000 7f4d75591700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:18.617 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.613+0000 7f4d75591700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d7015a570 con 0x7f4d70142290 2023-09-30T02:05:18.617 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.613+0000 7f4d75591700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:18.617 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.613+0000 7f4d75591700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d7006c800 con 0x7f4d700a5f40 2023-09-30T02:05:18.617 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.613+0000 7f4d75591700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:18.618 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.613+0000 7f4d75591700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d7013de60 con 0x7f4d700a5b70 2023-09-30T02:05:18.618 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.613+0000 7f4d75591700 10 monclient(hunting): _renew_subs 2023-09-30T02:05:18.618 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.613+0000 7f4d75591700 10 monclient(hunting): authenticate will time out at 1763.916445s 2023-09-30T02:05:18.619 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.614+0000 7f4d6f7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d70142290 0x7f4d7015aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:18.620 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.614+0000 7f4d6e7fc700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4d700a5f40 0x7f4d7013ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:18.620 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.614+0000 7f4d6f7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d70142290 0x7f4d7015aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:41570/0 (socket says 172.21.15.59:41570) 2023-09-30T02:05:18.620 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.615+0000 7f4d6f7fe700 1 -- 172.21.15.59:0/1135337625 learned_addr learned my addr 172.21.15.59:0/1135337625 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:05:18.620 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.615+0000 7f4d6f7fe700 10 monclient(hunting): get_auth_request con 0x7f4d70142290 auth_method 0 2023-09-30T02:05:18.620 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.615+0000 7f4d6f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:18.620 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.615+0000 7f4d6f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:18.620 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.615+0000 7f4d6f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:18.620 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.615+0000 7f4d6e7fc700 10 monclient(hunting): get_auth_request con 0x7f4d700a5f40 auth_method 0 2023-09-30T02:05:18.620 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.615+0000 7f4d6e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:18.620 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.615+0000 7f4d6e7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:18.621 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.615+0000 7f4d6e7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:18.621 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.615+0000 7f4d6effd700 1 --2- 172.21.15.59:0/1135337625 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4d700a5b70 0x7f4d7006ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:18.621 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.615+0000 7f4d6effd700 10 monclient(hunting): get_auth_request con 0x7f4d700a5b70 auth_method 0 2023-09-30T02:05:18.621 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.615+0000 7f4d6effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:18.621 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.615+0000 7f4d6effd700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:18.621 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.615+0000 7f4d6effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:18.621 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.615+0000 7f4d6f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:05:18.621 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.615+0000 7f4d6f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:05:18.621 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.615+0000 7f4d6f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:05:18.621 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.616+0000 7f4d6e7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:05:18.621 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.616+0000 7f4d6e7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:05:18.622 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.616+0000 7f4d6e7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:05:18.622 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.616+0000 7f4d6effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:05:18.622 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.616+0000 7f4d6effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:05:18.622 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.616+0000 7f4d6effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:05:18.622 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.621+0000 7f4d6e7fc700 10 monclient(hunting): handle_auth_done global_id 4171 payload 824 2023-09-30T02:05:18.622 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.621+0000 7f4d6e7fc700 1 -- 172.21.15.59:0/1135337625 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4d700a5b70 msgr2=0x7f4d7006ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:18.622 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.621+0000 7f4d6e7fc700 1 --2- 172.21.15.59:0/1135337625 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4d700a5b70 0x7f4d7006ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:18.623 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.621+0000 7f4d6e7fc700 1 -- 172.21.15.59:0/1135337625 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d70142290 msgr2=0x7f4d7015aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:18.623 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.621+0000 7f4d6e7fc700 1 --2- 172.21.15.59:0/1135337625 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d70142290 0x7f4d7015aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:18.623 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.621+0000 7f4d6e7fc700 10 monclient: _finish_hunting 0 2023-09-30T02:05:18.623 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.621+0000 7f4d6e7fc700 1 monclient: found mon.noname-b 2023-09-30T02:05:18.623 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.621+0000 7f4d6e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:05:18.623 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.621+0000 7f4d6e7fc700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:05:18.623 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.621+0000 7f4d6e7fc700 1 -- 172.21.15.59:0/1135337625 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d60009420 con 0x7f4d700a5f40 2023-09-30T02:05:18.625 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.625+0000 7f4d6e7fc700 1 --2- 172.21.15.59:0/1135337625 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4d700a5f40 0x7f4d7013ea60 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4d6400c590 tx=0x7f4d64007ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:05:18.625 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.625+0000 7f4d56ffd700 10 client.?.objecter ms_handle_connect 0x7f4d700a5f40 2023-09-30T02:05:18.625 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.625+0000 7f4d56ffd700 10 client.?.objecter resend_mon_ops 2023-09-30T02:05:18.631 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.631+0000 7f4d56ffd700 1 -- 172.21.15.59:0/1135337625 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d64011590 con 0x7f4d700a5f40 2023-09-30T02:05:18.631 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.631+0000 7f4d56ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:05:18.632 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.631+0000 7f4d56ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:05:18.632 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.631+0000 7f4d56ffd700 10 monclient: dump: 2023-09-30T02:05:18.632 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:05:18.632 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:18.632 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:18.632 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:18.632 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:05:18.632 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:05:18.632 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:05:18.632 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:05:18.633 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:05:18.633 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:05:18.633 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.631+0000 7f4d56ffd700 10 monclient: _finish_auth 0 2023-09-30T02:05:18.633 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.631+0000 7f4d56ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:05:18.633 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.631+0000 7f4d56ffd700 1 -- 172.21.15.59:0/1135337625 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d64011710 con 0x7f4d700a5f40 2023-09-30T02:05:18.633 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.631+0000 7f4d75591700 5 monclient: authenticate success, global_id 4171 2023-09-30T02:05:18.633 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.631+0000 7f4d75591700 10 monclient: _renew_subs 2023-09-30T02:05:18.633 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.631+0000 7f4d75591700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:05:18.634 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.631+0000 7f4d75591700 1 -- 172.21.15.59:0/1135337625 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d7013efa0 con 0x7f4d700a5f40 2023-09-30T02:05:18.634 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.631+0000 7f4d75591700 10 client.4171.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:05:18.634 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.631+0000 7f4d75591700 10 monclient: _renew_subs 2023-09-30T02:05:18.634 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.631+0000 7f4d75591700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:05:18.634 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.631+0000 7f4d75591700 1 -- 172.21.15.59:0/1135337625 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d7015af40 con 0x7f4d700a5f40 2023-09-30T02:05:18.634 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.631+0000 7f4d75591700 1 librados: init done 2023-09-30T02:05:18.635 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.631+0000 7f4d56ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:05:18.635 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.631+0000 7f4d56ffd700 1 -- 172.21.15.59:0/1135337625 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d64011ad0 con 0x7f4d700a5f40 2023-09-30T02:05:18.635 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.631+0000 7f4d56ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:05:18.635 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.631+0000 7f4d56ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:05:18.635 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.631+0000 7f4d56ffd700 10 monclient: dump: 2023-09-30T02:05:18.635 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:05:18.635 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:18.635 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:18.635 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:18.635 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:05:18.635 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:05:18.636 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:05:18.636 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:05:18.636 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:05:18.636 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:05:18.636 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.631+0000 7f4d6d7fa700 4 set_mon_vals no callback set 2023-09-30T02:05:18.636 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.632+0000 7f4d75591700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:05:18.636 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.632+0000 7f4d75591700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:05:18.636 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.632+0000 7f4d75591700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:05:18.636 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.632+0000 7f4d75591700 1 -- 172.21.15.59:0/1135337625 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d500050b0 con 0x7f4d700a5f40 2023-09-30T02:05:18.735 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.734+0000 7f4d56ffd700 1 -- 172.21.15.59:0/1135337625 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f4d6401c6c0 con 0x7f4d700a5f40 2023-09-30T02:05:18.735 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.734+0000 7f4d56ffd700 1 -- 172.21.15.59:0/1135337625 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1641+0+0 (secure 0 0 0) 0x7f4d64011e40 con 0x7f4d700a5f40 2023-09-30T02:05:18.735 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.734+0000 7f4d56ffd700 10 client.4171.objecter ms_dispatch 0x7f4d70001550 osd_map(10..10 src has 1..10) v4 2023-09-30T02:05:18.735 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.734+0000 7f4d56ffd700 3 client.4171.objecter handle_osd_map got epochs [10,10] > 0 2023-09-30T02:05:18.735 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.734+0000 7f4d56ffd700 3 client.4171.objecter handle_osd_map decoding full epoch 10 2023-09-30T02:05:18.735 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.734+0000 7f4d56ffd700 20 client.4171.objecter dump_active .. 0 homeless 2023-09-30T02:05:18.801 INFO:tasks.ceph.mgr.x.smithi110.stderr:2023-09-30T02:05:18.802+0000 7fae141da1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-09-30T02:05:18.817 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.817+0000 7f4d56ffd700 1 -- 172.21.15.59:0/1135337625 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f4d64017070 con 0x7f4d700a5f40 2023-09-30T02:05:18.817 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.817+0000 7f4d56ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:05:18.817 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.817+0000 7f4d56ffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:05:18.964 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.963+0000 7f4d75591700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-09-30T02:05:18.964 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.963+0000 7f4d75591700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T02:05:18.964 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.963+0000 7f4d75591700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:05:18.964 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:18.963+0000 7f4d75591700 1 -- 172.21.15.59:0/1135337625 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4d50005630 con 0x7f4d700a5f40 2023-09-30T02:05:18.975 INFO:tasks.ceph.mgr.x.smithi110.stderr:2023-09-30T02:05:18.975+0000 7fae141da1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-09-30T02:05:19.116 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:19.115+0000 7f4d56ffd700 1 -- 172.21.15.59:0/1135337625 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7013 (secure 0 0 0) 0x7f4d6401c970 con 0x7f4d700a5f40 2023-09-30T02:05:19.116 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:19.115+0000 7f4d56ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T02:05:19.116 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:19.115+0000 7f4d56ffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:05:19.119 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:05:19.119 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":10,"fsid":"623052cb-4f82-4b2b-b72b-3ed5be614613","created":"2023-09-30T02:05:11.679291+0000","modified":"2023-09-30T02:05:18.270595+0000","last_up_change":"0.000000","last_in_change":"2023-09-30T02:05:18.270595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"efdab44b-be57-4b81-8567-ff2deec1d3e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff9786d-8c39-4044-91d2-a31f33e3b16e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b276ed4b-317b-4ddf-96ab-7ad71b8e3ff7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"48bfaf43-5846-4385-b9c6-6bc424cd5acd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"76fa5443-1571-4e72-9d22-6ab19e59839a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e9436478-4140-4fd6-ba2e-04fa6f6cf367","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"76676042-2074-4cf8-9ef9-5c87bf604944","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"045479f4-f7f7-4b95-bc38-a2c52ac31b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T02:05:19.121 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:19.121+0000 7f4d75591700 10 librados: watch_flush enter 2023-09-30T02:05:19.121 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:19.121+0000 7f4d75591700 10 librados: watch_flush exit 2023-09-30T02:05:19.121 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:19.121+0000 7f4d75591700 20 client.4171.objecter shutdown clearing up homeless session... 2023-09-30T02:05:19.121 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:19.121+0000 7f4d75591700 10 client.4171.objecter successfully canceled tick 2023-09-30T02:05:19.122 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:19.122+0000 7f4d75591700 10 monclient: shutdown 2023-09-30T02:05:19.122 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:19.122+0000 7f4d75591700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:05:19.122 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:19.122+0000 7f4d75591700 1 -- 172.21.15.59:0/1135337625 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4d700a5f40 msgr2=0x7f4d7013ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:19.122 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:19.122+0000 7f4d75591700 1 --2- 172.21.15.59:0/1135337625 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4d700a5f40 0x7f4d7013ea60 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4d6400c590 tx=0x7f4d64007ec0 comp rx=0 tx=0).stop 2023-09-30T02:05:19.123 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:19.122+0000 7f4d75591700 1 -- 172.21.15.59:0/1135337625 shutdown_connections 2023-09-30T02:05:19.123 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:19.122+0000 7f4d75591700 1 --2- 172.21.15.59:0/1135337625 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4d70142290 0x7f4d7015aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:19.123 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:19.122+0000 7f4d75591700 1 --2- 172.21.15.59:0/1135337625 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4d700a5b70 0x7f4d7006ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:19.123 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:19.122+0000 7f4d75591700 1 --2- 172.21.15.59:0/1135337625 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4d700a5f40 0x7f4d7013ea60 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:19.123 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:19.122+0000 7f4d75591700 1 -- 172.21.15.59:0/1135337625 >> 172.21.15.59:0/1135337625 conn(0x7f4d700631f0 msgr2=0x7f4d700635c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:05:19.123 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:19.122+0000 7f4d75591700 1 -- 172.21.15.59:0/1135337625 shutdown_connections 2023-09-30T02:05:19.123 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:19.122+0000 7f4d75591700 1 -- 172.21.15.59:0/1135337625 wait complete. 2023-09-30T02:05:19.123 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:05:19.122+0000 7f4d75591700 1 librados: shutdown 2023-09-30T02:05:19.141 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-09-30T02:05:19.142 INFO:tasks.ceph:Waiting for OSDs to come up 2023-09-30T02:05:19.178 INFO:tasks.ceph.mgr.y.smithi059.stderr:2023-09-30T02:05:19.178+0000 7fc8624c21c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-09-30T02:05:19.460 INFO:tasks.ceph.mgr.y.smithi059.stderr:2023-09-30T02:05:19.459+0000 7fc8624c21c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-09-30T02:05:19.758 INFO:tasks.ceph.mgr.y.smithi059.stderr:2023-09-30T02:05:19.757+0000 7fc8624c21c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-09-30T02:05:19.947 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-09-30T02:05:19.961 INFO:tasks.ceph.mgr.y.smithi059.stderr:2023-09-30T02:05:19.956+0000 7fc8624c21c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-09-30T02:05:20.091 INFO:tasks.ceph.mgr.x.smithi110.stderr:2023-09-30T02:05:20.091+0000 7fada3684700 -1 log_channel(cluster) log [ERR] : Unhandled exception from module 'pg_autoscaler' while running on mgr.x: 'PgAutoscaler' object has no attribute 'noautoscale' 2023-09-30T02:05:20.091 INFO:tasks.ceph.mgr.x.smithi110.stderr:2023-09-30T02:05:20.091+0000 7fada3684700 -1 pg_autoscaler.serve: 2023-09-30T02:05:20.092 INFO:tasks.ceph.mgr.x.smithi110.stderr:2023-09-30T02:05:20.091+0000 7fada3684700 -1 Traceback (most recent call last): 2023-09-30T02:05:20.093 INFO:tasks.ceph.mgr.x.smithi110.stderr: File "/usr/share/ceph/mgr/pg_autoscaler/module.py", line 325, in serve 2023-09-30T02:05:20.093 INFO:tasks.ceph.mgr.x.smithi110.stderr: self._maybe_adjust() 2023-09-30T02:05:20.093 INFO:tasks.ceph.mgr.x.smithi110.stderr: File "/usr/share/ceph/mgr/pg_autoscaler/module.py", line 682, in _maybe_adjust 2023-09-30T02:05:20.093 INFO:tasks.ceph.mgr.x.smithi110.stderr: if self.noautoscale: 2023-09-30T02:05:20.093 INFO:tasks.ceph.mgr.x.smithi110.stderr:AttributeError: 'PgAutoscaler' object has no attribute 'noautoscale' 2023-09-30T02:05:20.093 INFO:tasks.ceph.mgr.x.smithi110.stderr: 2023-09-30T02:05:20.159 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.159+0000 7ffaaf4cd700 10 monclient: get_monmap_and_config 2023-09-30T02:05:20.160 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.159+0000 7ffaaf4cd700 10 monclient: build_initial_monmap 2023-09-30T02:05:20.162 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.159+0000 7ffaaf4cd700 10 monclient: monmap: 2023-09-30T02:05:20.162 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2023-09-30T02:05:20.162 INFO:teuthology.misc.health.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:20.162 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2023-09-30T02:05:20.160827+0000 2023-09-30T02:05:20.162 INFO:teuthology.misc.health.smithi059.stderr:created 2023-09-30T02:05:20.160827+0000 2023-09-30T02:05:20.162 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:05:20.162 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 3 2023-09-30T02:05:20.162 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:05:20.162 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:05:20.162 INFO:teuthology.misc.health.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:05:20.163 INFO:teuthology.misc.health.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:05:20.163 INFO:teuthology.misc.health.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:05:20.163 INFO:teuthology.misc.health.smithi059.stderr: 2023-09-30T02:05:20.163 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.160+0000 7ffaaf4cd700 1 Processor -- start 2023-09-30T02:05:20.164 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.160+0000 7ffaaf4cd700 1 -- start start 2023-09-30T02:05:20.164 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.160+0000 7ffaaf4cd700 10 monclient: init 2023-09-30T02:05:20.164 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.161+0000 7ffaaf4cd700 10 monclient: _reopen_session rank -1 2023-09-30T02:05:20.164 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.162+0000 7ffaaf4cd700 10 monclient: _add_conns ranks=[2,3,1,4,0] 2023-09-30T02:05:20.164 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.162+0000 7ffaaf4cd700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ffaa80a5cb0 0x7ffaa80a6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:20.164 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.162+0000 7ffaaf4cd700 10 monclient(hunting): picked mon.noname-b con 0x7ffaa80a5cb0 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:05:20.164 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.162+0000 7ffaaf4cd700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7ffaa80a6080 0x7ffaa813f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:20.164 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.162+0000 7ffaaf4cd700 10 monclient(hunting): picked mon.noname-a con 0x7ffaa80a6080 addr v2:172.21.15.59:3300/0 2023-09-30T02:05:20.164 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.162+0000 7ffaaf4cd700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ffaa813fcb0 addr v1:172.21.15.110:6789/0 2023-09-30T02:05:20.164 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.162+0000 7ffaaf4cd700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaa80af430 con 0x7ffaa813fcb0 2023-09-30T02:05:20.164 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.162+0000 7ffaaf4cd700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:20.165 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.162+0000 7ffaaf4cd700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffaa8142660 con 0x7ffaa80a6080 2023-09-30T02:05:20.165 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.162+0000 7ffaaf4cd700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:20.165 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.162+0000 7ffaaf4cd700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa80a7de0 con 0x7ffaa80a5cb0 2023-09-30T02:05:20.165 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.162+0000 7ffaaf4cd700 10 monclient(hunting): _renew_subs 2023-09-30T02:05:20.165 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.162+0000 7ffaaf4cd700 10 monclient(hunting): authenticate will time out at 1765.465159s 2023-09-30T02:05:20.165 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.163+0000 7ffaaca68700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7ffaa80a6080 0x7ffaa813f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:20.165 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.163+0000 7ffaaca68700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7ffaa80a6080 0x7ffaa813f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:41580/0 (socket says 172.21.15.59:41580) 2023-09-30T02:05:20.165 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.163+0000 7ffaaca68700 1 -- 172.21.15.59:0/1216609506 learned_addr learned my addr 172.21.15.59:0/1216609506 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:05:20.165 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.163+0000 7ffaaca68700 10 monclient(hunting): get_auth_request con 0x7ffaa80a6080 auth_method 0 2023-09-30T02:05:20.165 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.163+0000 7ffaaca68700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:20.165 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.163+0000 7ffaaca68700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:20.165 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.163+0000 7ffaaca68700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:20.166 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.165+0000 7ffaad269700 1 --2- 172.21.15.59:0/1216609506 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ffaa80a5cb0 0x7ffaa80a6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:20.166 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.165+0000 7ffaad269700 10 monclient(hunting): get_auth_request con 0x7ffaa80a5cb0 auth_method 0 2023-09-30T02:05:20.166 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.165+0000 7ffaad269700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:20.166 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.165+0000 7ffaad269700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:20.166 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.165+0000 7ffaad269700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:20.166 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.166+0000 7ffaad269700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:05:20.167 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.166+0000 7ffaad269700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:05:20.167 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.166+0000 7ffaad269700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:05:20.169 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.168+0000 7ffaad269700 10 monclient(hunting): handle_auth_done global_id 4157 payload 274 2023-09-30T02:05:20.169 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.168+0000 7ffaad269700 1 -- 172.21.15.59:0/1216609506 >> v2:172.21.15.59:3300/0 conn(0x7ffaa80a6080 msgr2=0x7ffaa813f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:05:20.169 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.168+0000 7ffaad269700 1 --2- 172.21.15.59:0/1216609506 >> v2:172.21.15.59:3300/0 conn(0x7ffaa80a6080 0x7ffaa813f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:20.169 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.168+0000 7ffaad269700 1 -- 172.21.15.59:0/1216609506 >> v1:172.21.15.110:6789/0 conn(0x7ffaa813fcb0 legacy=0x7ffaa8142ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:05:20.169 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.168+0000 7ffaad269700 10 monclient: _finish_hunting 0 2023-09-30T02:05:20.169 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.168+0000 7ffaad269700 1 monclient: found mon.noname-b 2023-09-30T02:05:20.169 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.168+0000 7ffaad269700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:05:20.169 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.168+0000 7ffaad269700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.59:3301/0 2023-09-30T02:05:20.169 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.168+0000 7ffaad269700 1 -- 172.21.15.59:0/1216609506 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaa81400c0 con 0x7ffaa80a5cb0 2023-09-30T02:05:20.177 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.176+0000 7ffaad269700 1 --2- 172.21.15.59:0/1216609506 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ffaa80a5cb0 0x7ffaa80a6a90 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ffa980098d0 tx=0x7ffa98009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=820124edaa3c860c server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:05:20.287 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.284+0000 7ffa9ffff700 1 -- 172.21.15.59:0/1216609506 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa98009e70 con 0x7ffaa80a5cb0 2023-09-30T02:05:20.287 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.284+0000 7ffa9ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:05:20.288 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.284+0000 7ffa9ffff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:05:20.288 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.284+0000 7ffa9ffff700 10 monclient: dump: 2023-09-30T02:05:20.288 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2023-09-30T02:05:20.288 INFO:teuthology.misc.health.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:20.288 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:20.288 INFO:teuthology.misc.health.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:20.288 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:05:20.288 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 3 2023-09-30T02:05:20.288 INFO:teuthology.misc.health.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:05:20.289 INFO:teuthology.misc.health.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:05:20.289 INFO:teuthology.misc.health.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:05:20.289 INFO:teuthology.misc.health.smithi059.stderr: 2023-09-30T02:05:20.289 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.284+0000 7ffa9ffff700 10 monclient: _finish_auth 0 2023-09-30T02:05:20.289 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.284+0000 7ffa9ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:05:20.289 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.284+0000 7ffa9ffff700 1 -- 172.21.15.59:0/1216609506 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa98005c10 con 0x7ffaa80a5cb0 2023-09-30T02:05:20.289 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.284+0000 7ffa9ffff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:05:20.289 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.284+0000 7ffa9ffff700 1 -- 172.21.15.59:0/1216609506 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa9800b610 con 0x7ffaa80a5cb0 2023-09-30T02:05:20.289 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.284+0000 7ffa9ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:05:20.289 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.284+0000 7ffa9ffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:05:20.289 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.284+0000 7ffa9ffff700 10 monclient: dump: 2023-09-30T02:05:20.290 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2023-09-30T02:05:20.290 INFO:teuthology.misc.health.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:20.290 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:20.290 INFO:teuthology.misc.health.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:20.290 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:05:20.290 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 3 2023-09-30T02:05:20.290 INFO:teuthology.misc.health.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:05:20.290 INFO:teuthology.misc.health.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:05:20.290 INFO:teuthology.misc.health.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:05:20.291 INFO:teuthology.misc.health.smithi059.stderr: 2023-09-30T02:05:20.291 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.284+0000 7ffaaf4cd700 5 monclient: authenticate success, global_id 4157 2023-09-30T02:05:20.291 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.284+0000 7ffaaf4cd700 10 monclient: get_monmap_and_config success 2023-09-30T02:05:20.291 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.284+0000 7ffaaf4cd700 4 set_mon_vals no callback set 2023-09-30T02:05:20.291 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.285+0000 7ffaaf4cd700 10 monclient: shutdown 2023-09-30T02:05:20.291 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.285+0000 7ffaaf4cd700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:05:20.291 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.285+0000 7ffaaf4cd700 1 -- 172.21.15.59:0/1216609506 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ffaa80a5cb0 msgr2=0x7ffaa80a6a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:20.292 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.285+0000 7ffaaf4cd700 1 --2- 172.21.15.59:0/1216609506 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ffaa80a5cb0 0x7ffaa80a6a90 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ffa980098d0 tx=0x7ffa98009120 comp rx=0 tx=0).stop 2023-09-30T02:05:20.292 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.286+0000 7ffaaf4cd700 1 -- 172.21.15.59:0/1216609506 shutdown_connections 2023-09-30T02:05:20.292 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.286+0000 7ffaaf4cd700 1 --2- 172.21.15.59:0/1216609506 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ffaa80a5cb0 0x7ffaa80a6a90 secure :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7ffa980098d0 tx=0x7ffa98009120 comp rx=0 tx=0).stop 2023-09-30T02:05:20.292 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.286+0000 7ffaaf4cd700 1 --2- 172.21.15.59:0/1216609506 >> v2:172.21.15.59:3300/0 conn(0x7ffaa80a6080 0x7ffaa813f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:20.292 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.286+0000 7ffaaf4cd700 1 -- 172.21.15.59:0/1216609506 >> 172.21.15.59:0/1216609506 conn(0x7ffaa8063d50 msgr2=0x7ffaa80b4ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:05:20.292 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.292+0000 7ffaaf4cd700 1 -- 172.21.15.59:0/1216609506 shutdown_connections 2023-09-30T02:05:20.295 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.292+0000 7ffaaf4cd700 1 -- 172.21.15.59:0/1216609506 wait complete. 2023-09-30T02:05:20.295 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.293+0000 7ffaaf4cd700 10 monclient: build_initial_monmap 2023-09-30T02:05:20.296 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.293+0000 7ffaaf4cd700 10 monclient: monmap: 2023-09-30T02:05:20.296 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2023-09-30T02:05:20.296 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:05:20.296 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2023-09-30T02:05:20.296 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2023-09-30T02:05:20.296 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:05:20.296 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2023-09-30T02:05:20.296 INFO:teuthology.misc.health.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:05:20.296 INFO:teuthology.misc.health.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:05:20.296 INFO:teuthology.misc.health.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:05:20.297 INFO:teuthology.misc.health.smithi059.stderr: 2023-09-30T02:05:20.297 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.293+0000 7ffaaf4cd700 1 librados: starting msgr at 2023-09-30T02:05:20.297 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.293+0000 7ffaaf4cd700 1 librados: starting objecter 2023-09-30T02:05:20.297 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.293+0000 7ffaaf4cd700 1 Processor -- start 2023-09-30T02:05:20.298 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.295+0000 7ffaaf4cd700 1 -- start start 2023-09-30T02:05:20.298 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.295+0000 7ffaaf4cd700 1 librados: setting wanted keys 2023-09-30T02:05:20.298 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.295+0000 7ffaaf4cd700 1 librados: calling monclient init 2023-09-30T02:05:20.298 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.295+0000 7ffaaf4cd700 10 monclient: init 2023-09-30T02:05:20.298 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.296+0000 7ffaaf4cd700 10 monclient: _reopen_session rank -1 2023-09-30T02:05:20.298 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.296+0000 7ffaaf4cd700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T02:05:20.299 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.296+0000 7ffaaf4cd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffaa80a6080 0x7ffaa8145f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:20.299 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.296+0000 7ffaaf4cd700 10 monclient(hunting): picked mon.noname-b con 0x7ffaa80a6080 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:05:20.299 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.296+0000 7ffaaf4cd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffaa813fcb0 0x7ffaa813b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:20.299 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.296+0000 7ffaaf4cd700 10 monclient(hunting): picked mon.noname-a con 0x7ffaa813fcb0 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:05:20.299 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.296+0000 7ffaaf4cd700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ffaa813b5c0 0x7ffaa806e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:20.299 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.296+0000 7ffaaf4cd700 10 monclient(hunting): picked mon.noname-c con 0x7ffaa813b5c0 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:05:20.299 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.296+0000 7ffaaf4cd700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:20.299 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.296+0000 7ffaaf4cd700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa8138690 con 0x7ffaa813fcb0 2023-09-30T02:05:20.299 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.296+0000 7ffaaf4cd700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:20.299 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.296+0000 7ffaaf4cd700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa813ab60 con 0x7ffaa80a6080 2023-09-30T02:05:20.299 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.296+0000 7ffaaf4cd700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:20.300 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.296+0000 7ffaaf4cd700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaa8146440 con 0x7ffaa813b5c0 2023-09-30T02:05:20.300 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.296+0000 7ffaaf4cd700 10 monclient(hunting): _renew_subs 2023-09-30T02:05:20.300 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.296+0000 7ffaaf4cd700 10 monclient(hunting): authenticate will time out at 1765.599587s 2023-09-30T02:05:20.300 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.297+0000 7ffaada6a700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ffaa813b5c0 0x7ffaa806e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:20.300 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.298+0000 7ffaada6a700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ffaa813b5c0 0x7ffaa806e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:55398/0 (socket says 172.21.15.59:55398) 2023-09-30T02:05:20.300 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.298+0000 7ffaada6a700 1 -- 172.21.15.59:0/1981487896 learned_addr learned my addr 172.21.15.59:0/1981487896 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:05:20.300 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.298+0000 7ffaada6a700 10 monclient(hunting): get_auth_request con 0x7ffaa813b5c0 auth_method 0 2023-09-30T02:05:20.300 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.298+0000 7ffaada6a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:20.300 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.298+0000 7ffaada6a700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:20.300 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.298+0000 7ffaada6a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:20.301 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.299+0000 7ffaad269700 1 --2- 172.21.15.59:0/1981487896 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffaa80a6080 0x7ffaa8145f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:20.301 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.299+0000 7ffaada6a700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:05:20.301 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.299+0000 7ffaada6a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:05:20.301 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.299+0000 7ffaada6a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:05:20.301 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.300+0000 7ffaad269700 10 monclient(hunting): get_auth_request con 0x7ffaa80a6080 auth_method 0 2023-09-30T02:05:20.301 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.300+0000 7ffaad269700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:20.301 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.300+0000 7ffaad269700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:20.302 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.300+0000 7ffaad269700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:20.302 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.300+0000 7ffaad269700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:05:20.302 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.300+0000 7ffaad269700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:05:20.302 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.300+0000 7ffaad269700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:05:20.306 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.306+0000 7ffaada6a700 10 monclient(hunting): handle_auth_done global_id 4160 payload 824 2023-09-30T02:05:20.306 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.306+0000 7ffaada6a700 1 -- 172.21.15.59:0/1981487896 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffaa80a6080 msgr2=0x7ffaa8145f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:20.306 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.306+0000 7ffaada6a700 1 --2- 172.21.15.59:0/1981487896 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffaa80a6080 0x7ffaa8145f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:20.307 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.306+0000 7ffaada6a700 1 -- 172.21.15.59:0/1981487896 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffaa813fcb0 msgr2=0x7ffaa813b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:20.307 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.306+0000 7ffaada6a700 1 --2- 172.21.15.59:0/1981487896 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffaa813fcb0 0x7ffaa813b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:20.307 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.306+0000 7ffaada6a700 10 monclient: _finish_hunting 0 2023-09-30T02:05:20.307 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.306+0000 7ffaada6a700 1 monclient: found mon.noname-c 2023-09-30T02:05:20.307 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.306+0000 7ffaada6a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:05:20.307 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.306+0000 7ffaada6a700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.59:3301/0 2023-09-30T02:05:20.307 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.306+0000 7ffaada6a700 1 -- 172.21.15.59:0/1981487896 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa980093c0 con 0x7ffaa813b5c0 2023-09-30T02:05:20.309 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.309+0000 7ffaada6a700 1 --2- 172.21.15.59:0/1981487896 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ffaa813b5c0 0x7ffaa806e6e0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ffaa000bbb0 tx=0x7ffaa0009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:05:20.309 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.309+0000 7ffa9d7fa700 10 client.?.objecter ms_handle_connect 0x7ffaa813b5c0 2023-09-30T02:05:20.309 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.309+0000 7ffa9d7fa700 10 client.?.objecter resend_mon_ops 2023-09-30T02:05:20.315 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.314+0000 7ffa9d7fa700 1 -- 172.21.15.59:0/1981487896 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaa00092e0 con 0x7ffaa813b5c0 2023-09-30T02:05:20.315 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.314+0000 7ffa9d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:05:20.315 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.314+0000 7ffa9d7fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:05:20.316 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.314+0000 7ffa9d7fa700 10 monclient: dump: 2023-09-30T02:05:20.316 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2023-09-30T02:05:20.316 INFO:teuthology.misc.health.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:20.316 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:20.316 INFO:teuthology.misc.health.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:20.316 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:05:20.316 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 3 2023-09-30T02:05:20.316 INFO:teuthology.misc.health.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:05:20.316 INFO:teuthology.misc.health.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:05:20.316 INFO:teuthology.misc.health.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:05:20.316 INFO:teuthology.misc.health.smithi059.stderr: 2023-09-30T02:05:20.316 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.314+0000 7ffa9d7fa700 10 monclient: _finish_auth 0 2023-09-30T02:05:20.317 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.314+0000 7ffa9d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:05:20.317 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.314+0000 7ffa9d7fa700 1 -- 172.21.15.59:0/1981487896 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaa000c6f0 con 0x7ffaa813b5c0 2023-09-30T02:05:20.317 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.314+0000 7ffa9d7fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:05:20.317 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.314+0000 7ffa9d7fa700 1 -- 172.21.15.59:0/1981487896 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaa000ca90 con 0x7ffaa813b5c0 2023-09-30T02:05:20.317 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.314+0000 7ffa9d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:05:20.317 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.314+0000 7ffa9d7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:05:20.317 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.314+0000 7ffa9d7fa700 10 monclient: dump: 2023-09-30T02:05:20.317 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2023-09-30T02:05:20.317 INFO:teuthology.misc.health.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:20.317 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:20.317 INFO:teuthology.misc.health.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:20.317 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:05:20.317 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 3 2023-09-30T02:05:20.318 INFO:teuthology.misc.health.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:05:20.318 INFO:teuthology.misc.health.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:05:20.318 INFO:teuthology.misc.health.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:05:20.318 INFO:teuthology.misc.health.smithi059.stderr: 2023-09-30T02:05:20.318 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.314+0000 7ffa9f7fe700 4 set_mon_vals no callback set 2023-09-30T02:05:20.318 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.315+0000 7ffaaf4cd700 5 monclient: authenticate success, global_id 4160 2023-09-30T02:05:20.318 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.315+0000 7ffaaf4cd700 10 monclient: _renew_subs 2023-09-30T02:05:20.318 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.315+0000 7ffaaf4cd700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:05:20.318 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.315+0000 7ffaaf4cd700 1 -- 172.21.15.59:0/1981487896 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffaa80aa150 con 0x7ffaa813b5c0 2023-09-30T02:05:20.318 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.315+0000 7ffaaf4cd700 10 client.4160.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:05:20.318 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.315+0000 7ffaaf4cd700 10 monclient: _renew_subs 2023-09-30T02:05:20.319 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.315+0000 7ffaaf4cd700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:05:20.319 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.315+0000 7ffaaf4cd700 1 -- 172.21.15.59:0/1981487896 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaa806ede0 con 0x7ffaa813b5c0 2023-09-30T02:05:20.319 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.315+0000 7ffaaf4cd700 1 librados: init done 2023-09-30T02:05:20.323 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.322+0000 7ffaaf4cd700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:05:20.323 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.323+0000 7ffaaf4cd700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:05:20.323 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.323+0000 7ffaaf4cd700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:05:20.323 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.323+0000 7ffaaf4cd700 1 -- 172.21.15.59:0/1981487896 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffaa8064a00 con 0x7ffaa813b5c0 2023-09-30T02:05:20.338 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.337+0000 7ffa9d7fa700 1 -- 172.21.15.59:0/1981487896 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 2) v1 ==== 42252+0+0 (secure 0 0 0) 0x7ffaa0012070 con 0x7ffaa813b5c0 2023-09-30T02:05:20.342 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.342+0000 7ffa9d7fa700 1 -- 172.21.15.59:0/1981487896 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1641+0+0 (secure 0 0 0) 0x7ffaa0012320 con 0x7ffaa813b5c0 2023-09-30T02:05:20.343 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.342+0000 7ffa9d7fa700 10 client.4160.objecter ms_dispatch 0x7ffaa8001550 osd_map(10..10 src has 1..10) v4 2023-09-30T02:05:20.343 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.342+0000 7ffa9d7fa700 3 client.4160.objecter handle_osd_map got epochs [10,10] > 0 2023-09-30T02:05:20.343 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.342+0000 7ffa9d7fa700 3 client.4160.objecter handle_osd_map decoding full epoch 10 2023-09-30T02:05:20.343 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.342+0000 7ffa9d7fa700 20 client.4160.objecter dump_active .. 0 homeless 2023-09-30T02:05:20.538 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.538+0000 7ffa9d7fa700 1 -- 172.21.15.59:0/1981487896 <== mon.2 v2:172.21.15.59:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7ffaa0013780 con 0x7ffaa813b5c0 2023-09-30T02:05:20.539 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.538+0000 7ffa9d7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:05:20.539 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.538+0000 7ffa9d7fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:05:20.690 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.690+0000 7ffaaf4cd700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-09-30T02:05:20.690 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.690+0000 7ffaaf4cd700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T02:05:20.691 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.690+0000 7ffaaf4cd700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:05:20.691 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.690+0000 7ffaaf4cd700 1 -- 172.21.15.59:0/1981487896 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffaa813ad20 con 0x7ffaa813b5c0 2023-09-30T02:05:20.826 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.825+0000 7ffa9d7fa700 1 -- 172.21.15.59:0/1981487896 <== mon.2 v2:172.21.15.59:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7013 (secure 0 0 0) 0x7ffaa001a020 con 0x7ffaa813b5c0 2023-09-30T02:05:20.826 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.825+0000 7ffa9d7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T02:05:20.826 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.825+0000 7ffa9d7fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:05:20.827 INFO:teuthology.misc.health.smithi059.stdout: 2023-09-30T02:05:20.827 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"623052cb-4f82-4b2b-b72b-3ed5be614613","created":"2023-09-30T02:05:11.679291+0000","modified":"2023-09-30T02:05:18.270595+0000","last_up_change":"0.000000","last_in_change":"2023-09-30T02:05:18.270595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"efdab44b-be57-4b81-8567-ff2deec1d3e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff9786d-8c39-4044-91d2-a31f33e3b16e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b276ed4b-317b-4ddf-96ab-7ad71b8e3ff7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"48bfaf43-5846-4385-b9c6-6bc424cd5acd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"76fa5443-1571-4e72-9d22-6ab19e59839a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e9436478-4140-4fd6-ba2e-04fa6f6cf367","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"76676042-2074-4cf8-9ef9-5c87bf604944","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"045479f4-f7f7-4b95-bc38-a2c52ac31b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T02:05:20.828 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.827+0000 7ffaaf4cd700 10 librados: watch_flush enter 2023-09-30T02:05:20.828 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.827+0000 7ffaaf4cd700 10 librados: watch_flush exit 2023-09-30T02:05:20.828 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.827+0000 7ffaaf4cd700 20 client.4160.objecter shutdown clearing up homeless session... 2023-09-30T02:05:20.828 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.827+0000 7ffaaf4cd700 10 client.4160.objecter successfully canceled tick 2023-09-30T02:05:20.828 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.827+0000 7ffaaf4cd700 10 monclient: shutdown 2023-09-30T02:05:20.828 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.827+0000 7ffaaf4cd700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:05:20.828 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.827+0000 7ffaaf4cd700 1 -- 172.21.15.59:0/1981487896 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ffaa813b5c0 msgr2=0x7ffaa806e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:20.828 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.827+0000 7ffaaf4cd700 1 --2- 172.21.15.59:0/1981487896 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ffaa813b5c0 0x7ffaa806e6e0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ffaa000bbb0 tx=0x7ffaa0009840 comp rx=0 tx=0).stop 2023-09-30T02:05:20.828 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.827+0000 7ffaaf4cd700 1 -- 172.21.15.59:0/1981487896 shutdown_connections 2023-09-30T02:05:20.829 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.827+0000 7ffaaf4cd700 1 --2- 172.21.15.59:0/1981487896 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ffaa80a6080 0x7ffaa8145f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:20.829 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.827+0000 7ffaaf4cd700 1 --2- 172.21.15.59:0/1981487896 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ffaa813b5c0 0x7ffaa806e6e0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:20.829 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.827+0000 7ffaaf4cd700 1 --2- 172.21.15.59:0/1981487896 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ffaa813fcb0 0x7ffaa813b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:20.829 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.827+0000 7ffaaf4cd700 1 -- 172.21.15.59:0/1981487896 >> 172.21.15.59:0/1981487896 conn(0x7ffaa8063d50 msgr2=0x7ffaa81418b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:05:20.829 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.829+0000 7ffaaf4cd700 1 -- 172.21.15.59:0/1981487896 shutdown_connections 2023-09-30T02:05:20.830 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.829+0000 7ffaaf4cd700 1 -- 172.21.15.59:0/1981487896 wait complete. 2023-09-30T02:05:20.830 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:20.829+0000 7ffaaf4cd700 1 librados: shutdown 2023-09-30T02:05:20.846 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-09-30T02:05:21.439 INFO:tasks.ceph.mgr.x.smithi110.stderr:2023-09-30T02:05:21.438+0000 7fadd88be700 -1 mgr.server handle_report got status from non-daemon mon.b 2023-09-30T02:05:21.484 INFO:tasks.ceph.mgr.x.smithi110.stderr:2023-09-30T02:05:21.484+0000 7fadd88be700 -1 mgr.server handle_report got status from non-daemon mon.c 2023-09-30T02:05:27.652 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-09-30T02:05:27.858 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.858+0000 7f74eb5c8700 10 monclient: get_monmap_and_config 2023-09-30T02:05:27.858 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.858+0000 7f74eb5c8700 10 monclient: build_initial_monmap 2023-09-30T02:05:27.858 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.858+0000 7f74eb5c8700 10 monclient: monmap: 2023-09-30T02:05:27.858 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2023-09-30T02:05:27.859 INFO:teuthology.misc.health.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:27.859 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2023-09-30T02:05:27.859442+0000 2023-09-30T02:05:27.859 INFO:teuthology.misc.health.smithi059.stderr:created 2023-09-30T02:05:27.859442+0000 2023-09-30T02:05:27.859 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:05:27.859 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 3 2023-09-30T02:05:27.859 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:05:27.859 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:05:27.859 INFO:teuthology.misc.health.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:05:27.859 INFO:teuthology.misc.health.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:05:27.859 INFO:teuthology.misc.health.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:05:27.859 INFO:teuthology.misc.health.smithi059.stderr: 2023-09-30T02:05:27.860 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.859+0000 7f74eb5c8700 1 Processor -- start 2023-09-30T02:05:27.860 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.859+0000 7f74eb5c8700 1 -- start start 2023-09-30T02:05:27.860 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.859+0000 7f74eb5c8700 10 monclient: init 2023-09-30T02:05:27.860 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.860+0000 7f74eb5c8700 10 monclient: _reopen_session rank -1 2023-09-30T02:05:27.860 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.860+0000 7f74eb5c8700 10 monclient: _add_conns ranks=[2,3,1,0,4] 2023-09-30T02:05:27.861 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.860+0000 7f74eb5c8700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f74e40b1660 0x7f74e40a54e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:27.861 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.860+0000 7f74eb5c8700 10 monclient(hunting): picked mon.noname-b con 0x7f74e40b1660 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:05:27.861 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.860+0000 7f74eb5c8700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f74e40a7630 0x7f74e40a7a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:27.861 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.860+0000 7f74eb5c8700 10 monclient(hunting): picked mon.noname-a con 0x7f74e40a7630 addr v2:172.21.15.59:3300/0 2023-09-30T02:05:27.861 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.860+0000 7f74eb5c8700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f74e40a7f80 addr v1:172.21.15.110:6789/0 2023-09-30T02:05:27.861 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.860+0000 7f74eb5c8700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74e40a5e40 con 0x7f74e40a7f80 2023-09-30T02:05:27.861 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.860+0000 7f74eb5c8700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:27.861 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.860+0000 7f74eb5c8700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74e40b1a30 con 0x7f74e40a7630 2023-09-30T02:05:27.862 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.860+0000 7f74eb5c8700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:27.862 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.860+0000 7f74eb5c8700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74e40a83e0 con 0x7f74e40b1660 2023-09-30T02:05:27.862 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.860+0000 7f74eb5c8700 10 monclient(hunting): _renew_subs 2023-09-30T02:05:27.862 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.860+0000 7f74eb5c8700 10 monclient(hunting): authenticate will time out at 1773.163559s 2023-09-30T02:05:27.863 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.861+0000 7f74e9b65700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f74e40a7f80 0x7f74e40af430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.59:53722/0 (socket says 172.21.15.59:53722) 2023-09-30T02:05:27.863 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.861+0000 7f74e9b65700 1 -- 172.21.15.59:0/1457476440 learned_addr learned my addr 172.21.15.59:0/1457476440 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:05:27.863 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.861+0000 7f74e8b63700 1 --2- 172.21.15.59:0/1457476440 >> v2:172.21.15.59:3300/0 conn(0x7f74e40a7630 0x7f74e40a7a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:27.863 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.861+0000 7f74e9364700 1 --2- 172.21.15.59:0/1457476440 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f74e40b1660 0x7f74e40a54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:27.863 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.862+0000 7f74e9364700 10 monclient(hunting): get_auth_request con 0x7f74e40b1660 auth_method 0 2023-09-30T02:05:27.863 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.862+0000 7f74e9364700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:27.863 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.862+0000 7f74e9364700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:27.863 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.862+0000 7f74e9364700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:27.863 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.862+0000 7f74e8b63700 10 monclient(hunting): get_auth_request con 0x7f74e40a7630 auth_method 0 2023-09-30T02:05:27.864 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.862+0000 7f74e8b63700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:27.864 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.862+0000 7f74e8b63700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:27.864 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.862+0000 7f74e8b63700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:27.864 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.863+0000 7f74e9364700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:05:27.864 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.863+0000 7f74e9364700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:05:27.864 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.863+0000 7f74e9364700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:05:27.864 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.863+0000 7f74e8b63700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:05:27.864 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.863+0000 7f74e8b63700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:05:27.864 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.863+0000 7f74e8b63700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:05:27.866 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.866+0000 7f74dbfff700 1 -- 172.21.15.59:0/1457476440 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1677913405 0 0) 0x7f74e40a5e40 con 0x7f74e40a7f80 2023-09-30T02:05:27.867 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.866+0000 7f74dbfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:27.867 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.866+0000 7f74dbfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:27.867 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.866+0000 7f74dbfff700 10 monclient(hunting): my global_id is 4177 2023-09-30T02:05:27.867 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.866+0000 7f74dbfff700 1 -- 172.21.15.59:0/1457476440 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74d0003570 con 0x7f74e40a7f80 2023-09-30T02:05:27.867 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.867+0000 7f74e9364700 10 monclient(hunting): handle_auth_done global_id 4163 payload 274 2023-09-30T02:05:27.867 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.867+0000 7f74e9364700 1 -- 172.21.15.59:0/1457476440 >> v2:172.21.15.59:3300/0 conn(0x7f74e40a7630 msgr2=0x7f74e40a7a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:05:27.867 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.867+0000 7f74e9364700 1 --2- 172.21.15.59:0/1457476440 >> v2:172.21.15.59:3300/0 conn(0x7f74e40a7630 0x7f74e40a7a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:27.868 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.867+0000 7f74e9364700 1 -- 172.21.15.59:0/1457476440 >> v1:172.21.15.110:6789/0 conn(0x7f74e40a7f80 legacy=0x7f74e40af430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:27.868 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.867+0000 7f74e9364700 10 monclient: _finish_hunting 0 2023-09-30T02:05:27.868 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.867+0000 7f74e9364700 1 monclient: found mon.noname-b 2023-09-30T02:05:27.868 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.867+0000 7f74e9364700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:05:27.868 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.867+0000 7f74e9364700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.59:3301/0 2023-09-30T02:05:27.868 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.867+0000 7f74e9364700 1 -- 172.21.15.59:0/1457476440 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74e40a6f30 con 0x7f74e40b1660 2023-09-30T02:05:27.870 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.870+0000 7f74e9364700 1 --2- 172.21.15.59:0/1457476440 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f74e40b1660 0x7f74e40a54e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f74e00098e0 tx=0x7f74e00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa110f3763ba882e server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:05:27.878 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.877+0000 7f74dbfff700 1 -- 172.21.15.59:0/1457476440 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74e0011630 con 0x7f74e40b1660 2023-09-30T02:05:27.878 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.877+0000 7f74dbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:05:27.878 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.877+0000 7f74dbfff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:05:27.878 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.877+0000 7f74dbfff700 10 monclient: dump: 2023-09-30T02:05:27.879 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2023-09-30T02:05:27.879 INFO:teuthology.misc.health.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:27.879 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:27.879 INFO:teuthology.misc.health.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:27.879 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:05:27.879 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 3 2023-09-30T02:05:27.879 INFO:teuthology.misc.health.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:05:27.879 INFO:teuthology.misc.health.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:05:27.880 INFO:teuthology.misc.health.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:05:27.880 INFO:teuthology.misc.health.smithi059.stderr: 2023-09-30T02:05:27.880 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.877+0000 7f74dbfff700 10 monclient: _finish_auth 0 2023-09-30T02:05:27.880 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.877+0000 7f74dbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:05:27.880 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.877+0000 7f74dbfff700 1 -- 172.21.15.59:0/1457476440 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74e0011790 con 0x7f74e40b1660 2023-09-30T02:05:27.880 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.877+0000 7f74eb5c8700 5 monclient: authenticate success, global_id 4163 2023-09-30T02:05:27.880 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.877+0000 7f74eb5c8700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:05:27.880 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.877+0000 7f74dbfff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:05:27.880 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.877+0000 7f74dbfff700 1 -- 172.21.15.59:0/1457476440 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74e0011b30 con 0x7f74e40b1660 2023-09-30T02:05:27.880 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.877+0000 7f74dbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:05:27.881 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.877+0000 7f74dbfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:05:27.881 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.877+0000 7f74dbfff700 10 monclient: dump: 2023-09-30T02:05:27.881 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2023-09-30T02:05:27.881 INFO:teuthology.misc.health.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:27.881 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:27.881 INFO:teuthology.misc.health.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:27.881 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:05:27.881 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 3 2023-09-30T02:05:27.881 INFO:teuthology.misc.health.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:05:27.881 INFO:teuthology.misc.health.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:05:27.881 INFO:teuthology.misc.health.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:05:27.882 INFO:teuthology.misc.health.smithi059.stderr: 2023-09-30T02:05:27.882 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.878+0000 7f74eb5c8700 10 monclient: get_monmap_and_config success 2023-09-30T02:05:27.882 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.878+0000 7f74eb5c8700 4 set_mon_vals no callback set 2023-09-30T02:05:27.883 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.878+0000 7f74eb5c8700 10 monclient: shutdown 2023-09-30T02:05:27.883 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.879+0000 7f74eb5c8700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:05:27.884 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.879+0000 7f74eb5c8700 1 -- 172.21.15.59:0/1457476440 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f74e40b1660 msgr2=0x7f74e40a54e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:27.884 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.879+0000 7f74eb5c8700 1 --2- 172.21.15.59:0/1457476440 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f74e40b1660 0x7f74e40a54e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f74e00098e0 tx=0x7f74e00091a0 comp rx=0 tx=0).stop 2023-09-30T02:05:27.884 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.879+0000 7f74eb5c8700 1 -- 172.21.15.59:0/1457476440 shutdown_connections 2023-09-30T02:05:27.884 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.879+0000 7f74eb5c8700 1 --2- 172.21.15.59:0/1457476440 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f74e40b1660 0x7f74e40a54e0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:27.884 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.879+0000 7f74eb5c8700 1 --2- 172.21.15.59:0/1457476440 >> v2:172.21.15.59:3300/0 conn(0x7f74e40a7630 0x7f74e40a7a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:27.884 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.879+0000 7f74eb5c8700 1 -- 172.21.15.59:0/1457476440 >> 172.21.15.59:0/1457476440 conn(0x7f74e4069130 msgr2=0x7f74e4063280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:05:27.884 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.879+0000 7f74eb5c8700 1 -- 172.21.15.59:0/1457476440 shutdown_connections 2023-09-30T02:05:27.884 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.879+0000 7f74eb5c8700 1 -- 172.21.15.59:0/1457476440 wait complete. 2023-09-30T02:05:27.884 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.880+0000 7f74eb5c8700 10 monclient: build_initial_monmap 2023-09-30T02:05:27.884 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.880+0000 7f74eb5c8700 10 monclient: monmap: 2023-09-30T02:05:27.884 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2023-09-30T02:05:27.885 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:05:27.885 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2023-09-30T02:05:27.885 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2023-09-30T02:05:27.885 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:05:27.885 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2023-09-30T02:05:27.885 INFO:teuthology.misc.health.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:05:27.885 INFO:teuthology.misc.health.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:05:27.885 INFO:teuthology.misc.health.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:05:27.885 INFO:teuthology.misc.health.smithi059.stderr: 2023-09-30T02:05:27.885 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.881+0000 7f74eb5c8700 1 librados: starting msgr at 2023-09-30T02:05:27.885 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.881+0000 7f74eb5c8700 1 librados: starting objecter 2023-09-30T02:05:27.885 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.881+0000 7f74eb5c8700 1 Processor -- start 2023-09-30T02:05:27.886 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.881+0000 7f74eb5c8700 1 -- start start 2023-09-30T02:05:27.886 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.881+0000 7f74eb5c8700 1 librados: setting wanted keys 2023-09-30T02:05:27.886 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.881+0000 7f74eb5c8700 1 librados: calling monclient init 2023-09-30T02:05:27.886 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.881+0000 7f74eb5c8700 10 monclient: init 2023-09-30T02:05:27.886 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.882+0000 7f74eb5c8700 10 monclient: _reopen_session rank -1 2023-09-30T02:05:27.886 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.882+0000 7f74eb5c8700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T02:05:27.886 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.882+0000 7f74eb5c8700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f74e40a7630 0x7f74e40ae470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:27.886 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.882+0000 7f74eb5c8700 10 monclient(hunting): picked mon.noname-b con 0x7f74e40a7630 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:05:27.886 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.882+0000 7f74eb5c8700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f74e40a7f80 0x7f74e4152800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:27.886 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.882+0000 7f74eb5c8700 10 monclient(hunting): picked mon.noname-a con 0x7f74e40a7f80 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:05:27.886 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.882+0000 7f74eb5c8700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f74e40b1660 0x7f74e4155d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:27.887 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.882+0000 7f74eb5c8700 10 monclient(hunting): picked mon.noname-c con 0x7f74e40b1660 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:05:27.887 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.882+0000 7f74eb5c8700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:27.887 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.882+0000 7f74eb5c8700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74e41558a0 con 0x7f74e40a7f80 2023-09-30T02:05:27.887 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.882+0000 7f74eb5c8700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:27.887 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.882+0000 7f74eb5c8700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74e40a9130 con 0x7f74e40a7630 2023-09-30T02:05:27.887 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.882+0000 7f74eb5c8700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:27.887 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.882+0000 7f74eb5c8700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74e4140fb0 con 0x7f74e40b1660 2023-09-30T02:05:27.887 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.882+0000 7f74eb5c8700 10 monclient(hunting): _renew_subs 2023-09-30T02:05:27.887 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.882+0000 7f74eb5c8700 10 monclient(hunting): authenticate will time out at 1773.185835s 2023-09-30T02:05:27.887 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.883+0000 7f74e9364700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f74e40a7630 0x7f74e40ae470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:27.887 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.883+0000 7f74e9b65700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f74e40b1660 0x7f74e4155d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:27.888 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.884+0000 7f74e9b65700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f74e40b1660 0x7f74e4155d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:45296/0 (socket says 172.21.15.59:45296) 2023-09-30T02:05:27.888 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.884+0000 7f74e9b65700 1 -- 172.21.15.59:0/2705493121 learned_addr learned my addr 172.21.15.59:0/2705493121 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:05:27.888 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.884+0000 7f74e9b65700 10 monclient(hunting): get_auth_request con 0x7f74e40b1660 auth_method 0 2023-09-30T02:05:27.888 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.884+0000 7f74e9b65700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:27.889 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.884+0000 7f74e9b65700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:27.889 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.884+0000 7f74e9b65700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:27.889 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.885+0000 7f74e9364700 10 monclient(hunting): get_auth_request con 0x7f74e40a7630 auth_method 0 2023-09-30T02:05:27.889 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.885+0000 7f74e9364700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:27.889 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.885+0000 7f74e9364700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:27.889 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.885+0000 7f74e9364700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:27.889 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.885+0000 7f74e9b65700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:05:27.889 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.885+0000 7f74e9b65700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:05:27.889 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.885+0000 7f74e9b65700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:05:27.889 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.887+0000 7f74e8b63700 1 --2- 172.21.15.59:0/2705493121 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f74e40a7f80 0x7f74e4152800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:27.889 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.887+0000 7f74e8b63700 10 monclient(hunting): get_auth_request con 0x7f74e40a7f80 auth_method 0 2023-09-30T02:05:27.889 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.887+0000 7f74e8b63700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:27.890 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.887+0000 7f74e8b63700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:27.890 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.887+0000 7f74e8b63700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:27.890 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.889+0000 7f74e9364700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:05:27.890 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.889+0000 7f74e9364700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:05:27.890 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.889+0000 7f74e9364700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:05:27.890 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.889+0000 7f74e8b63700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:05:27.890 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.889+0000 7f74e8b63700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:05:27.890 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.889+0000 7f74e8b63700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:05:27.892 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.891+0000 7f74e9b65700 10 monclient(hunting): handle_auth_done global_id 4166 payload 824 2023-09-30T02:05:27.892 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.891+0000 7f74e9b65700 1 -- 172.21.15.59:0/2705493121 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f74e40a7630 msgr2=0x7f74e40ae470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:27.892 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.891+0000 7f74e9b65700 1 --2- 172.21.15.59:0/2705493121 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f74e40a7630 0x7f74e40ae470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:27.892 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.891+0000 7f74e9b65700 1 -- 172.21.15.59:0/2705493121 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f74e40a7f80 msgr2=0x7f74e4152800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:27.892 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.891+0000 7f74e9b65700 1 --2- 172.21.15.59:0/2705493121 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f74e40a7f80 0x7f74e4152800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:27.892 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.891+0000 7f74e9b65700 10 monclient: _finish_hunting 0 2023-09-30T02:05:27.892 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.891+0000 7f74e9b65700 1 monclient: found mon.noname-c 2023-09-30T02:05:27.892 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.891+0000 7f74e9b65700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:05:27.892 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.891+0000 7f74e9b65700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.59:3301/0 2023-09-30T02:05:27.892 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.891+0000 7f74e9b65700 1 -- 172.21.15.59:0/2705493121 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74e0009420 con 0x7f74e40b1660 2023-09-30T02:05:27.895 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.895+0000 7f74e9b65700 1 --2- 172.21.15.59:0/2705493121 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f74e40b1660 0x7f74e4155d30 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f74dc001460 tx=0x7f74dc001ae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:05:27.895 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.895+0000 7f74d97fa700 10 client.?.objecter ms_handle_connect 0x7f74e40b1660 2023-09-30T02:05:27.896 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.895+0000 7f74d97fa700 10 client.?.objecter resend_mon_ops 2023-09-30T02:05:27.902 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.902+0000 7f74d97fa700 1 -- 172.21.15.59:0/2705493121 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74dc00a890 con 0x7f74e40b1660 2023-09-30T02:05:27.902 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.902+0000 7f74d97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:05:27.902 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.902+0000 7f74d97fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:05:27.902 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.902+0000 7f74d97fa700 10 monclient: dump: 2023-09-30T02:05:27.902 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2023-09-30T02:05:27.903 INFO:teuthology.misc.health.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:27.903 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:27.903 INFO:teuthology.misc.health.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:27.903 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:05:27.903 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 3 2023-09-30T02:05:27.903 INFO:teuthology.misc.health.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:05:27.903 INFO:teuthology.misc.health.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:05:27.903 INFO:teuthology.misc.health.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:05:27.903 INFO:teuthology.misc.health.smithi059.stderr: 2023-09-30T02:05:27.903 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.902+0000 7f74d97fa700 10 monclient: _finish_auth 0 2023-09-30T02:05:27.903 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.902+0000 7f74d97fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:05:27.903 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.902+0000 7f74d97fa700 1 -- 172.21.15.59:0/2705493121 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74dc00a9f0 con 0x7f74e40b1660 2023-09-30T02:05:27.904 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.902+0000 7f74eb5c8700 5 monclient: authenticate success, global_id 4166 2023-09-30T02:05:27.904 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.902+0000 7f74eb5c8700 10 monclient: _renew_subs 2023-09-30T02:05:27.904 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.902+0000 7f74eb5c8700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:05:27.904 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.902+0000 7f74eb5c8700 1 -- 172.21.15.59:0/2705493121 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74e4152d40 con 0x7f74e40b1660 2023-09-30T02:05:27.904 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.902+0000 7f74d97fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:05:27.904 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.902+0000 7f74d97fa700 1 -- 172.21.15.59:0/2705493121 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74dc00ad70 con 0x7f74e40b1660 2023-09-30T02:05:27.904 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.902+0000 7f74d97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:05:27.904 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.902+0000 7f74d97fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:05:27.904 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.902+0000 7f74d97fa700 10 monclient: dump: 2023-09-30T02:05:27.904 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2023-09-30T02:05:27.904 INFO:teuthology.misc.health.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:27.904 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:27.905 INFO:teuthology.misc.health.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:27.905 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:05:27.905 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 3 2023-09-30T02:05:27.905 INFO:teuthology.misc.health.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:05:27.905 INFO:teuthology.misc.health.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:05:27.905 INFO:teuthology.misc.health.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:05:27.905 INFO:teuthology.misc.health.smithi059.stderr: 2023-09-30T02:05:27.905 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.902+0000 7f74db7fe700 4 set_mon_vals no callback set 2023-09-30T02:05:27.905 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.902+0000 7f74eb5c8700 10 client.4166.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:05:27.905 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.902+0000 7f74eb5c8700 10 monclient: _renew_subs 2023-09-30T02:05:27.905 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.902+0000 7f74eb5c8700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:05:27.905 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.902+0000 7f74eb5c8700 1 -- 172.21.15.59:0/2705493121 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74e41562f0 con 0x7f74e40b1660 2023-09-30T02:05:27.906 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.902+0000 7f74eb5c8700 1 librados: init done 2023-09-30T02:05:27.906 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.903+0000 7f74eb5c8700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:05:27.906 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.903+0000 7f74eb5c8700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:05:27.906 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.903+0000 7f74eb5c8700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:05:27.906 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:27.903+0000 7f74eb5c8700 1 -- 172.21.15.59:0/2705493121 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74e4155a10 con 0x7f74e40b1660 2023-09-30T02:05:28.050 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:28.049+0000 7f74d97fa700 1 -- 172.21.15.59:0/2705493121 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 4) v1 ==== 84649+0+0 (secure 0 0 0) 0x7f74dc007300 con 0x7f74e40b1660 2023-09-30T02:05:28.050 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:28.050+0000 7f74d97fa700 1 --2- 172.21.15.59:0/2705493121 >> v2:172.21.15.110:6800/115870 conn(0x7f74c8068800 0x7f74c806acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:28.051 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:28.050+0000 7f74d97fa700 1 -- 172.21.15.59:0/2705493121 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1641+0+0 (secure 0 0 0) 0x7f74dc084210 con 0x7f74e40b1660 2023-09-30T02:05:28.051 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:28.050+0000 7f74d97fa700 10 client.4166.objecter ms_dispatch 0x7f74e4001550 osd_map(10..10 src has 1..10) v4 2023-09-30T02:05:28.051 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:28.050+0000 7f74d97fa700 3 client.4166.objecter handle_osd_map got epochs [10,10] > 0 2023-09-30T02:05:28.051 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:28.050+0000 7f74d97fa700 3 client.4166.objecter handle_osd_map decoding full epoch 10 2023-09-30T02:05:28.051 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:28.050+0000 7f74d97fa700 20 client.4166.objecter dump_active .. 0 homeless 2023-09-30T02:05:28.051 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:28.050+0000 7f74e9364700 1 --2- 172.21.15.59:0/2705493121 >> v2:172.21.15.110:6800/115870 conn(0x7f74c8068800 0x7f74c806acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:28.051 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:28.050+0000 7f74e9364700 10 monclient: get_auth_request con 0x7f74c8068800 auth_method 0 2023-09-30T02:05:28.052 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:28.051+0000 7f74d97fa700 10 client.4166.objecter ms_handle_connect 0x7f74c8068800 2023-09-30T02:05:28.052 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:28.051+0000 7f74e9364700 1 --2- 172.21.15.59:0/2705493121 >> v2:172.21.15.110:6800/115870 conn(0x7f74c8068800 0x7f74c806acb0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f74e00091e0 tx=0x7f74e0017000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:05:28.437 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:28.436+0000 7f74d97fa700 1 -- 172.21.15.59:0/2705493121 <== mon.2 v2:172.21.15.59:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f74dc0526a0 con 0x7f74e40b1660 2023-09-30T02:05:28.437 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:28.436+0000 7f74d97fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:05:28.437 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:28.436+0000 7f74d97fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:05:28.681 INFO:tasks.ceph.osd.4.smithi110.stderr:==00:00:00:10.244 116560== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:28.759 INFO:tasks.ceph.osd.7.smithi110.stderr:==00:00:00:10.313 116561== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:28.765 INFO:tasks.ceph.osd.4.smithi110.stderr:==00:00:00:10.327 116560== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:28.856 INFO:tasks.ceph.osd.7.smithi110.stderr:==00:00:00:10.410 116561== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:28.874 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:28.873+0000 7f74eb5c8700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-09-30T02:05:28.874 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:28.873+0000 7f74eb5c8700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T02:05:28.874 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:28.873+0000 7f74eb5c8700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:05:28.874 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:28.873+0000 7f74eb5c8700 1 -- 172.21.15.59:0/2705493121 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f74e4064a00 con 0x7f74e40b1660 2023-09-30T02:05:28.876 INFO:tasks.ceph.osd.5.smithi110.stderr:==00:00:00:10.414 116562== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:28.905 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:28.905+0000 7f74d97fa700 1 -- 172.21.15.59:0/2705493121 <== mon.2 v2:172.21.15.59:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7013 (secure 0 0 0) 0x7f74dc055cc0 con 0x7f74e40b1660 2023-09-30T02:05:28.905 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:28.905+0000 7f74d97fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T02:05:28.905 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:28.905+0000 7f74d97fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:05:28.906 INFO:teuthology.misc.health.smithi059.stdout: 2023-09-30T02:05:28.906 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"623052cb-4f82-4b2b-b72b-3ed5be614613","created":"2023-09-30T02:05:11.679291+0000","modified":"2023-09-30T02:05:18.270595+0000","last_up_change":"0.000000","last_in_change":"2023-09-30T02:05:18.270595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"efdab44b-be57-4b81-8567-ff2deec1d3e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff9786d-8c39-4044-91d2-a31f33e3b16e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b276ed4b-317b-4ddf-96ab-7ad71b8e3ff7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"48bfaf43-5846-4385-b9c6-6bc424cd5acd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"76fa5443-1571-4e72-9d22-6ab19e59839a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e9436478-4140-4fd6-ba2e-04fa6f6cf367","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"76676042-2074-4cf8-9ef9-5c87bf604944","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"045479f4-f7f7-4b95-bc38-a2c52ac31b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T02:05:28.908 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:28.908+0000 7f74eb5c8700 10 librados: watch_flush enter 2023-09-30T02:05:28.908 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:28.908+0000 7f74eb5c8700 10 librados: watch_flush exit 2023-09-30T02:05:28.908 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:28.908+0000 7f74eb5c8700 20 client.4166.objecter shutdown clearing up homeless session... 2023-09-30T02:05:28.908 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:28.908+0000 7f74eb5c8700 10 client.4166.objecter successfully canceled tick 2023-09-30T02:05:28.909 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:28.908+0000 7f74eb5c8700 1 -- 172.21.15.59:0/2705493121 >> v2:172.21.15.110:6800/115870 conn(0x7f74c8068800 msgr2=0x7f74c806acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:28.909 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:28.908+0000 7f74eb5c8700 1 --2- 172.21.15.59:0/2705493121 >> v2:172.21.15.110:6800/115870 conn(0x7f74c8068800 0x7f74c806acb0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f74e00091e0 tx=0x7f74e0017000 comp rx=0 tx=0).stop 2023-09-30T02:05:28.909 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:28.908+0000 7f74eb5c8700 10 monclient: shutdown 2023-09-30T02:05:28.909 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:28.908+0000 7f74eb5c8700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:05:28.909 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:28.908+0000 7f74eb5c8700 1 -- 172.21.15.59:0/2705493121 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f74e40b1660 msgr2=0x7f74e4155d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:28.910 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:28.908+0000 7f74eb5c8700 1 --2- 172.21.15.59:0/2705493121 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f74e40b1660 0x7f74e4155d30 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f74dc001460 tx=0x7f74dc001ae0 comp rx=0 tx=0).stop 2023-09-30T02:05:28.910 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:28.908+0000 7f74eb5c8700 1 -- 172.21.15.59:0/2705493121 shutdown_connections 2023-09-30T02:05:28.910 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:28.908+0000 7f74eb5c8700 1 --2- 172.21.15.59:0/2705493121 >> v2:172.21.15.110:6800/115870 conn(0x7f74c8068800 0x7f74c806acb0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:28.910 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:28.908+0000 7f74eb5c8700 1 --2- 172.21.15.59:0/2705493121 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f74e40a7630 0x7f74e40ae470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:28.910 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:28.908+0000 7f74eb5c8700 1 --2- 172.21.15.59:0/2705493121 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f74e40b1660 0x7f74e4155d30 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:28.910 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:28.908+0000 7f74eb5c8700 1 --2- 172.21.15.59:0/2705493121 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f74e40a7f80 0x7f74e4152800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:28.910 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:28.908+0000 7f74eb5c8700 1 -- 172.21.15.59:0/2705493121 >> 172.21.15.59:0/2705493121 conn(0x7f74e4069130 msgr2=0x7f74e4138260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:05:28.911 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:28.909+0000 7f74eb5c8700 1 -- 172.21.15.59:0/2705493121 shutdown_connections 2023-09-30T02:05:28.911 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:28.910+0000 7f74eb5c8700 1 -- 172.21.15.59:0/2705493121 wait complete. 2023-09-30T02:05:28.911 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:28.910+0000 7f74eb5c8700 1 librados: shutdown 2023-09-30T02:05:28.933 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-09-30T02:05:28.955 INFO:tasks.ceph.osd.5.smithi110.stderr:==00:00:00:10.493 116562== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:29.044 INFO:tasks.ceph.osd.6.smithi110.stderr:==00:00:00:10.582 116563== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:29.122 INFO:tasks.ceph.osd.6.smithi110.stderr:==00:00:00:10.658 116563== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:29.309 INFO:tasks.ceph.osd.4.smithi110.stderr:2023-09-30T02:05:29.301+0000 40451c0 -1 Falling back to public interface 2023-09-30T02:05:29.406 INFO:tasks.ceph.osd.7.smithi110.stderr:2023-09-30T02:05:29.397+0000 40451c0 -1 Falling back to public interface 2023-09-30T02:05:29.512 INFO:tasks.ceph.osd.5.smithi110.stderr:2023-09-30T02:05:29.504+0000 40451c0 -1 Falling back to public interface 2023-09-30T02:05:29.692 INFO:tasks.ceph.osd.6.smithi110.stderr:2023-09-30T02:05:29.680+0000 40451c0 -1 Falling back to public interface 2023-09-30T02:05:29.940 INFO:tasks.ceph.osd.4.smithi110.stderr:==00:00:00:11.502 116560== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:30.147 INFO:tasks.ceph.osd.7.smithi110.stderr:==00:00:00:11.701 116561== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:30.182 INFO:tasks.ceph.osd.5.smithi110.stderr:==00:00:00:11.719 116562== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:30.304 INFO:tasks.ceph.osd.4.smithi110.stderr:==00:00:00:11.866 116560== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:30.366 INFO:tasks.ceph.osd.2.smithi059.stderr:==00:00:00:11.893 116666== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:30.367 INFO:tasks.ceph.osd.0.smithi059.stderr:==00:00:00:11.914 116664== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:30.427 INFO:tasks.ceph.osd.6.smithi110.stderr:==00:00:00:11.965 116563== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:30.473 INFO:tasks.ceph.osd.0.smithi059.stderr:==00:00:00:12.020 116664== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:30.483 INFO:tasks.ceph.osd.2.smithi059.stderr:==00:00:00:12.011 116666== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:30.515 INFO:tasks.ceph.osd.7.smithi110.stderr:==00:00:00:12.069 116561== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:30.549 INFO:tasks.ceph.osd.5.smithi110.stderr:==00:00:00:12.086 116562== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:30.630 INFO:tasks.ceph.osd.4.smithi110.stderr:==00:00:00:12.193 116560== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:30.675 INFO:tasks.ceph.osd.1.smithi059.stderr:==00:00:00:12.213 116665== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:30.762 INFO:tasks.ceph.osd.1.smithi059.stderr:==00:00:00:12.300 116665== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:30.790 INFO:tasks.ceph.osd.6.smithi110.stderr:==00:00:00:12.328 116563== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:30.847 INFO:tasks.ceph.osd.7.smithi110.stderr:==00:00:00:12.401 116561== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:30.857 INFO:tasks.ceph.osd.3.smithi059.stderr:==00:00:00:12.380 116667== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:30.886 INFO:tasks.ceph.osd.5.smithi110.stderr:==00:00:00:12.424 116562== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:30.901 INFO:tasks.ceph.osd.4.smithi110.stderr:==00:00:00:12.463 116560== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:30.940 INFO:tasks.ceph.osd.3.smithi059.stderr:==00:00:00:12.462 116667== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:30.996 INFO:tasks.ceph.osd.0.smithi059.stderr:2023-09-30T02:05:30.988+0000 40451c0 -1 Falling back to public interface 2023-09-30T02:05:31.016 INFO:tasks.ceph.osd.2.smithi059.stderr:2023-09-30T02:05:31.007+0000 40451c0 -1 Falling back to public interface 2023-09-30T02:05:31.136 INFO:tasks.ceph.osd.7.smithi110.stderr:==00:00:00:12.690 116561== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:31.167 INFO:tasks.ceph.osd.6.smithi110.stderr:==00:00:00:12.705 116563== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:31.175 INFO:tasks.ceph.osd.5.smithi110.stderr:==00:00:00:12.713 116562== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:31.188 INFO:tasks.ceph.osd.4.smithi110.stderr:==00:00:00:12.750 116560== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:31.312 INFO:tasks.ceph.osd.1.smithi059.stderr:2023-09-30T02:05:31.303+0000 40451c0 -1 Falling back to public interface 2023-09-30T02:05:31.429 INFO:tasks.ceph.osd.7.smithi110.stderr:==00:00:00:12.983 116561== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:31.465 INFO:tasks.ceph.osd.6.smithi110.stderr:==00:00:00:13.003 116563== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:31.473 INFO:tasks.ceph.osd.5.smithi110.stderr:==00:00:00:13.011 116562== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:31.490 INFO:tasks.ceph.osd.4.smithi110.stderr:==00:00:00:13.052 116560== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:31.567 INFO:tasks.ceph.osd.3.smithi059.stderr:2023-09-30T02:05:31.556+0000 40451c0 -1 Falling back to public interface 2023-09-30T02:05:31.681 INFO:tasks.ceph.osd.0.smithi059.stderr:==00:00:00:13.227 116664== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:31.716 INFO:tasks.ceph.osd.2.smithi059.stderr:==00:00:00:13.244 116666== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:31.722 INFO:tasks.ceph.osd.7.smithi110.stderr:==00:00:00:13.276 116561== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:31.762 INFO:tasks.ceph.osd.6.smithi110.stderr:==00:00:00:13.299 116563== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:31.768 INFO:tasks.ceph.osd.5.smithi110.stderr:==00:00:00:13.305 116562== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:31.785 INFO:tasks.ceph.osd.4.smithi110.stderr:==00:00:00:13.347 116560== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:32.013 INFO:tasks.ceph.osd.7.smithi110.stderr:==00:00:00:13.567 116561== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:32.044 INFO:tasks.ceph.osd.1.smithi059.stderr:==00:00:00:13.582 116665== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:32.059 INFO:tasks.ceph.osd.6.smithi110.stderr:==00:00:00:13.597 116563== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:32.064 INFO:tasks.ceph.osd.5.smithi110.stderr:==00:00:00:13.602 116562== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:32.066 INFO:tasks.ceph.osd.0.smithi059.stderr:==00:00:00:13.612 116664== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:32.088 INFO:tasks.ceph.osd.4.smithi110.stderr:==00:00:00:13.650 116560== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:32.100 INFO:tasks.ceph.osd.2.smithi059.stderr:==00:00:00:13.627 116666== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:32.256 INFO:tasks.ceph.osd.3.smithi059.stderr:==00:00:00:13.778 116667== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:32.278 INFO:tasks.ceph.osd.7.smithi110.stderr:==00:00:00:13.832 116561== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:32.326 INFO:tasks.ceph.osd.6.smithi110.stderr:==00:00:00:13.864 116563== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:32.331 INFO:tasks.ceph.osd.5.smithi110.stderr:==00:00:00:13.869 116562== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:32.361 INFO:tasks.ceph.osd.4.smithi110.stderr:==00:00:00:13.923 116560== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:32.395 INFO:tasks.ceph.osd.0.smithi059.stderr:==00:00:00:13.942 116664== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:32.421 INFO:tasks.ceph.osd.1.smithi059.stderr:==00:00:00:13.960 116665== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:32.429 INFO:tasks.ceph.osd.2.smithi059.stderr:==00:00:00:13.957 116666== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:32.546 INFO:tasks.ceph.osd.7.smithi110.stderr:==00:00:00:14.100 116561== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:32.597 INFO:tasks.ceph.osd.6.smithi110.stderr:==00:00:00:14.135 116563== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:32.601 INFO:tasks.ceph.osd.5.smithi110.stderr:==00:00:00:14.139 116562== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:32.628 INFO:tasks.ceph.osd.3.smithi059.stderr:==00:00:00:14.151 116667== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:32.661 INFO:tasks.ceph.osd.0.smithi059.stderr:==00:00:00:14.208 116664== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:32.695 INFO:tasks.ceph.osd.2.smithi059.stderr:==00:00:00:14.223 116666== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:32.730 INFO:tasks.ceph.osd.4.smithi110.stderr:==00:00:00:14.293 116560== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:32.753 INFO:tasks.ceph.osd.1.smithi059.stderr:==00:00:00:14.292 116665== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:32.797 INFO:tasks.ceph.osd.4.smithi110.stderr:==00:00:00:14.359 116560== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:32.874 INFO:tasks.ceph.osd.6.smithi110.stderr:==00:00:00:14.412 116563== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:32.918 INFO:tasks.ceph.osd.7.smithi110.stderr:==00:00:00:14.472 116561== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:32.931 INFO:tasks.ceph.osd.0.smithi059.stderr:==00:00:00:14.477 116664== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:32.969 INFO:tasks.ceph.osd.2.smithi059.stderr:==00:00:00:14.496 116666== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:32.970 INFO:tasks.ceph.osd.3.smithi059.stderr:==00:00:00:14.493 116667== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:32.975 INFO:tasks.ceph.osd.5.smithi110.stderr:==00:00:00:14.513 116562== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:32.988 INFO:tasks.ceph.osd.7.smithi110.stderr:==00:00:00:14.542 116561== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:33.039 INFO:tasks.ceph.osd.1.smithi059.stderr:==00:00:00:14.577 116665== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:33.044 INFO:tasks.ceph.osd.5.smithi110.stderr:==00:00:00:14.582 116562== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:33.219 INFO:tasks.ceph.osd.0.smithi059.stderr:==00:00:00:14.765 116664== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:33.259 INFO:tasks.ceph.osd.2.smithi059.stderr:==00:00:00:14.786 116666== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:33.265 INFO:tasks.ceph.osd.3.smithi059.stderr:==00:00:00:14.787 116667== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:33.279 INFO:tasks.ceph.osd.6.smithi110.stderr:==00:00:00:14.817 116563== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:33.336 INFO:tasks.ceph.osd.1.smithi059.stderr:==00:00:00:14.875 116665== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:33.360 INFO:tasks.ceph.osd.6.smithi110.stderr:==00:00:00:14.898 116563== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:33.527 INFO:tasks.ceph.osd.0.smithi059.stderr:==00:00:00:15.073 116664== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:33.568 INFO:tasks.ceph.osd.2.smithi059.stderr:==00:00:00:15.095 116666== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:33.572 INFO:tasks.ceph.osd.3.smithi059.stderr:==00:00:00:15.095 116667== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:33.631 INFO:tasks.ceph.osd.1.smithi059.stderr:==00:00:00:15.169 116665== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:33.821 INFO:tasks.ceph.osd.0.smithi059.stderr:==00:00:00:15.368 116664== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:33.870 INFO:tasks.ceph.osd.2.smithi059.stderr:==00:00:00:15.397 116666== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:33.875 INFO:tasks.ceph.osd.3.smithi059.stderr:==00:00:00:15.398 116667== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:33.930 INFO:tasks.ceph.osd.1.smithi059.stderr:==00:00:00:15.469 116665== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:34.116 INFO:tasks.ceph.osd.0.smithi059.stderr:==00:00:00:15.662 116664== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:34.169 INFO:tasks.ceph.osd.2.smithi059.stderr:==00:00:00:15.696 116666== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:34.172 INFO:tasks.ceph.osd.3.smithi059.stderr:==00:00:00:15.694 116667== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:34.231 INFO:tasks.ceph.osd.1.smithi059.stderr:==00:00:00:15.769 116665== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:34.441 INFO:tasks.ceph.osd.3.smithi059.stderr:==00:00:00:15.963 116667== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:34.487 INFO:tasks.ceph.osd.0.smithi059.stderr:==00:00:00:16.034 116664== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:34.502 INFO:tasks.ceph.osd.1.smithi059.stderr:==00:00:00:16.040 116665== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:34.544 INFO:tasks.ceph.osd.2.smithi059.stderr:==00:00:00:16.071 116666== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:34.560 INFO:tasks.ceph.osd.0.smithi059.stderr:==00:00:00:16.106 116664== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:34.618 INFO:tasks.ceph.osd.2.smithi059.stderr:==00:00:00:16.145 116666== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:34.711 INFO:tasks.ceph.osd.3.smithi059.stderr:==00:00:00:16.233 116667== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:34.874 INFO:tasks.ceph.osd.1.smithi059.stderr:==00:00:00:16.412 116665== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:34.948 INFO:tasks.ceph.osd.1.smithi059.stderr:==00:00:00:16.486 116665== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:35.084 INFO:tasks.ceph.osd.3.smithi059.stderr:==00:00:00:16.607 116667== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:35.170 INFO:tasks.ceph.osd.3.smithi059.stderr:==00:00:00:16.692 116667== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:35.740 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-09-30T02:05:35.912 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.911+0000 7f8170357700 10 monclient: get_monmap_and_config 2023-09-30T02:05:35.912 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.911+0000 7f8170357700 10 monclient: build_initial_monmap 2023-09-30T02:05:35.912 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.911+0000 7f8170357700 10 monclient: monmap: 2023-09-30T02:05:35.912 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2023-09-30T02:05:35.912 INFO:teuthology.misc.health.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:35.912 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2023-09-30T02:05:35.913127+0000 2023-09-30T02:05:35.912 INFO:teuthology.misc.health.smithi059.stderr:created 2023-09-30T02:05:35.913127+0000 2023-09-30T02:05:35.913 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:05:35.913 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 3 2023-09-30T02:05:35.913 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:05:35.913 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:05:35.913 INFO:teuthology.misc.health.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:05:35.913 INFO:teuthology.misc.health.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:05:35.913 INFO:teuthology.misc.health.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:05:35.913 INFO:teuthology.misc.health.smithi059.stderr: 2023-09-30T02:05:35.914 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.912+0000 7f8170357700 1 Processor -- start 2023-09-30T02:05:35.914 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.912+0000 7f8170357700 1 -- start start 2023-09-30T02:05:35.915 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.912+0000 7f8170357700 10 monclient: init 2023-09-30T02:05:35.915 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.913+0000 7f8170357700 10 monclient: _reopen_session rank -1 2023-09-30T02:05:35.915 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.913+0000 7f8170357700 10 monclient: _add_conns ranks=[3,1,2,4,0] 2023-09-30T02:05:35.915 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.913+0000 7f8170357700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f81680b1660 0x7f81680a54e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:35.915 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.913+0000 7f8170357700 10 monclient(hunting): picked mon.noname-a con 0x7f81680b1660 addr v2:172.21.15.59:3300/0 2023-09-30T02:05:35.915 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.913+0000 7f8170357700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f81680a7630 addr v1:172.21.15.110:6789/0 2023-09-30T02:05:35.915 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.913+0000 7f8170357700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f81680a8150 0x7f816813af50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:35.915 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.913+0000 7f8170357700 10 monclient(hunting): picked mon.noname-b con 0x7f81680a8150 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:05:35.915 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.913+0000 7f8170357700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81680a5e40 con 0x7f81680a7630 2023-09-30T02:05:35.915 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.913+0000 7f8170357700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:35.915 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.913+0000 7f8170357700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81680b1a30 con 0x7f81680b1660 2023-09-30T02:05:35.915 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.913+0000 7f8170357700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:35.916 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.913+0000 7f8170357700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81680a6f30 con 0x7f81680a8150 2023-09-30T02:05:35.916 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.913+0000 7f8170357700 10 monclient(hunting): _renew_subs 2023-09-30T02:05:35.916 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.913+0000 7f8170357700 10 monclient(hunting): authenticate will time out at 1781.216529s 2023-09-30T02:05:35.916 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.914+0000 7f816d8f2700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f81680a7630 0x7f81680a7a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.59:53748/0 (socket says 172.21.15.59:53748) 2023-09-30T02:05:35.916 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.914+0000 7f816d8f2700 1 -- 172.21.15.59:0/2656600122 learned_addr learned my addr 172.21.15.59:0/2656600122 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:05:35.916 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.914+0000 7f816e8f4700 1 --2- 172.21.15.59:0/2656600122 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f81680a8150 0x7f816813af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:35.916 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.914+0000 7f816e0f3700 1 --2- 172.21.15.59:0/2656600122 >> v2:172.21.15.59:3300/0 conn(0x7f81680b1660 0x7f81680a54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:35.917 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.915+0000 7f816e8f4700 10 monclient(hunting): get_auth_request con 0x7f81680a8150 auth_method 0 2023-09-30T02:05:35.917 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.915+0000 7f816e8f4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:35.917 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.915+0000 7f816e8f4700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:35.917 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.915+0000 7f816e8f4700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:35.917 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.915+0000 7f816e0f3700 10 monclient(hunting): get_auth_request con 0x7f81680b1660 auth_method 0 2023-09-30T02:05:35.917 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.915+0000 7f816e0f3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:35.917 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.915+0000 7f816e0f3700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:35.917 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.915+0000 7f816e0f3700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:35.918 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.916+0000 7f816e8f4700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:05:35.918 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.916+0000 7f816e8f4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:05:35.918 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.916+0000 7f816e8f4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:05:35.918 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.916+0000 7f816e0f3700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:05:35.918 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.916+0000 7f816e0f3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:05:35.918 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.916+0000 7f816e0f3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:05:35.918 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.917+0000 7f816d0f1700 1 -- 172.21.15.59:0/2656600122 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 397805498 0 0) 0x7f81680a5e40 con 0x7f81680a7630 2023-09-30T02:05:35.919 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.917+0000 7f816d0f1700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:35.919 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.917+0000 7f816d0f1700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:35.919 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.917+0000 7f816d0f1700 10 monclient(hunting): my global_id is 4204 2023-09-30T02:05:35.919 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.917+0000 7f816d0f1700 1 -- 172.21.15.59:0/2656600122 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8154003570 con 0x7f81680a7630 2023-09-30T02:05:35.919 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.919+0000 7f816e8f4700 10 monclient(hunting): handle_auth_done global_id 4175 payload 274 2023-09-30T02:05:35.920 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.919+0000 7f816e8f4700 1 -- 172.21.15.59:0/2656600122 >> v2:172.21.15.59:3300/0 conn(0x7f81680b1660 msgr2=0x7f81680a54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:05:35.920 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.919+0000 7f816e8f4700 1 --2- 172.21.15.59:0/2656600122 >> v2:172.21.15.59:3300/0 conn(0x7f81680b1660 0x7f81680a54e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:35.920 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.919+0000 7f816e8f4700 1 -- 172.21.15.59:0/2656600122 >> v1:172.21.15.110:6789/0 conn(0x7f81680a7630 legacy=0x7f81680a7a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:35.920 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.919+0000 7f816e8f4700 10 monclient: _finish_hunting 0 2023-09-30T02:05:35.920 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.919+0000 7f816e8f4700 1 monclient: found mon.noname-b 2023-09-30T02:05:35.920 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.919+0000 7f816e8f4700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:05:35.920 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.919+0000 7f816e8f4700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.59:3301/0 2023-09-30T02:05:35.920 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.919+0000 7f816e8f4700 1 -- 172.21.15.59:0/2656600122 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81680a5a20 con 0x7f81680a8150 2023-09-30T02:05:35.923 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.922+0000 7f816e8f4700 1 --2- 172.21.15.59:0/2656600122 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f81680a8150 0x7f816813af50 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8160009e60 tx=0x7f8160009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9544a0dd4429b9f server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:05:35.929 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.928+0000 7f816d0f1700 1 -- 172.21.15.59:0/2656600122 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f816000e070 con 0x7f81680a8150 2023-09-30T02:05:35.929 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.928+0000 7f816d0f1700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:05:35.929 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.928+0000 7f816d0f1700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:05:35.929 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.928+0000 7f816d0f1700 10 monclient: dump: 2023-09-30T02:05:35.929 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2023-09-30T02:05:35.929 INFO:teuthology.misc.health.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:35.929 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:35.929 INFO:teuthology.misc.health.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:35.930 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:05:35.930 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 3 2023-09-30T02:05:35.930 INFO:teuthology.misc.health.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:05:35.930 INFO:teuthology.misc.health.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:05:35.930 INFO:teuthology.misc.health.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:05:35.930 INFO:teuthology.misc.health.smithi059.stderr: 2023-09-30T02:05:35.930 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.928+0000 7f816d0f1700 10 monclient: _finish_auth 0 2023-09-30T02:05:35.930 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.928+0000 7f816d0f1700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:05:35.930 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.928+0000 7f816d0f1700 1 -- 172.21.15.59:0/2656600122 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f816000bab0 con 0x7f81680a8150 2023-09-30T02:05:35.930 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.928+0000 7f816d0f1700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:05:35.930 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.928+0000 7f816d0f1700 1 -- 172.21.15.59:0/2656600122 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f816000be70 con 0x7f81680a8150 2023-09-30T02:05:35.930 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.928+0000 7f816d0f1700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:05:35.931 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.928+0000 7f816d0f1700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:05:35.931 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.928+0000 7f816d0f1700 10 monclient: dump: 2023-09-30T02:05:35.931 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2023-09-30T02:05:35.931 INFO:teuthology.misc.health.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:35.931 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:35.931 INFO:teuthology.misc.health.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:35.931 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:05:35.931 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 3 2023-09-30T02:05:35.931 INFO:teuthology.misc.health.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:05:35.931 INFO:teuthology.misc.health.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:05:35.931 INFO:teuthology.misc.health.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:05:35.931 INFO:teuthology.misc.health.smithi059.stderr: 2023-09-30T02:05:35.932 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.929+0000 7f8170357700 5 monclient: authenticate success, global_id 4175 2023-09-30T02:05:35.932 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.929+0000 7f8170357700 10 monclient: get_monmap_and_config success 2023-09-30T02:05:35.932 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.929+0000 7f8170357700 4 set_mon_vals no callback set 2023-09-30T02:05:35.934 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.929+0000 7f8170357700 10 monclient: shutdown 2023-09-30T02:05:35.934 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.929+0000 7f8170357700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:05:35.934 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.929+0000 7f8170357700 1 -- 172.21.15.59:0/2656600122 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f81680a8150 msgr2=0x7f816813af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:35.934 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.929+0000 7f8170357700 1 --2- 172.21.15.59:0/2656600122 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f81680a8150 0x7f816813af50 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8160009e60 tx=0x7f8160009780 comp rx=0 tx=0).stop 2023-09-30T02:05:35.934 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.930+0000 7f8170357700 1 -- 172.21.15.59:0/2656600122 shutdown_connections 2023-09-30T02:05:35.934 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.930+0000 7f8170357700 1 --2- 172.21.15.59:0/2656600122 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f81680a8150 0x7f816813af50 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:35.934 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.930+0000 7f8170357700 1 --2- 172.21.15.59:0/2656600122 >> v2:172.21.15.59:3300/0 conn(0x7f81680b1660 0x7f81680a54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:35.935 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.930+0000 7f8170357700 1 -- 172.21.15.59:0/2656600122 >> 172.21.15.59:0/2656600122 conn(0x7f8168069130 msgr2=0x7f8168063280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:05:35.935 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.930+0000 7f8170357700 1 -- 172.21.15.59:0/2656600122 shutdown_connections 2023-09-30T02:05:35.935 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.930+0000 7f8170357700 1 -- 172.21.15.59:0/2656600122 wait complete. 2023-09-30T02:05:35.935 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.931+0000 7f8170357700 10 monclient: build_initial_monmap 2023-09-30T02:05:35.935 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.931+0000 7f8170357700 10 monclient: monmap: 2023-09-30T02:05:35.935 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2023-09-30T02:05:35.935 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:05:35.935 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2023-09-30T02:05:35.935 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2023-09-30T02:05:35.935 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:05:35.935 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2023-09-30T02:05:35.936 INFO:teuthology.misc.health.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:05:35.936 INFO:teuthology.misc.health.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:05:35.936 INFO:teuthology.misc.health.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:05:35.936 INFO:teuthology.misc.health.smithi059.stderr: 2023-09-30T02:05:35.936 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.931+0000 7f8170357700 1 librados: starting msgr at 2023-09-30T02:05:35.936 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.931+0000 7f8170357700 1 librados: starting objecter 2023-09-30T02:05:35.936 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.931+0000 7f8170357700 1 Processor -- start 2023-09-30T02:05:35.936 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.932+0000 7f8170357700 1 -- start start 2023-09-30T02:05:35.936 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.932+0000 7f8170357700 1 librados: setting wanted keys 2023-09-30T02:05:35.936 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.932+0000 7f8170357700 1 librados: calling monclient init 2023-09-30T02:05:35.936 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.932+0000 7f8170357700 10 monclient: init 2023-09-30T02:05:35.936 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.932+0000 7f8170357700 10 monclient: _reopen_session rank -1 2023-09-30T02:05:35.936 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.932+0000 7f8170357700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T02:05:35.937 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.932+0000 7f8170357700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f81680a7630 0x7f816813ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:35.937 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.932+0000 7f8170357700 10 monclient(hunting): picked mon.noname-a con 0x7f81680a7630 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:05:35.937 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.932+0000 7f8170357700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f81680a8150 0x7f8168140fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:35.937 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.932+0000 7f8170357700 10 monclient(hunting): picked mon.noname-b con 0x7f81680a8150 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:05:35.937 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.932+0000 7f8170357700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f81680b1660 0x7f81680a8d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:35.937 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.932+0000 7f8170357700 10 monclient(hunting): picked mon.noname-c con 0x7f81680b1660 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:05:35.937 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.932+0000 7f8170357700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:35.937 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.932+0000 7f8170357700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8168140900 con 0x7f81680a7630 2023-09-30T02:05:35.937 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.932+0000 7f8170357700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:35.937 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.932+0000 7f8170357700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8168140be0 con 0x7f81680a8150 2023-09-30T02:05:35.937 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.932+0000 7f8170357700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:35.938 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.932+0000 7f8170357700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8168142560 con 0x7f81680b1660 2023-09-30T02:05:35.938 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.932+0000 7f8170357700 10 monclient(hunting): _renew_subs 2023-09-30T02:05:35.938 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.932+0000 7f8170357700 10 monclient(hunting): authenticate will time out at 1781.235979s 2023-09-30T02:05:35.938 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.933+0000 7f816d8f2700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f81680a8150 0x7f8168140fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:35.938 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.933+0000 7f816e8f4700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f81680b1660 0x7f81680a8d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:35.938 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.933+0000 7f816d8f2700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f81680a8150 0x7f8168140fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:44608/0 (socket says 172.21.15.59:44608) 2023-09-30T02:05:35.938 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.933+0000 7f816d8f2700 1 -- 172.21.15.59:0/2276140937 learned_addr learned my addr 172.21.15.59:0/2276140937 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:05:35.938 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.933+0000 7f816d8f2700 10 monclient(hunting): get_auth_request con 0x7f81680a8150 auth_method 0 2023-09-30T02:05:35.938 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.933+0000 7f816d8f2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:35.938 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.933+0000 7f816d8f2700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:35.938 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.933+0000 7f816d8f2700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:35.939 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.934+0000 7f816e8f4700 10 monclient(hunting): get_auth_request con 0x7f81680b1660 auth_method 0 2023-09-30T02:05:35.939 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.934+0000 7f816e8f4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:35.939 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.934+0000 7f816e8f4700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:35.939 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.934+0000 7f816e8f4700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:35.939 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.934+0000 7f816e0f3700 1 --2- 172.21.15.59:0/2276140937 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f81680a7630 0x7f816813ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:35.940 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.934+0000 7f816d8f2700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:05:35.940 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.934+0000 7f816d8f2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:05:35.940 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.934+0000 7f816d8f2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:05:35.940 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.934+0000 7f816e0f3700 10 monclient(hunting): get_auth_request con 0x7f81680a7630 auth_method 0 2023-09-30T02:05:35.940 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.934+0000 7f816e0f3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:35.940 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.934+0000 7f816e0f3700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:35.940 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.934+0000 7f816e0f3700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:35.940 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.934+0000 7f816e8f4700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:05:35.940 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.934+0000 7f816e8f4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:05:35.940 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.934+0000 7f816e8f4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:05:35.940 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.935+0000 7f816e0f3700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:05:35.940 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.935+0000 7f816e0f3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:05:35.941 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.935+0000 7f816e0f3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:05:35.941 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.937+0000 7f816d8f2700 10 monclient(hunting): handle_auth_done global_id 4207 payload 824 2023-09-30T02:05:35.941 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.937+0000 7f816d8f2700 1 -- 172.21.15.59:0/2276140937 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f81680b1660 msgr2=0x7f81680a8d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:35.941 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.937+0000 7f816d8f2700 1 --2- 172.21.15.59:0/2276140937 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f81680b1660 0x7f81680a8d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:35.941 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.937+0000 7f816d8f2700 1 -- 172.21.15.59:0/2276140937 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f81680a7630 msgr2=0x7f816813ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:35.941 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.937+0000 7f816d8f2700 1 --2- 172.21.15.59:0/2276140937 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f81680a7630 0x7f816813ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:35.941 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.937+0000 7f816d8f2700 10 monclient: _finish_hunting 0 2023-09-30T02:05:35.941 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.937+0000 7f816d8f2700 1 monclient: found mon.noname-b 2023-09-30T02:05:35.941 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.937+0000 7f816d8f2700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:05:35.941 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.937+0000 7f816d8f2700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:05:35.941 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.937+0000 7f816d8f2700 1 -- 172.21.15.59:0/2276140937 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8160009a20 con 0x7f81680a8150 2023-09-30T02:05:35.942 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.940+0000 7f816d8f2700 1 --2- 172.21.15.59:0/2276140937 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f81680a8150 0x7f8168140fb0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f815800c7c0 tx=0x7f8158007e00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:05:35.942 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.940+0000 7f815e7fc700 10 client.?.objecter ms_handle_connect 0x7f81680a8150 2023-09-30T02:05:35.942 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.940+0000 7f815e7fc700 10 client.?.objecter resend_mon_ops 2023-09-30T02:05:35.945 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.944+0000 7f815e7fc700 1 -- 172.21.15.59:0/2276140937 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8158005cd0 con 0x7f81680a8150 2023-09-30T02:05:35.945 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.944+0000 7f815e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:05:35.945 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.944+0000 7f815e7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:05:35.945 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.944+0000 7f815e7fc700 10 monclient: dump: 2023-09-30T02:05:35.945 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2023-09-30T02:05:35.945 INFO:teuthology.misc.health.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:35.945 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:35.945 INFO:teuthology.misc.health.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:35.945 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:05:35.945 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 3 2023-09-30T02:05:35.946 INFO:teuthology.misc.health.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:05:35.946 INFO:teuthology.misc.health.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:05:35.946 INFO:teuthology.misc.health.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:05:35.946 INFO:teuthology.misc.health.smithi059.stderr: 2023-09-30T02:05:35.946 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.944+0000 7f815e7fc700 10 monclient: _finish_auth 0 2023-09-30T02:05:35.946 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.944+0000 7f815e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:05:35.946 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.944+0000 7f815e7fc700 1 -- 172.21.15.59:0/2276140937 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8158005e30 con 0x7f81680a8150 2023-09-30T02:05:35.946 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.944+0000 7f815e7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:05:35.946 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.944+0000 7f815e7fc700 1 -- 172.21.15.59:0/2276140937 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8158007320 con 0x7f81680a8150 2023-09-30T02:05:35.946 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.944+0000 7f815e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:05:35.946 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.944+0000 7f815e7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:05:35.946 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.944+0000 7f815e7fc700 10 monclient: dump: 2023-09-30T02:05:35.947 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2023-09-30T02:05:35.947 INFO:teuthology.misc.health.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:35.947 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:35.947 INFO:teuthology.misc.health.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:35.947 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:05:35.947 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 3 2023-09-30T02:05:35.947 INFO:teuthology.misc.health.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:05:35.947 INFO:teuthology.misc.health.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:05:35.947 INFO:teuthology.misc.health.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:05:35.947 INFO:teuthology.misc.health.smithi059.stderr: 2023-09-30T02:05:35.947 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.944+0000 7f8170357700 5 monclient: authenticate success, global_id 4207 2023-09-30T02:05:35.947 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.944+0000 7f8170357700 10 monclient: _renew_subs 2023-09-30T02:05:35.948 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.944+0000 7f8170357700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:05:35.948 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.944+0000 7f8170357700 1 -- 172.21.15.59:0/2276140937 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f816813f880 con 0x7f81680a8150 2023-09-30T02:05:35.948 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.944+0000 7f8170357700 10 client.4207.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:05:35.948 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.944+0000 7f8170357700 10 monclient: _renew_subs 2023-09-30T02:05:35.948 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.944+0000 7f8170357700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:05:35.948 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.944+0000 7f8170357700 1 -- 172.21.15.59:0/2276140937 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8168140680 con 0x7f81680a8150 2023-09-30T02:05:35.948 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.944+0000 7f8170357700 1 librados: init done 2023-09-30T02:05:35.948 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.944+0000 7f816c8f0700 4 set_mon_vals no callback set 2023-09-30T02:05:35.949 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.946+0000 7f8170357700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:05:35.949 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.946+0000 7f8170357700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:05:35.949 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.946+0000 7f8170357700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:05:35.949 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:35.946+0000 7f8170357700 1 -- 172.21.15.59:0/2276140937 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81500050b0 con 0x7f81680a8150 2023-09-30T02:05:36.116 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:36.115+0000 7f815e7fc700 1 -- 172.21.15.59:0/2276140937 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 4) v1 ==== 84649+0+0 (secure 0 0 0) 0x7f815801b3b0 con 0x7f81680a8150 2023-09-30T02:05:36.116 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:36.116+0000 7f815e7fc700 1 --2- 172.21.15.59:0/2276140937 >> v2:172.21.15.110:6800/115870 conn(0x7f814c068800 0x7f814c06acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:36.116 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:36.116+0000 7f816e0f3700 1 --2- 172.21.15.59:0/2276140937 >> v2:172.21.15.110:6800/115870 conn(0x7f814c068800 0x7f814c06acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:36.117 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:36.116+0000 7f816e0f3700 10 monclient: get_auth_request con 0x7f814c068800 auth_method 0 2023-09-30T02:05:36.117 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:36.117+0000 7f816e0f3700 1 --2- 172.21.15.59:0/2276140937 >> v2:172.21.15.110:6800/115870 conn(0x7f814c068800 0x7f814c06acb0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f8164009d10 tx=0x7f81640091a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:05:36.117 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:36.117+0000 7f815e7fc700 10 client.4207.objecter ms_handle_connect 0x7f814c068800 2023-09-30T02:05:36.127 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:36.127+0000 7f815e7fc700 1 -- 172.21.15.59:0/2276140937 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1641+0+0 (secure 0 0 0) 0x7f8158055600 con 0x7f81680a8150 2023-09-30T02:05:36.127 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:36.127+0000 7f815e7fc700 10 client.4207.objecter ms_dispatch 0x7f8168001550 osd_map(10..10 src has 1..10) v4 2023-09-30T02:05:36.127 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:36.127+0000 7f815e7fc700 3 client.4207.objecter handle_osd_map got epochs [10,10] > 0 2023-09-30T02:05:36.128 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:36.127+0000 7f815e7fc700 3 client.4207.objecter handle_osd_map decoding full epoch 10 2023-09-30T02:05:36.128 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:36.127+0000 7f815e7fc700 20 client.4207.objecter dump_active .. 0 homeless 2023-09-30T02:05:36.222 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:36.221+0000 7f815e7fc700 1 -- 172.21.15.59:0/2276140937 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8158051240 con 0x7f81680a8150 2023-09-30T02:05:36.222 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:36.221+0000 7f815e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:05:36.222 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:36.221+0000 7f815e7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:05:36.429 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:36.428+0000 7f8170357700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-09-30T02:05:36.430 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:36.428+0000 7f8170357700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T02:05:36.430 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:36.428+0000 7f8170357700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:05:36.430 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:36.428+0000 7f8170357700 1 -- 172.21.15.59:0/2276140937 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8150005630 con 0x7f81680a8150 2023-09-30T02:05:36.460 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:36.459+0000 7f815e7fc700 1 -- 172.21.15.59:0/2276140937 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7013 (secure 0 0 0) 0x7f8158017020 con 0x7f81680a8150 2023-09-30T02:05:36.460 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:36.459+0000 7f815e7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T02:05:36.460 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:36.459+0000 7f815e7fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:05:36.460 INFO:teuthology.misc.health.smithi059.stdout: 2023-09-30T02:05:36.461 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"623052cb-4f82-4b2b-b72b-3ed5be614613","created":"2023-09-30T02:05:11.679291+0000","modified":"2023-09-30T02:05:18.270595+0000","last_up_change":"0.000000","last_in_change":"2023-09-30T02:05:18.270595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"efdab44b-be57-4b81-8567-ff2deec1d3e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff9786d-8c39-4044-91d2-a31f33e3b16e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b276ed4b-317b-4ddf-96ab-7ad71b8e3ff7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"48bfaf43-5846-4385-b9c6-6bc424cd5acd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"76fa5443-1571-4e72-9d22-6ab19e59839a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e9436478-4140-4fd6-ba2e-04fa6f6cf367","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"76676042-2074-4cf8-9ef9-5c87bf604944","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"045479f4-f7f7-4b95-bc38-a2c52ac31b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T02:05:36.463 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:36.462+0000 7f8170357700 10 librados: watch_flush enter 2023-09-30T02:05:36.463 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:36.462+0000 7f8170357700 10 librados: watch_flush exit 2023-09-30T02:05:36.463 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:36.462+0000 7f8170357700 20 client.4207.objecter shutdown clearing up homeless session... 2023-09-30T02:05:36.463 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:36.462+0000 7f8170357700 10 client.4207.objecter successfully canceled tick 2023-09-30T02:05:36.464 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:36.462+0000 7f8170357700 1 -- 172.21.15.59:0/2276140937 >> v2:172.21.15.110:6800/115870 conn(0x7f814c068800 msgr2=0x7f814c06acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:36.464 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:36.462+0000 7f8170357700 1 --2- 172.21.15.59:0/2276140937 >> v2:172.21.15.110:6800/115870 conn(0x7f814c068800 0x7f814c06acb0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f8164009d10 tx=0x7f81640091a0 comp rx=0 tx=0).stop 2023-09-30T02:05:36.464 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:36.462+0000 7f8170357700 10 monclient: shutdown 2023-09-30T02:05:36.464 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:36.462+0000 7f8170357700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:05:36.464 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:36.462+0000 7f8170357700 1 -- 172.21.15.59:0/2276140937 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f81680a8150 msgr2=0x7f8168140fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:36.464 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:36.462+0000 7f8170357700 1 --2- 172.21.15.59:0/2276140937 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f81680a8150 0x7f8168140fb0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f815800c7c0 tx=0x7f8158007e00 comp rx=0 tx=0).stop 2023-09-30T02:05:36.464 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:36.463+0000 7f8170357700 1 -- 172.21.15.59:0/2276140937 shutdown_connections 2023-09-30T02:05:36.464 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:36.463+0000 7f8170357700 1 --2- 172.21.15.59:0/2276140937 >> v2:172.21.15.110:6800/115870 conn(0x7f814c068800 0x7f814c06acb0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:36.464 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:36.463+0000 7f8170357700 1 --2- 172.21.15.59:0/2276140937 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f81680b1660 0x7f81680a8d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:36.464 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:36.463+0000 7f8170357700 1 --2- 172.21.15.59:0/2276140937 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f81680a7630 0x7f816813ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:36.464 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:36.463+0000 7f8170357700 1 --2- 172.21.15.59:0/2276140937 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f81680a8150 0x7f8168140fb0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:36.464 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:36.463+0000 7f8170357700 1 -- 172.21.15.59:0/2276140937 >> 172.21.15.59:0/2276140937 conn(0x7f8168069130 msgr2=0x7f81680b5180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:05:36.465 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:36.463+0000 7f8170357700 1 -- 172.21.15.59:0/2276140937 shutdown_connections 2023-09-30T02:05:36.465 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:36.463+0000 7f8170357700 1 -- 172.21.15.59:0/2276140937 wait complete. 2023-09-30T02:05:36.465 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:36.463+0000 7f8170357700 1 librados: shutdown 2023-09-30T02:05:36.476 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-09-30T02:05:39.155 INFO:tasks.ceph.osd.5.smithi110.stderr:==00:00:00:20.693 116562== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:39.310 INFO:tasks.ceph.osd.6.smithi110.stderr:==00:00:00:20.848 116563== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:39.350 INFO:tasks.ceph.osd.4.smithi110.stderr:==00:00:00:20.912 116560== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:39.390 INFO:tasks.ceph.osd.7.smithi110.stderr:==00:00:00:20.944 116561== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:39.662 INFO:tasks.ceph.mon.a.smithi059.stderr:2023-09-30T02:05:39.658+0000 13c1c700 -1 log_channel(cluster) log [ERR] : Health check failed: Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale' (MGR_MODULE_ERROR) 2023-09-30T02:05:41.110 INFO:tasks.ceph.osd.0.smithi059.stderr:==00:00:00:22.657 116664== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:41.172 INFO:tasks.ceph.osd.2.smithi059.stderr:==00:00:00:22.699 116666== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:41.535 INFO:tasks.ceph.osd.1.smithi059.stderr:==00:00:00:23.074 116665== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:42.015 INFO:tasks.ceph.osd.3.smithi059.stderr:==00:00:00:23.537 116667== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:43.282 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-09-30T02:05:43.454 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.454+0000 7f369bae8700 10 monclient: get_monmap_and_config 2023-09-30T02:05:43.454 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.454+0000 7f369bae8700 10 monclient: build_initial_monmap 2023-09-30T02:05:43.455 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.454+0000 7f369bae8700 10 monclient: monmap: 2023-09-30T02:05:43.455 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2023-09-30T02:05:43.455 INFO:teuthology.misc.health.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:43.455 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2023-09-30T02:05:43.455542+0000 2023-09-30T02:05:43.455 INFO:teuthology.misc.health.smithi059.stderr:created 2023-09-30T02:05:43.455542+0000 2023-09-30T02:05:43.455 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:05:43.455 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 3 2023-09-30T02:05:43.455 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:05:43.455 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:05:43.456 INFO:teuthology.misc.health.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:05:43.456 INFO:teuthology.misc.health.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:05:43.456 INFO:teuthology.misc.health.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:05:43.456 INFO:teuthology.misc.health.smithi059.stderr: 2023-09-30T02:05:43.457 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.454+0000 7f369bae8700 1 Processor -- start 2023-09-30T02:05:43.457 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.455+0000 7f369bae8700 1 -- start start 2023-09-30T02:05:43.457 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.455+0000 7f369bae8700 10 monclient: init 2023-09-30T02:05:43.457 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.456+0000 7f369bae8700 10 monclient: _reopen_session rank -1 2023-09-30T02:05:43.457 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.456+0000 7f369bae8700 10 monclient: _add_conns ranks=[2,3,0,4,1] 2023-09-30T02:05:43.458 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.456+0000 7f369bae8700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f36940b1660 0x7f36940a54e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:43.458 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.456+0000 7f369bae8700 10 monclient(hunting): picked mon.noname-b con 0x7f36940b1660 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:05:43.458 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.456+0000 7f369bae8700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f36940a7630 0x7f36940a7a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:43.458 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.456+0000 7f369bae8700 10 monclient(hunting): picked mon.noname-a con 0x7f36940a7630 addr v2:172.21.15.59:3300/0 2023-09-30T02:05:43.458 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.456+0000 7f369bae8700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f36940a7f80 addr v1:172.21.15.59:6789/0 2023-09-30T02:05:43.458 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.456+0000 7f369bae8700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36940a5e40 con 0x7f36940a7f80 2023-09-30T02:05:43.458 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.456+0000 7f369bae8700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:43.458 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.456+0000 7f369bae8700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36940b1a30 con 0x7f36940a7630 2023-09-30T02:05:43.458 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.456+0000 7f369bae8700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:43.458 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.456+0000 7f369bae8700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36940a83e0 con 0x7f36940b1660 2023-09-30T02:05:43.459 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.456+0000 7f369bae8700 10 monclient(hunting): _renew_subs 2023-09-30T02:05:43.459 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.456+0000 7f369bae8700 10 monclient(hunting): authenticate will time out at 1788.759326s 2023-09-30T02:05:43.459 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.457+0000 7f3699884700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f36940b1660 0x7f36940a54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:43.459 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.457+0000 7f3699083700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f36940a7630 0x7f36940a7a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:43.460 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.457+0000 7f3699884700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f36940b1660 0x7f36940a54e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:49548/0 (socket says 172.21.15.59:49548) 2023-09-30T02:05:43.460 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.457+0000 7f3699884700 1 -- 172.21.15.59:0/1576374053 learned_addr learned my addr 172.21.15.59:0/1576374053 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:05:43.460 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.457+0000 7f3699884700 10 monclient(hunting): get_auth_request con 0x7f36940b1660 auth_method 0 2023-09-30T02:05:43.460 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.457+0000 7f3699884700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:43.460 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.457+0000 7f3699884700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:43.460 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.457+0000 7f3699884700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:43.460 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.457+0000 7f3699083700 10 monclient(hunting): get_auth_request con 0x7f36940a7630 auth_method 0 2023-09-30T02:05:43.460 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.457+0000 7f3699083700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:43.460 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.457+0000 7f3699083700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:43.461 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.457+0000 7f3699083700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:43.461 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.458+0000 7f3699884700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:05:43.461 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.458+0000 7f3699884700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:05:43.461 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.458+0000 7f3699884700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:05:43.461 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.459+0000 7f3699083700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:05:43.461 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.459+0000 7f3699083700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:05:43.461 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.459+0000 7f3699083700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:05:43.462 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.460+0000 7f3699884700 10 monclient(hunting): handle_auth_done global_id 4181 payload 274 2023-09-30T02:05:43.462 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.460+0000 7f3699884700 1 -- 172.21.15.59:0/1576374053 >> v2:172.21.15.59:3300/0 conn(0x7f36940a7630 msgr2=0x7f36940a7a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:05:43.462 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.460+0000 7f3699884700 1 --2- 172.21.15.59:0/1576374053 >> v2:172.21.15.59:3300/0 conn(0x7f36940a7630 0x7f36940a7a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:43.462 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.460+0000 7f3699884700 1 -- 172.21.15.59:0/1576374053 >> v1:172.21.15.59:6789/0 conn(0x7f36940a7f80 legacy=0x7f36940af430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:43.462 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.460+0000 7f3699884700 10 monclient: _finish_hunting 0 2023-09-30T02:05:43.462 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.460+0000 7f3699884700 1 monclient: found mon.noname-b 2023-09-30T02:05:43.462 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.460+0000 7f3699884700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:05:43.462 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.460+0000 7f3699884700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.59:3301/0 2023-09-30T02:05:43.462 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.460+0000 7f3699884700 1 -- 172.21.15.59:0/1576374053 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36940a6f30 con 0x7f36940b1660 2023-09-30T02:05:43.463 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.462+0000 7f3699884700 1 --2- 172.21.15.59:0/1576374053 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f36940b1660 0x7f36940a54e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f36900099b0 tx=0x7f3690009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f2c610e587f2d61 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:05:43.467 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.467+0000 7f3698882700 1 -- 172.21.15.59:0/1576374053 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3690009ea0 con 0x7f36940b1660 2023-09-30T02:05:43.467 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.467+0000 7f3698882700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:05:43.467 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.467+0000 7f3698882700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:05:43.467 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.467+0000 7f3698882700 10 monclient: dump: 2023-09-30T02:05:43.467 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2023-09-30T02:05:43.467 INFO:teuthology.misc.health.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:43.468 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:43.468 INFO:teuthology.misc.health.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:43.468 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:05:43.468 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 3 2023-09-30T02:05:43.468 INFO:teuthology.misc.health.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:05:43.468 INFO:teuthology.misc.health.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:05:43.468 INFO:teuthology.misc.health.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:05:43.468 INFO:teuthology.misc.health.smithi059.stderr: 2023-09-30T02:05:43.469 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.467+0000 7f3698882700 10 monclient: _finish_auth 0 2023-09-30T02:05:43.469 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.467+0000 7f3698882700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:05:43.469 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.467+0000 7f3698882700 1 -- 172.21.15.59:0/1576374053 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f369000b860 con 0x7f36940b1660 2023-09-30T02:05:43.469 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.467+0000 7f3698882700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:05:43.469 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.467+0000 7f3698882700 1 -- 172.21.15.59:0/1576374053 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f369000bc00 con 0x7f36940b1660 2023-09-30T02:05:43.469 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.467+0000 7f3698882700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:05:43.469 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.467+0000 7f3698882700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:05:43.469 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.467+0000 7f3698882700 10 monclient: dump: 2023-09-30T02:05:43.469 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2023-09-30T02:05:43.469 INFO:teuthology.misc.health.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:43.470 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:43.470 INFO:teuthology.misc.health.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:43.470 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:05:43.470 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 3 2023-09-30T02:05:43.470 INFO:teuthology.misc.health.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:05:43.470 INFO:teuthology.misc.health.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:05:43.470 INFO:teuthology.misc.health.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:05:43.470 INFO:teuthology.misc.health.smithi059.stderr: 2023-09-30T02:05:43.471 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.467+0000 7f369bae8700 5 monclient: authenticate success, global_id 4181 2023-09-30T02:05:43.471 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.467+0000 7f369bae8700 10 monclient: get_monmap_and_config success 2023-09-30T02:05:43.471 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.467+0000 7f369bae8700 4 set_mon_vals no callback set 2023-09-30T02:05:43.471 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.467+0000 7f369bae8700 10 monclient: shutdown 2023-09-30T02:05:43.471 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.467+0000 7f369bae8700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:05:43.472 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.467+0000 7f369bae8700 1 -- 172.21.15.59:0/1576374053 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f36940b1660 msgr2=0x7f36940a54e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:43.472 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.467+0000 7f369bae8700 1 --2- 172.21.15.59:0/1576374053 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f36940b1660 0x7f36940a54e0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f36900099b0 tx=0x7f3690009250 comp rx=0 tx=0).stop 2023-09-30T02:05:43.472 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.468+0000 7f369bae8700 1 -- 172.21.15.59:0/1576374053 shutdown_connections 2023-09-30T02:05:43.472 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.468+0000 7f369bae8700 1 --2- 172.21.15.59:0/1576374053 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f36940b1660 0x7f36940a54e0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:43.472 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.468+0000 7f369bae8700 1 --2- 172.21.15.59:0/1576374053 >> v2:172.21.15.59:3300/0 conn(0x7f36940a7630 0x7f36940a7a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:43.472 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.468+0000 7f369bae8700 1 -- 172.21.15.59:0/1576374053 >> 172.21.15.59:0/1576374053 conn(0x7f3694069130 msgr2=0x7f3694063280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:05:43.472 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.469+0000 7f369bae8700 1 -- 172.21.15.59:0/1576374053 shutdown_connections 2023-09-30T02:05:43.472 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.470+0000 7f369bae8700 1 -- 172.21.15.59:0/1576374053 wait complete. 2023-09-30T02:05:43.473 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.471+0000 7f369bae8700 10 monclient: build_initial_monmap 2023-09-30T02:05:43.473 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.471+0000 7f369bae8700 10 monclient: monmap: 2023-09-30T02:05:43.473 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2023-09-30T02:05:43.473 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:05:43.473 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2023-09-30T02:05:43.473 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2023-09-30T02:05:43.473 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:05:43.473 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2023-09-30T02:05:43.473 INFO:teuthology.misc.health.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:05:43.473 INFO:teuthology.misc.health.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:05:43.473 INFO:teuthology.misc.health.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:05:43.474 INFO:teuthology.misc.health.smithi059.stderr: 2023-09-30T02:05:43.474 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.472+0000 7f369bae8700 1 librados: starting msgr at 2023-09-30T02:05:43.474 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.472+0000 7f369bae8700 1 librados: starting objecter 2023-09-30T02:05:43.474 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.472+0000 7f369bae8700 1 Processor -- start 2023-09-30T02:05:43.475 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.472+0000 7f369bae8700 1 -- start start 2023-09-30T02:05:43.475 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.472+0000 7f369bae8700 1 librados: setting wanted keys 2023-09-30T02:05:43.475 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.472+0000 7f369bae8700 1 librados: calling monclient init 2023-09-30T02:05:43.475 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.472+0000 7f369bae8700 10 monclient: init 2023-09-30T02:05:43.475 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.473+0000 7f369bae8700 10 monclient: _reopen_session rank -1 2023-09-30T02:05:43.475 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.473+0000 7f369bae8700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T02:05:43.475 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.473+0000 7f369bae8700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f36940a7630 0x7f369413e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:43.475 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.473+0000 7f369bae8700 10 monclient(hunting): picked mon.noname-b con 0x7f36940a7630 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:05:43.475 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.473+0000 7f369bae8700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f36940a7f80 0x7f3694139170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:43.475 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.473+0000 7f369bae8700 10 monclient(hunting): picked mon.noname-c con 0x7f36940a7f80 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:05:43.476 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.473+0000 7f369bae8700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36940b1660 0x7f3694155c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:43.476 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.473+0000 7f369bae8700 10 monclient(hunting): picked mon.noname-a con 0x7f36940b1660 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:05:43.476 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.473+0000 7f369bae8700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:43.476 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.473+0000 7f369bae8700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f369413d760 con 0x7f36940b1660 2023-09-30T02:05:43.476 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.473+0000 7f369bae8700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:43.476 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.473+0000 7f369bae8700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3694155800 con 0x7f36940a7630 2023-09-30T02:05:43.476 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.473+0000 7f369bae8700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:43.476 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.473+0000 7f369bae8700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f369413dad0 con 0x7f36940a7f80 2023-09-30T02:05:43.476 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.473+0000 7f369bae8700 10 monclient(hunting): _renew_subs 2023-09-30T02:05:43.476 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.473+0000 7f369bae8700 10 monclient(hunting): authenticate will time out at 1788.776280s 2023-09-30T02:05:43.476 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.473+0000 7f3699083700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f36940a7f80 0x7f3694139170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:43.476 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.473+0000 7f369a085700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36940b1660 0x7f3694155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:43.477 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.474+0000 7f3699083700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f36940a7f80 0x7f3694139170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:49564/0 (socket says 172.21.15.59:49564) 2023-09-30T02:05:43.477 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.474+0000 7f3699083700 1 -- 172.21.15.59:0/1895960825 learned_addr learned my addr 172.21.15.59:0/1895960825 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:05:43.477 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.474+0000 7f3699083700 10 monclient(hunting): get_auth_request con 0x7f36940a7f80 auth_method 0 2023-09-30T02:05:43.477 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.474+0000 7f3699083700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:43.477 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.474+0000 7f3699083700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:43.477 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.474+0000 7f3699083700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:43.477 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.474+0000 7f369a085700 10 monclient(hunting): get_auth_request con 0x7f36940b1660 auth_method 0 2023-09-30T02:05:43.477 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.474+0000 7f369a085700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:43.477 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.474+0000 7f369a085700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:43.477 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.474+0000 7f369a085700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:43.477 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.475+0000 7f3699884700 1 --2- 172.21.15.59:0/1895960825 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f36940a7630 0x7f369413e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:43.478 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.475+0000 7f3699083700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:05:43.478 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.475+0000 7f3699083700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:05:43.478 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.475+0000 7f3699083700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:05:43.478 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.475+0000 7f369a085700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:05:43.478 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.475+0000 7f369a085700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:05:43.478 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.475+0000 7f369a085700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:05:43.478 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.475+0000 7f3699884700 10 monclient(hunting): get_auth_request con 0x7f36940a7630 auth_method 0 2023-09-30T02:05:43.478 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.475+0000 7f3699884700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:43.479 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.475+0000 7f3699884700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:43.479 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.475+0000 7f3699884700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:43.479 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.476+0000 7f3699884700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:05:43.479 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.476+0000 7f3699884700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:05:43.479 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.476+0000 7f3699884700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:05:43.480 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.480+0000 7f3699083700 10 monclient(hunting): handle_auth_done global_id 4184 payload 824 2023-09-30T02:05:43.480 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.480+0000 7f3699083700 1 -- 172.21.15.59:0/1895960825 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f36940a7630 msgr2=0x7f369413e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:43.481 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.480+0000 7f3699083700 1 --2- 172.21.15.59:0/1895960825 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f36940a7630 0x7f369413e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:43.481 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.480+0000 7f3699083700 1 -- 172.21.15.59:0/1895960825 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36940b1660 msgr2=0x7f3694155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:43.481 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.480+0000 7f3699083700 1 --2- 172.21.15.59:0/1895960825 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36940b1660 0x7f3694155c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:43.481 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.480+0000 7f3699083700 10 monclient: _finish_hunting 0 2023-09-30T02:05:43.481 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.480+0000 7f3699083700 1 monclient: found mon.noname-c 2023-09-30T02:05:43.481 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.480+0000 7f3699083700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:05:43.481 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.480+0000 7f3699083700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.59:3301/0 2023-09-30T02:05:43.481 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.480+0000 7f3699083700 1 -- 172.21.15.59:0/1895960825 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36900094f0 con 0x7f36940a7f80 2023-09-30T02:05:43.484 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.484+0000 7f3699083700 1 --2- 172.21.15.59:0/1895960825 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f36940a7f80 0x7f3694139170 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f368400ecb0 tx=0x7f36840091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:05:43.484 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.484+0000 7f3689ffb700 10 client.?.objecter ms_handle_connect 0x7f36940a7f80 2023-09-30T02:05:43.484 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.484+0000 7f3689ffb700 10 client.?.objecter resend_mon_ops 2023-09-30T02:05:43.487 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.486+0000 7f3689ffb700 1 -- 172.21.15.59:0/1895960825 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f368400c850 con 0x7f36940a7f80 2023-09-30T02:05:43.487 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.486+0000 7f3689ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:05:43.487 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.487+0000 7f3689ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:05:43.487 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.487+0000 7f3689ffb700 10 monclient: dump: 2023-09-30T02:05:43.487 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2023-09-30T02:05:43.487 INFO:teuthology.misc.health.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:43.487 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:43.487 INFO:teuthology.misc.health.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:43.488 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:05:43.488 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 3 2023-09-30T02:05:43.488 INFO:teuthology.misc.health.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:05:43.488 INFO:teuthology.misc.health.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:05:43.488 INFO:teuthology.misc.health.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:05:43.488 INFO:teuthology.misc.health.smithi059.stderr: 2023-09-30T02:05:43.488 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.487+0000 7f3689ffb700 10 monclient: _finish_auth 0 2023-09-30T02:05:43.488 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.487+0000 7f3689ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:05:43.489 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.487+0000 7f369bae8700 5 monclient: authenticate success, global_id 4184 2023-09-30T02:05:43.489 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.487+0000 7f369bae8700 10 monclient: _renew_subs 2023-09-30T02:05:43.489 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.487+0000 7f369bae8700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:05:43.489 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.487+0000 7f369bae8700 1 -- 172.21.15.59:0/1895960825 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36941396b0 con 0x7f36940a7f80 2023-09-30T02:05:43.489 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.487+0000 7f369bae8700 10 client.4184.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:05:43.489 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.487+0000 7f369bae8700 10 monclient: _renew_subs 2023-09-30T02:05:43.489 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.487+0000 7f369bae8700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:05:43.489 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.487+0000 7f369bae8700 1 -- 172.21.15.59:0/1895960825 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3694156410 con 0x7f36940a7f80 2023-09-30T02:05:43.489 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.487+0000 7f369bae8700 1 librados: init done 2023-09-30T02:05:43.490 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.490+0000 7f369bae8700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:05:43.490 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.490+0000 7f369bae8700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:05:43.490 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.490+0000 7f369bae8700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:05:43.490 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.490+0000 7f369bae8700 1 -- 172.21.15.59:0/1895960825 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3694064a00 con 0x7f36940a7f80 2023-09-30T02:05:43.491 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.490+0000 7f3689ffb700 1 -- 172.21.15.59:0/1895960825 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f368400cbc0 con 0x7f36940a7f80 2023-09-30T02:05:43.491 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.490+0000 7f3689ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:05:43.491 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.490+0000 7f368bfff700 4 set_mon_vals no callback set 2023-09-30T02:05:43.496 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.496+0000 7f3689ffb700 1 -- 172.21.15.59:0/1895960825 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f368400cd30 con 0x7f36940a7f80 2023-09-30T02:05:43.496 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.496+0000 7f3689ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:05:43.496 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.496+0000 7f3689ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:05:43.496 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.496+0000 7f3689ffb700 10 monclient: dump: 2023-09-30T02:05:43.496 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2023-09-30T02:05:43.496 INFO:teuthology.misc.health.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:43.496 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:43.497 INFO:teuthology.misc.health.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:43.497 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:05:43.497 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 3 2023-09-30T02:05:43.497 INFO:teuthology.misc.health.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:05:43.497 INFO:teuthology.misc.health.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:05:43.497 INFO:teuthology.misc.health.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:05:43.497 INFO:teuthology.misc.health.smithi059.stderr: 2023-09-30T02:05:43.640 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.639+0000 7f3689ffb700 1 -- 172.21.15.59:0/1895960825 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 4) v1 ==== 84649+0+0 (secure 0 0 0) 0x7f3684005c10 con 0x7f36940a7f80 2023-09-30T02:05:43.641 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.640+0000 7f3689ffb700 1 --2- 172.21.15.59:0/1895960825 >> v2:172.21.15.110:6800/115870 conn(0x7f3678068800 0x7f367806acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:43.641 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.640+0000 7f3689ffb700 1 -- 172.21.15.59:0/1895960825 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1641+0+0 (secure 0 0 0) 0x7f3684083af0 con 0x7f36940a7f80 2023-09-30T02:05:43.641 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.640+0000 7f3689ffb700 10 client.4184.objecter ms_dispatch 0x7f3694001550 osd_map(10..10 src has 1..10) v4 2023-09-30T02:05:43.641 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.640+0000 7f3689ffb700 3 client.4184.objecter handle_osd_map got epochs [10,10] > 0 2023-09-30T02:05:43.641 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.640+0000 7f3689ffb700 3 client.4184.objecter handle_osd_map decoding full epoch 10 2023-09-30T02:05:43.641 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.640+0000 7f3689ffb700 20 client.4184.objecter dump_active .. 0 homeless 2023-09-30T02:05:43.641 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.640+0000 7f3699884700 1 --2- 172.21.15.59:0/1895960825 >> v2:172.21.15.110:6800/115870 conn(0x7f3678068800 0x7f367806acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:43.641 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.640+0000 7f3699884700 10 monclient: get_auth_request con 0x7f3678068800 auth_method 0 2023-09-30T02:05:43.642 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.641+0000 7f3689ffb700 10 client.4184.objecter ms_handle_connect 0x7f3678068800 2023-09-30T02:05:43.642 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.641+0000 7f3699884700 1 --2- 172.21.15.59:0/1895960825 >> v2:172.21.15.110:6800/115870 conn(0x7f3678068800 0x7f367806acb0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f36900099b0 tx=0x7f3690017000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:05:43.864 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.863+0000 7f3689ffb700 1 -- 172.21.15.59:0/1895960825 <== mon.2 v2:172.21.15.59:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3684051d80 con 0x7f36940a7f80 2023-09-30T02:05:43.864 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.863+0000 7f3689ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:05:43.864 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:43.863+0000 7f3689ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:05:44.207 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:44.207+0000 7f369bae8700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-09-30T02:05:44.207 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:44.207+0000 7f369bae8700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T02:05:44.207 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:44.207+0000 7f369bae8700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:05:44.207 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:44.207+0000 7f369bae8700 1 -- 172.21.15.59:0/1895960825 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f36940a5a20 con 0x7f36940a7f80 2023-09-30T02:05:44.236 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:44.236+0000 7f3689ffb700 1 -- 172.21.15.59:0/1895960825 <== mon.2 v2:172.21.15.59:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7013 (secure 0 0 0) 0x7f3684021090 con 0x7f36940a7f80 2023-09-30T02:05:44.236 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:44.236+0000 7f3689ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T02:05:44.236 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:44.236+0000 7f3689ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:05:44.237 INFO:teuthology.misc.health.smithi059.stdout: 2023-09-30T02:05:44.237 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"623052cb-4f82-4b2b-b72b-3ed5be614613","created":"2023-09-30T02:05:11.679291+0000","modified":"2023-09-30T02:05:18.270595+0000","last_up_change":"0.000000","last_in_change":"2023-09-30T02:05:18.270595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"efdab44b-be57-4b81-8567-ff2deec1d3e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff9786d-8c39-4044-91d2-a31f33e3b16e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b276ed4b-317b-4ddf-96ab-7ad71b8e3ff7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"48bfaf43-5846-4385-b9c6-6bc424cd5acd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"76fa5443-1571-4e72-9d22-6ab19e59839a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e9436478-4140-4fd6-ba2e-04fa6f6cf367","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"76676042-2074-4cf8-9ef9-5c87bf604944","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"045479f4-f7f7-4b95-bc38-a2c52ac31b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T02:05:44.239 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:44.239+0000 7f369bae8700 10 librados: watch_flush enter 2023-09-30T02:05:44.239 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:44.239+0000 7f369bae8700 10 librados: watch_flush exit 2023-09-30T02:05:44.239 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:44.239+0000 7f369bae8700 20 client.4184.objecter shutdown clearing up homeless session... 2023-09-30T02:05:44.240 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:44.239+0000 7f369bae8700 10 client.4184.objecter successfully canceled tick 2023-09-30T02:05:44.240 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:44.239+0000 7f369bae8700 1 -- 172.21.15.59:0/1895960825 >> v2:172.21.15.110:6800/115870 conn(0x7f3678068800 msgr2=0x7f367806acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:44.240 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:44.239+0000 7f369bae8700 1 --2- 172.21.15.59:0/1895960825 >> v2:172.21.15.110:6800/115870 conn(0x7f3678068800 0x7f367806acb0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f36900099b0 tx=0x7f3690017000 comp rx=0 tx=0).stop 2023-09-30T02:05:44.240 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:44.239+0000 7f369bae8700 10 monclient: shutdown 2023-09-30T02:05:44.240 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:44.239+0000 7f369bae8700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:05:44.240 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:44.239+0000 7f369bae8700 1 -- 172.21.15.59:0/1895960825 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f36940a7f80 msgr2=0x7f3694139170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:44.240 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:44.239+0000 7f369bae8700 1 --2- 172.21.15.59:0/1895960825 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f36940a7f80 0x7f3694139170 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f368400ecb0 tx=0x7f36840091a0 comp rx=0 tx=0).stop 2023-09-30T02:05:44.240 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:44.240+0000 7f369bae8700 1 -- 172.21.15.59:0/1895960825 shutdown_connections 2023-09-30T02:05:44.240 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:44.240+0000 7f369bae8700 1 --2- 172.21.15.59:0/1895960825 >> v2:172.21.15.110:6800/115870 conn(0x7f3678068800 0x7f367806acb0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:44.241 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:44.240+0000 7f369bae8700 1 --2- 172.21.15.59:0/1895960825 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f36940a7630 0x7f369413e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:44.241 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:44.240+0000 7f369bae8700 1 --2- 172.21.15.59:0/1895960825 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36940b1660 0x7f3694155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:44.241 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:44.240+0000 7f369bae8700 1 --2- 172.21.15.59:0/1895960825 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f36940a7f80 0x7f3694139170 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:44.241 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:44.240+0000 7f369bae8700 1 -- 172.21.15.59:0/1895960825 >> 172.21.15.59:0/1895960825 conn(0x7f3694069130 msgr2=0x7f36940aae30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:05:44.242 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:44.242+0000 7f369bae8700 1 -- 172.21.15.59:0/1895960825 shutdown_connections 2023-09-30T02:05:44.243 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:44.242+0000 7f369bae8700 1 -- 172.21.15.59:0/1895960825 wait complete. 2023-09-30T02:05:44.243 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:44.243+0000 7f369bae8700 1 librados: shutdown 2023-09-30T02:05:44.260 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-09-30T02:05:44.523 INFO:tasks.ceph.osd.6.smithi110.stderr:==00:00:00:26.061 116563== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:44.528 INFO:tasks.ceph.osd.6.smithi110.stderr:==00:00:00:26.066 116563== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:44.834 INFO:tasks.ceph.osd.5.smithi110.stderr:==00:00:00:26.371 116562== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:44.839 INFO:tasks.ceph.osd.5.smithi110.stderr:==00:00:00:26.377 116562== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:45.077 INFO:tasks.ceph.osd.4.smithi110.stderr:==00:00:00:26.639 116560== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:45.078 INFO:tasks.ceph.osd.7.smithi110.stderr:==00:00:00:26.632 116561== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:45.082 INFO:tasks.ceph.osd.4.smithi110.stderr:==00:00:00:26.645 116560== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:45.086 INFO:tasks.ceph.osd.7.smithi110.stderr:==00:00:00:26.641 116561== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:47.095 INFO:tasks.ceph.osd.0.smithi059.stderr:==00:00:00:28.641 116664== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:47.100 INFO:tasks.ceph.osd.0.smithi059.stderr:==00:00:00:28.647 116664== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:47.302 INFO:tasks.ceph.osd.2.smithi059.stderr:==00:00:00:28.830 116666== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:47.309 INFO:tasks.ceph.osd.2.smithi059.stderr:==00:00:00:28.836 116666== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:47.401 INFO:tasks.ceph.osd.1.smithi059.stderr:==00:00:00:28.939 116665== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:47.410 INFO:tasks.ceph.osd.1.smithi059.stderr:==00:00:00:28.949 116665== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:48.209 INFO:tasks.ceph.osd.3.smithi059.stderr:==00:00:00:29.731 116667== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:48.218 INFO:tasks.ceph.osd.3.smithi059.stderr:==00:00:00:29.740 116667== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:49.275 INFO:tasks.ceph.osd.6.smithi110.stderr:==00:00:00:30.812 116563== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:49.455 INFO:tasks.ceph.osd.5.smithi110.stderr:==00:00:00:30.993 116562== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:49.890 INFO:tasks.ceph.osd.7.smithi110.stderr:==00:00:00:31.444 116561== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:50.094 INFO:tasks.ceph.osd.4.smithi110.stderr:==00:00:00:31.656 116560== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:51.067 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-09-30T02:05:51.231 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.230+0000 7faedfa48700 10 monclient: get_monmap_and_config 2023-09-30T02:05:51.231 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.230+0000 7faedfa48700 10 monclient: build_initial_monmap 2023-09-30T02:05:51.231 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.230+0000 7faedfa48700 10 monclient: monmap: 2023-09-30T02:05:51.231 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2023-09-30T02:05:51.231 INFO:teuthology.misc.health.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:51.231 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2023-09-30T02:05:51.232061+0000 2023-09-30T02:05:51.231 INFO:teuthology.misc.health.smithi059.stderr:created 2023-09-30T02:05:51.232061+0000 2023-09-30T02:05:51.231 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:05:51.231 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 3 2023-09-30T02:05:51.232 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:05:51.232 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:05:51.232 INFO:teuthology.misc.health.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:05:51.232 INFO:teuthology.misc.health.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:05:51.232 INFO:teuthology.misc.health.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:05:51.232 INFO:teuthology.misc.health.smithi059.stderr: 2023-09-30T02:05:51.233 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.231+0000 7faedfa48700 1 Processor -- start 2023-09-30T02:05:51.233 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.231+0000 7faedfa48700 1 -- start start 2023-09-30T02:05:51.233 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.231+0000 7faedfa48700 10 monclient: init 2023-09-30T02:05:51.233 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.232+0000 7faedfa48700 10 monclient: _reopen_session rank -1 2023-09-30T02:05:51.233 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.232+0000 7faedfa48700 10 monclient: _add_conns ranks=[0,1,4,2,3] 2023-09-30T02:05:51.233 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.232+0000 7faedfa48700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7faed80b1660 addr v1:172.21.15.59:6789/0 2023-09-30T02:05:51.233 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.232+0000 7faedfa48700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7faed80a7630 addr v1:172.21.15.110:6789/0 2023-09-30T02:05:51.233 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.232+0000 7faedfa48700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7faed80a8150 0x7faed8148d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:51.233 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.232+0000 7faedfa48700 10 monclient(hunting): picked mon.noname-c con 0x7faed80a8150 addr v2:172.21.15.110:3300/0 2023-09-30T02:05:51.234 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.232+0000 7faedfa48700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faed80a6d00 con 0x7faed80b1660 2023-09-30T02:05:51.234 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.232+0000 7faedfa48700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faed80a5e40 con 0x7faed80a7630 2023-09-30T02:05:51.234 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.232+0000 7faedfa48700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:51.234 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.232+0000 7faedfa48700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faed80b1a30 con 0x7faed80a8150 2023-09-30T02:05:51.234 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.232+0000 7faedfa48700 10 monclient(hunting): _renew_subs 2023-09-30T02:05:51.234 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.232+0000 7faedfa48700 10 monclient(hunting): authenticate will time out at 1796.535941s 2023-09-30T02:05:51.235 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.233+0000 7faeddfe5700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7faed80a8150 0x7faed8148d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:51.235 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.234+0000 7faedcfe3700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7faed80a7630 0x7faed80a7a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.59:49224/0 (socket says 172.21.15.59:49224) 2023-09-30T02:05:51.235 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.234+0000 7faedcfe3700 1 -- 172.21.15.59:0/417853089 learned_addr learned my addr 172.21.15.59:0/417853089 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:05:51.235 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.234+0000 7faeddfe5700 10 monclient(hunting): get_auth_request con 0x7faed80a8150 auth_method 0 2023-09-30T02:05:51.235 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.234+0000 7faeddfe5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:51.235 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.234+0000 7faeddfe5700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:51.235 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.234+0000 7faeddfe5700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:51.237 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.237+0000 7faeddfe5700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:05:51.237 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.237+0000 7faeddfe5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:05:51.237 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.237+0000 7faeddfe5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:05:51.238 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.238+0000 7faecffff700 1 -- 172.21.15.59:0/417853089 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 230080067 0 0) 0x7faed80a5e40 con 0x7faed80a7630 2023-09-30T02:05:51.238 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.238+0000 7faecffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:51.238 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.238+0000 7faecffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:51.238 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.238+0000 7faecffff700 10 monclient(hunting): my global_id is 4216 2023-09-30T02:05:51.238 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.238+0000 7faecffff700 1 -- 172.21.15.59:0/417853089 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faec4003570 con 0x7faed80a7630 2023-09-30T02:05:51.241 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.240+0000 7faeddfe5700 10 monclient(hunting): handle_auth_done global_id 4213 payload 274 2023-09-30T02:05:51.241 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.240+0000 7faeddfe5700 1 -- 172.21.15.59:0/417853089 >> v1:172.21.15.110:6789/0 conn(0x7faed80a7630 legacy=0x7faed80a7a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:51.241 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.240+0000 7faeddfe5700 1 -- 172.21.15.59:0/417853089 >> v1:172.21.15.59:6789/0 conn(0x7faed80b1660 legacy=0x7faed80a54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:05:51.241 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.240+0000 7faeddfe5700 10 monclient: _finish_hunting 0 2023-09-30T02:05:51.241 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.240+0000 7faeddfe5700 1 monclient: found mon.noname-c 2023-09-30T02:05:51.241 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.240+0000 7faeddfe5700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:05:51.241 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.240+0000 7faeddfe5700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3300/0 2023-09-30T02:05:51.241 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.240+0000 7faeddfe5700 1 -- 172.21.15.59:0/417853089 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faed80a6f30 con 0x7faed80a8150 2023-09-30T02:05:51.247 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.247+0000 7faeddfe5700 1 --2- 172.21.15.59:0/417853089 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faed80a8150 0x7faed8148d80 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7faed0009e60 tx=0x7faed0009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b4ac849c01bcef9c server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:05:51.256 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.255+0000 7faecffff700 1 -- 172.21.15.59:0/417853089 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed000e070 con 0x7faed80a8150 2023-09-30T02:05:51.256 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.255+0000 7faecffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:05:51.256 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.255+0000 7faecffff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:05:51.256 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.255+0000 7faecffff700 10 monclient: dump: 2023-09-30T02:05:51.256 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2023-09-30T02:05:51.256 INFO:teuthology.misc.health.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:51.256 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:51.256 INFO:teuthology.misc.health.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:51.256 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:05:51.256 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 3 2023-09-30T02:05:51.256 INFO:teuthology.misc.health.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:05:51.256 INFO:teuthology.misc.health.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:05:51.257 INFO:teuthology.misc.health.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:05:51.257 INFO:teuthology.misc.health.smithi059.stderr: 2023-09-30T02:05:51.257 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.255+0000 7faecffff700 10 monclient: _finish_auth 0 2023-09-30T02:05:51.257 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.255+0000 7faecffff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:05:51.257 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.255+0000 7faedfa48700 5 monclient: authenticate success, global_id 4213 2023-09-30T02:05:51.257 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.255+0000 7faedfa48700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:05:51.258 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.258+0000 7faecffff700 1 -- 172.21.15.59:0/417853089 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faed000bb20 con 0x7faed80a8150 2023-09-30T02:05:51.258 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.258+0000 7faecffff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:05:51.258 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.258+0000 7faecffff700 1 -- 172.21.15.59:0/417853089 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed000bea0 con 0x7faed80a8150 2023-09-30T02:05:51.259 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.258+0000 7faedfa48700 10 monclient: get_monmap_and_config success 2023-09-30T02:05:51.259 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.258+0000 7faedfa48700 4 set_mon_vals no callback set 2023-09-30T02:05:51.259 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.258+0000 7faecffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:05:51.259 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.258+0000 7faecffff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:05:51.259 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.258+0000 7faecffff700 10 monclient: dump: 2023-09-30T02:05:51.259 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2023-09-30T02:05:51.259 INFO:teuthology.misc.health.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:51.259 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:51.259 INFO:teuthology.misc.health.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:51.259 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:05:51.259 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 3 2023-09-30T02:05:51.259 INFO:teuthology.misc.health.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:05:51.260 INFO:teuthology.misc.health.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:05:51.260 INFO:teuthology.misc.health.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:05:51.260 INFO:teuthology.misc.health.smithi059.stderr: 2023-09-30T02:05:51.260 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.259+0000 7faedfa48700 10 monclient: shutdown 2023-09-30T02:05:51.260 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.259+0000 7faedfa48700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:05:51.260 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.259+0000 7faedfa48700 1 -- 172.21.15.59:0/417853089 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faed80a8150 msgr2=0x7faed8148d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:51.261 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.259+0000 7faedfa48700 1 --2- 172.21.15.59:0/417853089 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faed80a8150 0x7faed8148d80 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7faed0009e60 tx=0x7faed0009780 comp rx=0 tx=0).stop 2023-09-30T02:05:51.261 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.259+0000 7faedfa48700 1 -- 172.21.15.59:0/417853089 shutdown_connections 2023-09-30T02:05:51.261 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.259+0000 7faedfa48700 1 --2- 172.21.15.59:0/417853089 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faed80a8150 0x7faed8148d80 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:51.261 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.259+0000 7faedfa48700 1 -- 172.21.15.59:0/417853089 >> 172.21.15.59:0/417853089 conn(0x7faed8069130 msgr2=0x7faed8063280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:05:51.261 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.259+0000 7faedfa48700 1 -- 172.21.15.59:0/417853089 shutdown_connections 2023-09-30T02:05:51.261 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.259+0000 7faedfa48700 1 -- 172.21.15.59:0/417853089 wait complete. 2023-09-30T02:05:51.261 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.260+0000 7faedfa48700 10 monclient: build_initial_monmap 2023-09-30T02:05:51.261 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.260+0000 7faedfa48700 10 monclient: monmap: 2023-09-30T02:05:51.261 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2023-09-30T02:05:51.262 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:05:51.262 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2023-09-30T02:05:51.262 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2023-09-30T02:05:51.262 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:05:51.262 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2023-09-30T02:05:51.262 INFO:teuthology.misc.health.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:05:51.262 INFO:teuthology.misc.health.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:05:51.262 INFO:teuthology.misc.health.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:05:51.262 INFO:teuthology.misc.health.smithi059.stderr: 2023-09-30T02:05:51.262 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.260+0000 7faedfa48700 1 librados: starting msgr at 2023-09-30T02:05:51.262 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.260+0000 7faedfa48700 1 librados: starting objecter 2023-09-30T02:05:51.262 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.261+0000 7faedfa48700 1 Processor -- start 2023-09-30T02:05:51.263 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.261+0000 7faedfa48700 1 -- start start 2023-09-30T02:05:51.263 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.261+0000 7faedfa48700 1 librados: setting wanted keys 2023-09-30T02:05:51.263 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.261+0000 7faedfa48700 1 librados: calling monclient init 2023-09-30T02:05:51.263 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.261+0000 7faedfa48700 10 monclient: init 2023-09-30T02:05:51.263 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.262+0000 7faedfa48700 10 monclient: _reopen_session rank -1 2023-09-30T02:05:51.263 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.262+0000 7faedfa48700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T02:05:51.263 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.262+0000 7faedfa48700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7faed80a7630 0x7faed813cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:51.264 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.262+0000 7faedfa48700 10 monclient(hunting): picked mon.noname-c con 0x7faed80a7630 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:05:51.264 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.262+0000 7faedfa48700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faed80a8150 0x7faed8140520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:51.264 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.262+0000 7faedfa48700 10 monclient(hunting): picked mon.noname-a con 0x7faed80a8150 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:05:51.264 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.262+0000 7faedfa48700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faed80b1660 0x7faed8140a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:51.264 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.262+0000 7faedfa48700 10 monclient(hunting): picked mon.noname-b con 0x7faed80b1660 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:05:51.264 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.262+0000 7faedfa48700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:51.264 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.262+0000 7faedfa48700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faed813e570 con 0x7faed80a8150 2023-09-30T02:05:51.264 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.262+0000 7faedfa48700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:51.264 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.262+0000 7faedfa48700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faed813e820 con 0x7faed80b1660 2023-09-30T02:05:51.264 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.262+0000 7faedfa48700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:51.264 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.262+0000 7faedfa48700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faed813d260 con 0x7faed80a7630 2023-09-30T02:05:51.264 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.262+0000 7faedfa48700 10 monclient(hunting): _renew_subs 2023-09-30T02:05:51.265 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.262+0000 7faedfa48700 10 monclient(hunting): authenticate will time out at 1796.565473s 2023-09-30T02:05:51.265 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.263+0000 7faeddfe5700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faed80b1660 0x7faed8140a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:51.265 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.263+0000 7faeddfe5700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faed80b1660 0x7faed8140a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:37302/0 (socket says 172.21.15.59:37302) 2023-09-30T02:05:51.265 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.263+0000 7faeddfe5700 1 -- 172.21.15.59:0/929483002 learned_addr learned my addr 172.21.15.59:0/929483002 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:05:51.266 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.263+0000 7faeddfe5700 10 monclient(hunting): get_auth_request con 0x7faed80b1660 auth_method 0 2023-09-30T02:05:51.266 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.263+0000 7faeddfe5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:51.266 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.263+0000 7faeddfe5700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:51.266 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.263+0000 7faeddfe5700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:51.266 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.263+0000 7faedd7e4700 1 --2- 172.21.15.59:0/929483002 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7faed80a7630 0x7faed813cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:51.266 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.264+0000 7faedcfe3700 1 --2- 172.21.15.59:0/929483002 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faed80a8150 0x7faed8140520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:51.266 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.264+0000 7faedd7e4700 10 monclient(hunting): get_auth_request con 0x7faed80a7630 auth_method 0 2023-09-30T02:05:51.266 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.264+0000 7faedd7e4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:51.266 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.264+0000 7faedd7e4700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:51.266 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.264+0000 7faedd7e4700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:51.266 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.264+0000 7faeddfe5700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:05:51.266 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.264+0000 7faeddfe5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:05:51.267 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.264+0000 7faeddfe5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:05:51.267 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.264+0000 7faedcfe3700 10 monclient(hunting): get_auth_request con 0x7faed80a8150 auth_method 0 2023-09-30T02:05:51.267 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.264+0000 7faedcfe3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:51.267 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.264+0000 7faedcfe3700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:51.267 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.264+0000 7faedcfe3700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:51.267 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.265+0000 7faedd7e4700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:05:51.267 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.265+0000 7faedd7e4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:05:51.267 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.265+0000 7faedd7e4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:05:51.268 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.265+0000 7faedcfe3700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:05:51.268 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.265+0000 7faedcfe3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:05:51.268 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.265+0000 7faedcfe3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:05:51.270 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.269+0000 7faeddfe5700 10 monclient(hunting): handle_auth_done global_id 4219 payload 824 2023-09-30T02:05:51.270 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.269+0000 7faeddfe5700 1 -- 172.21.15.59:0/929483002 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7faed80a7630 msgr2=0x7faed813cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:51.270 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.269+0000 7faeddfe5700 1 --2- 172.21.15.59:0/929483002 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7faed80a7630 0x7faed813cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:51.270 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.269+0000 7faeddfe5700 1 -- 172.21.15.59:0/929483002 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faed80a8150 msgr2=0x7faed8140520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:51.270 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.269+0000 7faeddfe5700 1 --2- 172.21.15.59:0/929483002 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faed80a8150 0x7faed8140520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:51.270 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.269+0000 7faeddfe5700 10 monclient: _finish_hunting 0 2023-09-30T02:05:51.270 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.269+0000 7faeddfe5700 1 monclient: found mon.noname-b 2023-09-30T02:05:51.270 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.269+0000 7faeddfe5700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:05:51.270 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.269+0000 7faeddfe5700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:05:51.270 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.269+0000 7faeddfe5700 1 -- 172.21.15.59:0/929483002 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faed0009a20 con 0x7faed80b1660 2023-09-30T02:05:51.273 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.272+0000 7faeddfe5700 1 --2- 172.21.15.59:0/929483002 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faed80b1660 0x7faed8140a60 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7faed00097c0 tx=0x7faed0018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:05:51.273 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.272+0000 7faecd7fa700 10 client.?.objecter ms_handle_connect 0x7faed80b1660 2023-09-30T02:05:51.273 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.272+0000 7faecd7fa700 10 client.?.objecter resend_mon_ops 2023-09-30T02:05:51.277 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.277+0000 7faecd7fa700 1 -- 172.21.15.59:0/929483002 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed0012c40 con 0x7faed80b1660 2023-09-30T02:05:51.277 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.277+0000 7faecd7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:05:51.277 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.277+0000 7faecd7fa700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:05:51.277 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.277+0000 7faecd7fa700 10 monclient: dump: 2023-09-30T02:05:51.277 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2023-09-30T02:05:51.277 INFO:teuthology.misc.health.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:51.278 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:51.278 INFO:teuthology.misc.health.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:51.278 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:05:51.278 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 3 2023-09-30T02:05:51.278 INFO:teuthology.misc.health.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:05:51.278 INFO:teuthology.misc.health.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:05:51.278 INFO:teuthology.misc.health.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:05:51.278 INFO:teuthology.misc.health.smithi059.stderr: 2023-09-30T02:05:51.278 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.277+0000 7faecd7fa700 10 monclient: _finish_auth 0 2023-09-30T02:05:51.278 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.277+0000 7faecd7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:05:51.278 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.277+0000 7faedfa48700 5 monclient: authenticate success, global_id 4219 2023-09-30T02:05:51.278 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.277+0000 7faecd7fa700 1 -- 172.21.15.59:0/929483002 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faed0021070 con 0x7faed80b1660 2023-09-30T02:05:51.279 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.277+0000 7faecd7fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:05:51.279 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.277+0000 7faecd7fa700 1 -- 172.21.15.59:0/929483002 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed0026430 con 0x7faed80b1660 2023-09-30T02:05:51.279 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.277+0000 7faecd7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:05:51.279 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.277+0000 7faecd7fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:05:51.279 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.277+0000 7faecd7fa700 10 monclient: dump: 2023-09-30T02:05:51.279 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2023-09-30T02:05:51.279 INFO:teuthology.misc.health.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:51.279 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:51.279 INFO:teuthology.misc.health.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:51.279 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:05:51.279 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 3 2023-09-30T02:05:51.280 INFO:teuthology.misc.health.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:05:51.280 INFO:teuthology.misc.health.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:05:51.280 INFO:teuthology.misc.health.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:05:51.280 INFO:teuthology.misc.health.smithi059.stderr: 2023-09-30T02:05:51.280 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.277+0000 7faecf7fe700 4 set_mon_vals no callback set 2023-09-30T02:05:51.280 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.278+0000 7faedfa48700 10 monclient: _renew_subs 2023-09-30T02:05:51.280 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.278+0000 7faedfa48700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:05:51.281 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.278+0000 7faedfa48700 1 -- 172.21.15.59:0/929483002 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faed813ec40 con 0x7faed80b1660 2023-09-30T02:05:51.281 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.278+0000 7faedfa48700 10 client.4219.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:05:51.281 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.278+0000 7faedfa48700 10 monclient: _renew_subs 2023-09-30T02:05:51.281 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.278+0000 7faedfa48700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:05:51.281 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.278+0000 7faedfa48700 1 -- 172.21.15.59:0/929483002 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faed8137970 con 0x7faed80b1660 2023-09-30T02:05:51.281 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.278+0000 7faedfa48700 1 librados: init done 2023-09-30T02:05:51.281 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.281+0000 7faedfa48700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:05:51.281 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.281+0000 7faedfa48700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:05:51.281 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.281+0000 7faedfa48700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:05:51.282 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.281+0000 7faedfa48700 1 -- 172.21.15.59:0/929483002 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faed8063330 con 0x7faed80b1660 2023-09-30T02:05:51.382 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.381+0000 7faecd7fa700 1 -- 172.21.15.59:0/929483002 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 4) v1 ==== 84649+0+0 (secure 0 0 0) 0x7faed0023030 con 0x7faed80b1660 2023-09-30T02:05:51.383 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.382+0000 7faecd7fa700 1 --2- 172.21.15.59:0/929483002 >> v2:172.21.15.110:6800/115870 conn(0x7faebc068800 0x7faebc06acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:51.383 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.382+0000 7faecd7fa700 1 -- 172.21.15.59:0/929483002 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1641+0+0 (secure 0 0 0) 0x7faed0085d50 con 0x7faed80b1660 2023-09-30T02:05:51.383 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.382+0000 7faecd7fa700 10 client.4219.objecter ms_dispatch 0x7faed8001550 osd_map(10..10 src has 1..10) v4 2023-09-30T02:05:51.383 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.382+0000 7faecd7fa700 3 client.4219.objecter handle_osd_map got epochs [10,10] > 0 2023-09-30T02:05:51.383 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.382+0000 7faecd7fa700 3 client.4219.objecter handle_osd_map decoding full epoch 10 2023-09-30T02:05:51.383 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.382+0000 7faecd7fa700 20 client.4219.objecter dump_active .. 0 homeless 2023-09-30T02:05:51.383 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.382+0000 7faedd7e4700 1 --2- 172.21.15.59:0/929483002 >> v2:172.21.15.110:6800/115870 conn(0x7faebc068800 0x7faebc06acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:51.384 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.382+0000 7faedd7e4700 10 monclient: get_auth_request con 0x7faebc068800 auth_method 0 2023-09-30T02:05:51.384 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.383+0000 7faecd7fa700 10 client.4219.objecter ms_handle_connect 0x7faebc068800 2023-09-30T02:05:51.384 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.383+0000 7faedd7e4700 1 --2- 172.21.15.59:0/929483002 >> v2:172.21.15.110:6800/115870 conn(0x7faebc068800 0x7faebc06acb0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7faec8009af0 tx=0x7faec8009060 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:05:51.654 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.654+0000 7faecd7fa700 1 -- 172.21.15.59:0/929483002 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7faed0054060 con 0x7faed80b1660 2023-09-30T02:05:51.655 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.654+0000 7faecd7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:05:51.655 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.654+0000 7faecd7fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:05:51.964 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.964+0000 7faedfa48700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-09-30T02:05:51.964 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.964+0000 7faedfa48700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T02:05:51.964 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.964+0000 7faedfa48700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:05:51.965 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.964+0000 7faedfa48700 1 -- 172.21.15.59:0/929483002 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faed8064a00 con 0x7faed80b1660 2023-09-30T02:05:51.989 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.989+0000 7faecd7fa700 1 -- 172.21.15.59:0/929483002 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7013 (secure 0 0 0) 0x7faed0057680 con 0x7faed80b1660 2023-09-30T02:05:51.989 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.989+0000 7faecd7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T02:05:51.989 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.989+0000 7faecd7fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:05:51.989 INFO:teuthology.misc.health.smithi059.stdout: 2023-09-30T02:05:51.989 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"623052cb-4f82-4b2b-b72b-3ed5be614613","created":"2023-09-30T02:05:11.679291+0000","modified":"2023-09-30T02:05:18.270595+0000","last_up_change":"0.000000","last_in_change":"2023-09-30T02:05:18.270595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"efdab44b-be57-4b81-8567-ff2deec1d3e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff9786d-8c39-4044-91d2-a31f33e3b16e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b276ed4b-317b-4ddf-96ab-7ad71b8e3ff7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"48bfaf43-5846-4385-b9c6-6bc424cd5acd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"76fa5443-1571-4e72-9d22-6ab19e59839a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e9436478-4140-4fd6-ba2e-04fa6f6cf367","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"76676042-2074-4cf8-9ef9-5c87bf604944","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"045479f4-f7f7-4b95-bc38-a2c52ac31b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T02:05:51.992 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.991+0000 7faedfa48700 10 librados: watch_flush enter 2023-09-30T02:05:51.992 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.991+0000 7faedfa48700 10 librados: watch_flush exit 2023-09-30T02:05:51.992 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.991+0000 7faedfa48700 20 client.4219.objecter shutdown clearing up homeless session... 2023-09-30T02:05:51.993 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.991+0000 7faedfa48700 10 client.4219.objecter successfully canceled tick 2023-09-30T02:05:51.993 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.991+0000 7faedfa48700 1 -- 172.21.15.59:0/929483002 >> v2:172.21.15.110:6800/115870 conn(0x7faebc068800 msgr2=0x7faebc06acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:51.993 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.991+0000 7faedfa48700 1 --2- 172.21.15.59:0/929483002 >> v2:172.21.15.110:6800/115870 conn(0x7faebc068800 0x7faebc06acb0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7faec8009af0 tx=0x7faec8009060 comp rx=0 tx=0).stop 2023-09-30T02:05:51.993 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.991+0000 7faedfa48700 10 monclient: shutdown 2023-09-30T02:05:51.993 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.991+0000 7faedfa48700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:05:51.993 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.991+0000 7faedfa48700 1 -- 172.21.15.59:0/929483002 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faed80b1660 msgr2=0x7faed8140a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:51.993 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.991+0000 7faedfa48700 1 --2- 172.21.15.59:0/929483002 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faed80b1660 0x7faed8140a60 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7faed00097c0 tx=0x7faed0018000 comp rx=0 tx=0).stop 2023-09-30T02:05:51.993 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.992+0000 7faedfa48700 1 -- 172.21.15.59:0/929483002 shutdown_connections 2023-09-30T02:05:51.993 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.992+0000 7faedfa48700 1 --2- 172.21.15.59:0/929483002 >> v2:172.21.15.110:6800/115870 conn(0x7faebc068800 0x7faebc06acb0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:51.993 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.992+0000 7faedfa48700 1 --2- 172.21.15.59:0/929483002 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7faed80b1660 0x7faed8140a60 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:51.993 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.992+0000 7faedfa48700 1 --2- 172.21.15.59:0/929483002 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7faed80a7630 0x7faed813cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:51.994 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.992+0000 7faedfa48700 1 --2- 172.21.15.59:0/929483002 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faed80a8150 0x7faed8140520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:51.994 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.992+0000 7faedfa48700 1 -- 172.21.15.59:0/929483002 >> 172.21.15.59:0/929483002 conn(0x7faed8069130 msgr2=0x7faed80afb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:05:51.996 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.996+0000 7faedfa48700 1 -- 172.21.15.59:0/929483002 shutdown_connections 2023-09-30T02:05:51.996 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.996+0000 7faedfa48700 1 -- 172.21.15.59:0/929483002 wait complete. 2023-09-30T02:05:51.996 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:51.996+0000 7faedfa48700 1 librados: shutdown 2023-09-30T02:05:52.012 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-09-30T02:05:52.084 INFO:tasks.ceph.osd.0.smithi059.stderr:==00:00:00:33.631 116664== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:52.350 INFO:tasks.ceph.osd.2.smithi059.stderr:==00:00:00:33.878 116666== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:52.792 INFO:tasks.ceph.osd.1.smithi059.stderr:==00:00:00:34.330 116665== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:53.474 INFO:tasks.ceph.osd.3.smithi059.stderr:==00:00:00:34.997 116667== Warning: unimplemented fcntl command: 1038 2023-09-30T02:05:58.817 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-09-30T02:05:58.972 INFO:tasks.ceph.osd.6.smithi110.stderr:2023-09-30T02:05:58.969+0000 40451c0 -1 osd.6 0 log_to_monitors true 2023-09-30T02:05:58.990 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:58.991+0000 7f65811f9700 10 monclient: get_monmap_and_config 2023-09-30T02:05:58.991 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:58.991+0000 7f65811f9700 10 monclient: build_initial_monmap 2023-09-30T02:05:58.991 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:58.991+0000 7f65811f9700 10 monclient: monmap: 2023-09-30T02:05:58.991 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2023-09-30T02:05:58.991 INFO:teuthology.misc.health.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:58.991 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2023-09-30T02:05:58.991818+0000 2023-09-30T02:05:58.991 INFO:teuthology.misc.health.smithi059.stderr:created 2023-09-30T02:05:58.991818+0000 2023-09-30T02:05:58.991 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:05:58.991 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 3 2023-09-30T02:05:58.991 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:05:58.991 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:05:58.991 INFO:teuthology.misc.health.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:05:58.992 INFO:teuthology.misc.health.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:05:58.992 INFO:teuthology.misc.health.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:05:58.992 INFO:teuthology.misc.health.smithi059.stderr: 2023-09-30T02:05:58.993 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:58.992+0000 7f65811f9700 1 Processor -- start 2023-09-30T02:05:58.993 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:58.992+0000 7f65811f9700 1 -- start start 2023-09-30T02:05:58.993 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:58.992+0000 7f65811f9700 10 monclient: init 2023-09-30T02:05:58.993 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:58.992+0000 7f65811f9700 10 monclient: _reopen_session rank -1 2023-09-30T02:05:58.993 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:58.992+0000 7f65811f9700 10 monclient: _add_conns ranks=[3,4,1,2,0] 2023-09-30T02:05:58.993 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:58.992+0000 7f65811f9700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f657c0b1660 0x7f657c0a54e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:58.993 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:58.992+0000 7f65811f9700 10 monclient(hunting): picked mon.noname-a con 0x7f657c0b1660 addr v2:172.21.15.59:3300/0 2023-09-30T02:05:58.993 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:58.992+0000 7f65811f9700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f657c0a7630 0x7f657c0a7a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:58.993 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:58.992+0000 7f65811f9700 10 monclient(hunting): picked mon.noname-c con 0x7f657c0a7630 addr v2:172.21.15.110:3300/0 2023-09-30T02:05:58.993 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:58.992+0000 7f65811f9700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f657c0a7f80 addr v1:172.21.15.110:6789/0 2023-09-30T02:05:58.993 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:58.993+0000 7f65811f9700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f657c0a5e40 con 0x7f657c0a7f80 2023-09-30T02:05:58.994 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:58.993+0000 7f65811f9700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:58.994 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:58.993+0000 7f65811f9700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f657c0b1a30 con 0x7f657c0b1660 2023-09-30T02:05:58.994 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:58.993+0000 7f65811f9700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:58.994 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:58.993+0000 7f65811f9700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f657c0a8370 con 0x7f657c0a7630 2023-09-30T02:05:58.994 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:58.993+0000 7f65811f9700 10 monclient(hunting): _renew_subs 2023-09-30T02:05:58.994 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:58.993+0000 7f65811f9700 10 monclient(hunting): authenticate will time out at 1804.295038s 2023-09-30T02:05:58.994 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:58.993+0000 7f657ad9d700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f657c0b1660 0x7f657c0a54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:58.994 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:58.994+0000 7f657a59c700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f657c0a7630 0x7f657c0a7a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:58.995 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:58.994+0000 7f657b59e700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f657c0a7f80 0x7f657c0af430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.59:53674/0 (socket says 172.21.15.59:53674) 2023-09-30T02:05:58.995 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:58.994+0000 7f657b59e700 1 -- 172.21.15.59:0/2588865910 learned_addr learned my addr 172.21.15.59:0/2588865910 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:05:58.995 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:58.994+0000 7f657ad9d700 10 monclient(hunting): get_auth_request con 0x7f657c0b1660 auth_method 0 2023-09-30T02:05:58.995 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:58.994+0000 7f657ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:58.995 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:58.994+0000 7f657ad9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:58.996 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:58.994+0000 7f657ad9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:58.996 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:58.995+0000 7f657a59c700 10 monclient(hunting): get_auth_request con 0x7f657c0a7630 auth_method 0 2023-09-30T02:05:58.996 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:58.995+0000 7f657a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:58.996 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:58.995+0000 7f657a59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:58.996 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:58.995+0000 7f657a59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:58.996 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:58.995+0000 7f657ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:05:58.996 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:58.995+0000 7f657ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:05:58.996 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:58.995+0000 7f657ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:05:58.996 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:58.996+0000 7f657a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:05:58.996 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:58.996+0000 7f657a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:05:58.996 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:58.996+0000 7f657a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:05:58.997 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:58.997+0000 7f657ad9d700 10 monclient(hunting): handle_auth_done global_id 4218 payload 274 2023-09-30T02:05:58.997 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:58.997+0000 7f657ad9d700 1 -- 172.21.15.59:0/2588865910 >> v2:172.21.15.110:3300/0 conn(0x7f657c0a7630 msgr2=0x7f657c0a7a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:05:58.997 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:58.997+0000 7f657ad9d700 1 --2- 172.21.15.59:0/2588865910 >> v2:172.21.15.110:3300/0 conn(0x7f657c0a7630 0x7f657c0a7a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:58.998 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:58.997+0000 7f657ad9d700 1 -- 172.21.15.59:0/2588865910 >> v1:172.21.15.110:6789/0 conn(0x7f657c0a7f80 legacy=0x7f657c0af430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:58.998 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:58.997+0000 7f657ad9d700 10 monclient: _finish_hunting 0 2023-09-30T02:05:58.998 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:58.997+0000 7f657ad9d700 1 monclient: found mon.noname-a 2023-09-30T02:05:58.998 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:58.997+0000 7f657ad9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:05:58.998 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:58.997+0000 7f657ad9d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.59:3300/0 2023-09-30T02:05:58.998 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:58.997+0000 7f657ad9d700 1 -- 172.21.15.59:0/2588865910 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f657c0a6f30 con 0x7f657c0b1660 2023-09-30T02:05:58.999 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.000+0000 7f657ad9d700 1 --2- 172.21.15.59:0/2588865910 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f657c0b1660 0x7f657c0a54e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f657c06a030 tx=0x7f656c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=465825d4f0d83d53 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:05:59.006 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.006+0000 7f6579d9b700 1 -- 172.21.15.59:0/2588865910 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f656c011630 con 0x7f657c0b1660 2023-09-30T02:05:59.007 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.006+0000 7f6579d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:05:59.007 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.006+0000 7f6579d9b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T02:05:59.007 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.006+0000 7f6579d9b700 10 monclient: dump: 2023-09-30T02:05:59.007 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2023-09-30T02:05:59.007 INFO:teuthology.misc.health.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:59.007 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:59.007 INFO:teuthology.misc.health.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:59.007 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:05:59.007 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 3 2023-09-30T02:05:59.007 INFO:teuthology.misc.health.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:05:59.007 INFO:teuthology.misc.health.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:05:59.007 INFO:teuthology.misc.health.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:05:59.008 INFO:teuthology.misc.health.smithi059.stderr: 2023-09-30T02:05:59.008 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.006+0000 7f6579d9b700 10 monclient: _finish_auth 0 2023-09-30T02:05:59.008 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.006+0000 7f6579d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:05:59.008 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.006+0000 7f6579d9b700 1 -- 172.21.15.59:0/2588865910 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f656c011790 con 0x7f657c0b1660 2023-09-30T02:05:59.008 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.006+0000 7f6579d9b700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:05:59.008 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.006+0000 7f6579d9b700 1 -- 172.21.15.59:0/2588865910 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f656c011b30 con 0x7f657c0b1660 2023-09-30T02:05:59.008 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.006+0000 7f6579d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:05:59.008 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.006+0000 7f6579d9b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T02:05:59.008 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.006+0000 7f6579d9b700 10 monclient: dump: 2023-09-30T02:05:59.008 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2023-09-30T02:05:59.008 INFO:teuthology.misc.health.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:59.008 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:59.009 INFO:teuthology.misc.health.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:59.009 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:05:59.009 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 3 2023-09-30T02:05:59.009 INFO:teuthology.misc.health.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:05:59.009 INFO:teuthology.misc.health.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:05:59.009 INFO:teuthology.misc.health.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:05:59.009 INFO:teuthology.misc.health.smithi059.stderr: 2023-09-30T02:05:59.009 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.006+0000 7f65811f9700 5 monclient: authenticate success, global_id 4218 2023-09-30T02:05:59.009 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.006+0000 7f65811f9700 10 monclient: get_monmap_and_config success 2023-09-30T02:05:59.009 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.006+0000 7f65811f9700 4 set_mon_vals no callback set 2023-09-30T02:05:59.009 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.006+0000 7f65811f9700 10 monclient: shutdown 2023-09-30T02:05:59.009 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.006+0000 7f65811f9700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:05:59.010 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.006+0000 7f65811f9700 1 -- 172.21.15.59:0/2588865910 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f657c0b1660 msgr2=0x7f657c0a54e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:59.010 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.007+0000 7f65811f9700 1 --2- 172.21.15.59:0/2588865910 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f657c0b1660 0x7f657c0a54e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f657c06a030 tx=0x7f656c0091a0 comp rx=0 tx=0).stop 2023-09-30T02:05:59.010 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.007+0000 7f65811f9700 1 -- 172.21.15.59:0/2588865910 shutdown_connections 2023-09-30T02:05:59.010 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.007+0000 7f65811f9700 1 --2- 172.21.15.59:0/2588865910 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f657c0b1660 0x7f657c0a54e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:59.010 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.007+0000 7f65811f9700 1 --2- 172.21.15.59:0/2588865910 >> v2:172.21.15.110:3300/0 conn(0x7f657c0a7630 0x7f657c0a7a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:59.010 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.007+0000 7f65811f9700 1 -- 172.21.15.59:0/2588865910 >> 172.21.15.59:0/2588865910 conn(0x7f657c069130 msgr2=0x7f657c063280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:05:59.013 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.008+0000 7f65811f9700 1 -- 172.21.15.59:0/2588865910 shutdown_connections 2023-09-30T02:05:59.013 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.008+0000 7f65811f9700 1 -- 172.21.15.59:0/2588865910 wait complete. 2023-09-30T02:05:59.013 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.009+0000 7f65811f9700 10 monclient: build_initial_monmap 2023-09-30T02:05:59.013 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.009+0000 7f65811f9700 10 monclient: monmap: 2023-09-30T02:05:59.014 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2023-09-30T02:05:59.014 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:05:59.014 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2023-09-30T02:05:59.014 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2023-09-30T02:05:59.014 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:05:59.014 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2023-09-30T02:05:59.014 INFO:teuthology.misc.health.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:05:59.014 INFO:teuthology.misc.health.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:05:59.014 INFO:teuthology.misc.health.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:05:59.014 INFO:teuthology.misc.health.smithi059.stderr: 2023-09-30T02:05:59.015 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.009+0000 7f65811f9700 1 librados: starting msgr at 2023-09-30T02:05:59.015 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.009+0000 7f65811f9700 1 librados: starting objecter 2023-09-30T02:05:59.015 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.009+0000 7f65811f9700 1 Processor -- start 2023-09-30T02:05:59.015 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.010+0000 7f65811f9700 1 -- start start 2023-09-30T02:05:59.015 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.010+0000 7f65811f9700 1 librados: setting wanted keys 2023-09-30T02:05:59.015 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.010+0000 7f65811f9700 1 librados: calling monclient init 2023-09-30T02:05:59.015 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.010+0000 7f65811f9700 10 monclient: init 2023-09-30T02:05:59.015 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.010+0000 7f65811f9700 10 monclient: _reopen_session rank -1 2023-09-30T02:05:59.015 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.010+0000 7f65811f9700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T02:05:59.015 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.010+0000 7f65811f9700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f657c0a7630 0x7f657c13abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:59.015 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.010+0000 7f65811f9700 10 monclient(hunting): picked mon.noname-b con 0x7f657c0a7630 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:05:59.015 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.010+0000 7f65811f9700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f657c0a7f80 0x7f657c1514e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:59.016 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.010+0000 7f65811f9700 10 monclient(hunting): picked mon.noname-c con 0x7f657c0a7f80 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:05:59.016 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.011+0000 7f65811f9700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f657c0b1660 0x7f657c151a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:59.016 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.011+0000 7f65811f9700 10 monclient(hunting): picked mon.noname-a con 0x7f657c0b1660 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:05:59.016 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.011+0000 7f65811f9700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:59.016 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.011+0000 7f65811f9700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f657c13be10 con 0x7f657c0b1660 2023-09-30T02:05:59.016 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.011+0000 7f65811f9700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:59.016 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.011+0000 7f65811f9700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f657c0ad9a0 con 0x7f657c0a7630 2023-09-30T02:05:59.016 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.011+0000 7f65811f9700 10 monclient(hunting): start opening mon connection 2023-09-30T02:05:59.016 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.011+0000 7f65811f9700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f657c0ae270 con 0x7f657c0a7f80 2023-09-30T02:05:59.016 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.011+0000 7f65811f9700 10 monclient(hunting): _renew_subs 2023-09-30T02:05:59.016 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.011+0000 7f65811f9700 10 monclient(hunting): authenticate will time out at 1804.313045s 2023-09-30T02:05:59.016 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.011+0000 7f657a59c700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f657c0a7f80 0x7f657c1514e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:59.017 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.012+0000 7f657b59e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f657c0b1660 0x7f657c151a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:59.017 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.012+0000 7f657ad9d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f657c0a7630 0x7f657c13abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:59.017 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.012+0000 7f657ad9d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f657c0a7630 0x7f657c13abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:59254/0 (socket says 172.21.15.59:59254) 2023-09-30T02:05:59.017 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.012+0000 7f657ad9d700 1 -- 172.21.15.59:0/1671755734 learned_addr learned my addr 172.21.15.59:0/1671755734 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:05:59.017 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.012+0000 7f657ad9d700 10 monclient(hunting): get_auth_request con 0x7f657c0a7630 auth_method 0 2023-09-30T02:05:59.017 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.012+0000 7f657ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:59.017 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.012+0000 7f657ad9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:59.017 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.012+0000 7f657ad9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:59.017 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.012+0000 7f657a59c700 10 monclient(hunting): get_auth_request con 0x7f657c0a7f80 auth_method 0 2023-09-30T02:05:59.017 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.012+0000 7f657a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:59.017 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.012+0000 7f657a59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:59.017 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.012+0000 7f657a59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:59.018 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.012+0000 7f657b59e700 10 monclient(hunting): get_auth_request con 0x7f657c0b1660 auth_method 0 2023-09-30T02:05:59.018 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.012+0000 7f657b59e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:05:59.018 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.012+0000 7f657b59e700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:05:59.018 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.012+0000 7f657b59e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:05:59.018 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.013+0000 7f657ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:05:59.018 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.013+0000 7f657ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:05:59.018 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.013+0000 7f657ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:05:59.018 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.013+0000 7f657a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:05:59.018 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.013+0000 7f657a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:05:59.018 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.013+0000 7f657a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:05:59.018 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.013+0000 7f657b59e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:05:59.018 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.013+0000 7f657b59e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:05:59.018 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.013+0000 7f657b59e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:05:59.019 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.018+0000 7f657ad9d700 10 monclient(hunting): handle_auth_done global_id 4228 payload 824 2023-09-30T02:05:59.019 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.018+0000 7f657ad9d700 1 -- 172.21.15.59:0/1671755734 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f657c0a7f80 msgr2=0x7f657c1514e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:59.019 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.018+0000 7f657ad9d700 1 --2- 172.21.15.59:0/1671755734 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f657c0a7f80 0x7f657c1514e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:59.019 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.018+0000 7f657ad9d700 1 -- 172.21.15.59:0/1671755734 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f657c0b1660 msgr2=0x7f657c151a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:59.019 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.018+0000 7f657ad9d700 1 --2- 172.21.15.59:0/1671755734 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f657c0b1660 0x7f657c151a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:59.019 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.018+0000 7f657ad9d700 10 monclient: _finish_hunting 0 2023-09-30T02:05:59.019 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.018+0000 7f657ad9d700 1 monclient: found mon.noname-b 2023-09-30T02:05:59.019 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.018+0000 7f657ad9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:05:59.020 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.018+0000 7f657ad9d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:05:59.020 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.018+0000 7f657ad9d700 1 -- 172.21.15.59:0/1671755734 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f656c009420 con 0x7f657c0a7630 2023-09-30T02:05:59.021 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.022+0000 7f657ad9d700 1 --2- 172.21.15.59:0/1671755734 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f657c0a7630 0x7f657c13abe0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f657c0b1e00 tx=0x7f656c017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:05:59.021 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.022+0000 7f6562ffd700 10 client.?.objecter ms_handle_connect 0x7f657c0a7630 2023-09-30T02:05:59.021 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.022+0000 7f6562ffd700 10 client.?.objecter resend_mon_ops 2023-09-30T02:05:59.026 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.027+0000 7f6562ffd700 1 -- 172.21.15.59:0/1671755734 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f656c000d10 con 0x7f657c0a7630 2023-09-30T02:05:59.026 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.027+0000 7f6562ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:05:59.026 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.027+0000 7f6562ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:05:59.027 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.027+0000 7f6562ffd700 10 monclient: dump: 2023-09-30T02:05:59.027 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2023-09-30T02:05:59.027 INFO:teuthology.misc.health.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:59.027 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:59.027 INFO:teuthology.misc.health.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:59.027 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:05:59.027 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 3 2023-09-30T02:05:59.027 INFO:teuthology.misc.health.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:05:59.027 INFO:teuthology.misc.health.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:05:59.027 INFO:teuthology.misc.health.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:05:59.027 INFO:teuthology.misc.health.smithi059.stderr: 2023-09-30T02:05:59.028 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.027+0000 7f6562ffd700 10 monclient: _finish_auth 0 2023-09-30T02:05:59.028 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.027+0000 7f6562ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:05:59.028 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.027+0000 7f6562ffd700 1 -- 172.21.15.59:0/1671755734 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f656c011b10 con 0x7f657c0a7630 2023-09-30T02:05:59.028 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.027+0000 7f6562ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:05:59.028 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.027+0000 7f6562ffd700 1 -- 172.21.15.59:0/1671755734 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f656c023660 con 0x7f657c0a7630 2023-09-30T02:05:59.028 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.027+0000 7f6562ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:05:59.028 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.027+0000 7f6562ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:05:59.028 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.027+0000 7f657959a700 4 set_mon_vals no callback set 2023-09-30T02:05:59.028 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.027+0000 7f6562ffd700 10 monclient: dump: 2023-09-30T02:05:59.028 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2023-09-30T02:05:59.028 INFO:teuthology.misc.health.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:05:59.028 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:59.029 INFO:teuthology.misc.health.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:05:59.029 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:05:59.029 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 3 2023-09-30T02:05:59.029 INFO:teuthology.misc.health.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:05:59.029 INFO:teuthology.misc.health.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:05:59.029 INFO:teuthology.misc.health.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:05:59.029 INFO:teuthology.misc.health.smithi059.stderr: 2023-09-30T02:05:59.029 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.027+0000 7f65811f9700 5 monclient: authenticate success, global_id 4228 2023-09-30T02:05:59.029 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.027+0000 7f65811f9700 10 monclient: _renew_subs 2023-09-30T02:05:59.029 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.027+0000 7f65811f9700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:05:59.029 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.027+0000 7f65811f9700 1 -- 172.21.15.59:0/1671755734 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f657c13b230 con 0x7f657c0a7630 2023-09-30T02:05:59.029 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.027+0000 7f65811f9700 10 client.4228.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:05:59.030 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.027+0000 7f65811f9700 10 monclient: _renew_subs 2023-09-30T02:05:59.030 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.027+0000 7f65811f9700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:05:59.030 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.027+0000 7f65811f9700 1 -- 172.21.15.59:0/1671755734 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f657c0ad660 con 0x7f657c0a7630 2023-09-30T02:05:59.030 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.027+0000 7f65811f9700 1 librados: init done 2023-09-30T02:05:59.030 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.028+0000 7f65811f9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:05:59.030 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.028+0000 7f65811f9700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:05:59.030 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.028+0000 7f65811f9700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:05:59.031 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.028+0000 7f65811f9700 1 -- 172.21.15.59:0/1671755734 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f657c064a00 con 0x7f657c0a7630 2023-09-30T02:05:59.122 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.123+0000 7f6562ffd700 1 -- 172.21.15.59:0/1671755734 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 4) v1 ==== 84649+0+0 (secure 0 0 0) 0x7f656c0193b0 con 0x7f657c0a7630 2023-09-30T02:05:59.123 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.124+0000 7f6562ffd700 1 --2- 172.21.15.59:0/1671755734 >> v2:172.21.15.110:6800/115870 conn(0x7f6568068800 0x7f656806acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:05:59.123 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.124+0000 7f657a59c700 1 --2- 172.21.15.59:0/1671755734 >> v2:172.21.15.110:6800/115870 conn(0x7f6568068800 0x7f656806acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:05:59.123 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.124+0000 7f657a59c700 10 monclient: get_auth_request con 0x7f6568068800 auth_method 0 2023-09-30T02:05:59.124 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.125+0000 7f6562ffd700 10 client.4228.objecter ms_handle_connect 0x7f6568068800 2023-09-30T02:05:59.124 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.125+0000 7f657a59c700 1 --2- 172.21.15.59:0/1671755734 >> v2:172.21.15.110:6800/115870 conn(0x7f6568068800 0x7f656806acb0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f657c0ae5e0 tx=0x7f65700091a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:05:59.143 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.144+0000 7f6562ffd700 1 -- 172.21.15.59:0/1671755734 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1641+0+0 (secure 0 0 0) 0x7f656c057430 con 0x7f657c0a7630 2023-09-30T02:05:59.144 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.144+0000 7f6562ffd700 10 client.4228.objecter ms_dispatch 0x7f657c001550 osd_map(10..10 src has 1..10) v4 2023-09-30T02:05:59.144 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.144+0000 7f6562ffd700 3 client.4228.objecter handle_osd_map got epochs [10,10] > 0 2023-09-30T02:05:59.144 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.144+0000 7f6562ffd700 3 client.4228.objecter handle_osd_map decoding full epoch 10 2023-09-30T02:05:59.144 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.144+0000 7f6562ffd700 20 client.4228.objecter dump_active .. 0 homeless 2023-09-30T02:05:59.371 INFO:tasks.ceph.osd.7.smithi110.stderr:2023-09-30T02:05:59.367+0000 40451c0 -1 osd.7 0 log_to_monitors true 2023-09-30T02:05:59.494 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.495+0000 7f6562ffd700 1 -- 172.21.15.59:0/1671755734 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f656c053070 con 0x7f657c0a7630 2023-09-30T02:05:59.494 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.495+0000 7f6562ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:05:59.495 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.495+0000 7f6562ffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:05:59.727 INFO:tasks.ceph.osd.5.smithi110.stderr:2023-09-30T02:05:59.723+0000 40451c0 -1 osd.5 0 log_to_monitors true 2023-09-30T02:05:59.760 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.761+0000 7f65811f9700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-09-30T02:05:59.760 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.761+0000 7f65811f9700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T02:05:59.760 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.761+0000 7f65811f9700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:05:59.761 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.761+0000 7f65811f9700 1 -- 172.21.15.59:0/1671755734 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f657c0a5a20 con 0x7f657c0a7630 2023-09-30T02:05:59.779 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.780+0000 7f6562ffd700 1 -- 172.21.15.59:0/1671755734 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7013 (secure 0 0 0) 0x7f656c028090 con 0x7f657c0a7630 2023-09-30T02:05:59.779 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.780+0000 7f6562ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T02:05:59.779 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.780+0000 7f6562ffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:05:59.780 INFO:teuthology.misc.health.smithi059.stdout: 2023-09-30T02:05:59.780 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":10,"fsid":"623052cb-4f82-4b2b-b72b-3ed5be614613","created":"2023-09-30T02:05:11.679291+0000","modified":"2023-09-30T02:05:18.270595+0000","last_up_change":"0.000000","last_in_change":"2023-09-30T02:05:18.270595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"efdab44b-be57-4b81-8567-ff2deec1d3e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff9786d-8c39-4044-91d2-a31f33e3b16e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b276ed4b-317b-4ddf-96ab-7ad71b8e3ff7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"48bfaf43-5846-4385-b9c6-6bc424cd5acd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"76fa5443-1571-4e72-9d22-6ab19e59839a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"e9436478-4140-4fd6-ba2e-04fa6f6cf367","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"76676042-2074-4cf8-9ef9-5c87bf604944","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"045479f4-f7f7-4b95-bc38-a2c52ac31b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T02:05:59.781 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.782+0000 7f65811f9700 10 librados: watch_flush enter 2023-09-30T02:05:59.781 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.782+0000 7f65811f9700 10 librados: watch_flush exit 2023-09-30T02:05:59.782 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.782+0000 7f65811f9700 20 client.4228.objecter shutdown clearing up homeless session... 2023-09-30T02:05:59.782 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.782+0000 7f65811f9700 10 client.4228.objecter successfully canceled tick 2023-09-30T02:05:59.783 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.783+0000 7f65811f9700 1 -- 172.21.15.59:0/1671755734 >> v2:172.21.15.110:6800/115870 conn(0x7f6568068800 msgr2=0x7f656806acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:59.783 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.783+0000 7f65811f9700 1 --2- 172.21.15.59:0/1671755734 >> v2:172.21.15.110:6800/115870 conn(0x7f6568068800 0x7f656806acb0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f657c0ae5e0 tx=0x7f65700091a0 comp rx=0 tx=0).stop 2023-09-30T02:05:59.783 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.783+0000 7f65811f9700 10 monclient: shutdown 2023-09-30T02:05:59.783 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.783+0000 7f65811f9700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:05:59.783 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.783+0000 7f65811f9700 1 -- 172.21.15.59:0/1671755734 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f657c0a7630 msgr2=0x7f657c13abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:05:59.783 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.783+0000 7f65811f9700 1 --2- 172.21.15.59:0/1671755734 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f657c0a7630 0x7f657c13abe0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f657c0b1e00 tx=0x7f656c017000 comp rx=0 tx=0).stop 2023-09-30T02:05:59.783 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.783+0000 7f65811f9700 1 -- 172.21.15.59:0/1671755734 shutdown_connections 2023-09-30T02:05:59.783 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.783+0000 7f65811f9700 1 --2- 172.21.15.59:0/1671755734 >> v2:172.21.15.110:6800/115870 conn(0x7f6568068800 0x7f656806acb0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:59.783 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.783+0000 7f65811f9700 1 --2- 172.21.15.59:0/1671755734 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f657c0a7630 0x7f657c13abe0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:59.784 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.783+0000 7f65811f9700 1 --2- 172.21.15.59:0/1671755734 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f657c0b1660 0x7f657c151a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:59.784 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.783+0000 7f65811f9700 1 --2- 172.21.15.59:0/1671755734 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f657c0a7f80 0x7f657c1514e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:05:59.784 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.783+0000 7f65811f9700 1 -- 172.21.15.59:0/1671755734 >> 172.21.15.59:0/1671755734 conn(0x7f657c069130 msgr2=0x7f657c144340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:05:59.784 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.783+0000 7f65811f9700 1 -- 172.21.15.59:0/1671755734 shutdown_connections 2023-09-30T02:05:59.784 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.784+0000 7f65811f9700 1 -- 172.21.15.59:0/1671755734 wait complete. 2023-09-30T02:05:59.784 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:05:59.784+0000 7f65811f9700 1 librados: shutdown 2023-09-30T02:05:59.801 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-09-30T02:05:59.881 INFO:tasks.ceph.osd.4.smithi110.stderr:2023-09-30T02:05:59.878+0000 40451c0 -1 osd.4 0 log_to_monitors true 2023-09-30T02:06:02.739 INFO:tasks.ceph.osd.6.smithi110.stderr:2023-09-30T02:06:02.735+0000 fe8c700 -1 osd.6 0 waiting for initial osdmap 2023-09-30T02:06:02.842 INFO:tasks.ceph.osd.7.smithi110.stderr:2023-09-30T02:06:02.839+0000 e689700 -1 osd.7 0 waiting for initial osdmap 2023-09-30T02:06:02.901 INFO:tasks.ceph.osd.0.smithi059.stderr:2023-09-30T02:06:02.896+0000 40451c0 -1 osd.0 0 log_to_monitors true 2023-09-30T02:06:03.173 INFO:tasks.ceph.osd.2.smithi059.stderr:2023-09-30T02:06:03.171+0000 40451c0 -1 osd.2 0 log_to_monitors true 2023-09-30T02:06:03.277 INFO:tasks.ceph.osd.6.smithi110.stderr:2023-09-30T02:06:03.264+0000 15091700 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T02:06:03.402 INFO:tasks.ceph.osd.7.smithi110.stderr:2023-09-30T02:06:03.389+0000 14890700 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T02:06:03.717 INFO:tasks.ceph.osd.1.smithi059.stderr:2023-09-30T02:06:03.714+0000 40451c0 -1 osd.1 0 log_to_monitors true 2023-09-30T02:06:04.107 INFO:tasks.ceph.osd.5.smithi110.stderr:2023-09-30T02:06:04.104+0000 fe8c700 -1 osd.5 0 waiting for initial osdmap 2023-09-30T02:06:04.136 INFO:tasks.ceph.osd.3.smithi059.stderr:2023-09-30T02:06:04.132+0000 40451c0 -1 osd.3 0 log_to_monitors true 2023-09-30T02:06:04.147 INFO:tasks.ceph.osd.4.smithi110.stderr:2023-09-30T02:06:04.134+0000 fe8c700 -1 osd.4 0 waiting for initial osdmap 2023-09-30T02:06:04.710 INFO:tasks.ceph.osd.5.smithi110.stderr:2023-09-30T02:06:04.696+0000 14890700 -1 osd.5 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T02:06:04.717 INFO:tasks.ceph.osd.4.smithi110.stderr:2023-09-30T02:06:04.699+0000 14890700 -1 osd.4 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T02:06:06.607 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-09-30T02:06:06.740 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.741+0000 7f7984a80700 10 monclient: get_monmap_and_config 2023-09-30T02:06:06.741 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.741+0000 7f7984a80700 10 monclient: build_initial_monmap 2023-09-30T02:06:06.741 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.741+0000 7f7984a80700 10 monclient: monmap: 2023-09-30T02:06:06.741 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2023-09-30T02:06:06.741 INFO:teuthology.misc.health.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:06.741 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2023-09-30T02:06:06.741911+0000 2023-09-30T02:06:06.741 INFO:teuthology.misc.health.smithi059.stderr:created 2023-09-30T02:06:06.741911+0000 2023-09-30T02:06:06.741 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:06.741 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:06.742 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:06:06.742 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:06:06.742 INFO:teuthology.misc.health.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:06:06.742 INFO:teuthology.misc.health.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:06:06.742 INFO:teuthology.misc.health.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:06:06.742 INFO:teuthology.misc.health.smithi059.stderr: 2023-09-30T02:06:06.744 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.742+0000 7f7984a80700 1 Processor -- start 2023-09-30T02:06:06.744 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.742+0000 7f7984a80700 1 -- start start 2023-09-30T02:06:06.744 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.742+0000 7f7984a80700 10 monclient: init 2023-09-30T02:06:06.744 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.743+0000 7f7984a80700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:06.744 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.743+0000 7f7984a80700 10 monclient: _add_conns ranks=[2,3,1,4,0] 2023-09-30T02:06:06.744 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.743+0000 7f7984a80700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f79800b13e0 0x7f79800b17b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:06.745 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.743+0000 7f7984a80700 10 monclient(hunting): picked mon.noname-b con 0x7f79800b13e0 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:06.745 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.743+0000 7f7984a80700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f79800a56f0 0x7f7980141740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:06.745 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.743+0000 7f7984a80700 10 monclient(hunting): picked mon.noname-a con 0x7f79800a56f0 addr v2:172.21.15.59:3300/0 2023-09-30T02:06:06.745 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.743+0000 7f7984a80700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7980141c80 addr v1:172.21.15.110:6789/0 2023-09-30T02:06:06.745 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.743+0000 7f7984a80700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79800a5c10 con 0x7f7980141c80 2023-09-30T02:06:06.745 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.743+0000 7f7984a80700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:06.745 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.743+0000 7f7984a80700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79800a7560 con 0x7f79800a56f0 2023-09-30T02:06:06.745 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.743+0000 7f7984a80700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:06.745 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.743+0000 7f7984a80700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f798013bce0 con 0x7f79800b13e0 2023-09-30T02:06:06.745 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.743+0000 7f7984a80700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:06.745 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.743+0000 7f7984a80700 10 monclient(hunting): authenticate will time out at 1812.045138s 2023-09-30T02:06:06.745 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.743+0000 7f797e59c700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f79800b13e0 0x7f79800b17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:06.746 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.743+0000 7f797ed9d700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f7980141c80 0x7f79800af400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.59:53688/0 (socket says 172.21.15.59:53688) 2023-09-30T02:06:06.746 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.744+0000 7f797ed9d700 1 -- 172.21.15.59:0/60375251 learned_addr learned my addr 172.21.15.59:0/60375251 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:06:06.746 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.744+0000 7f797e59c700 10 monclient(hunting): get_auth_request con 0x7f79800b13e0 auth_method 0 2023-09-30T02:06:06.746 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.744+0000 7f797e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:06.746 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.744+0000 7f797e59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:06.746 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.744+0000 7f797e59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:06.746 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.744+0000 7f797e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:06.746 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.744+0000 7f797e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:06.746 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.744+0000 7f797e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:06.747 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.747+0000 7f797d59a700 1 -- 172.21.15.59:0/60375251 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1829414827 0 0) 0x7f79800a5c10 con 0x7f7980141c80 2023-09-30T02:06:06.747 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.747+0000 7f797d59a700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:06.747 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.747+0000 7f797d59a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:06.747 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.747+0000 7f797d59a700 10 monclient(hunting): my global_id is 4255 2023-09-30T02:06:06.747 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.747+0000 7f797d59a700 1 -- 172.21.15.59:0/60375251 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7968003570 con 0x7f7980141c80 2023-09-30T02:06:06.748 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.747+0000 7f797e59c700 10 monclient(hunting): handle_auth_done global_id 4214 payload 274 2023-09-30T02:06:06.748 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.747+0000 7f797e59c700 1 -- 172.21.15.59:0/60375251 >> v2:172.21.15.59:3300/0 conn(0x7f79800a56f0 msgr2=0x7f7980141740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:06:06.748 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.747+0000 7f797e59c700 1 --2- 172.21.15.59:0/60375251 >> v2:172.21.15.59:3300/0 conn(0x7f79800a56f0 0x7f7980141740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:06.748 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.747+0000 7f797e59c700 1 -- 172.21.15.59:0/60375251 >> v1:172.21.15.110:6789/0 conn(0x7f7980141c80 legacy=0x7f79800af400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:06.748 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.747+0000 7f797e59c700 10 monclient: _finish_hunting 0 2023-09-30T02:06:06.748 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.747+0000 7f797e59c700 1 monclient: found mon.noname-b 2023-09-30T02:06:06.748 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.747+0000 7f797e59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:06.748 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.747+0000 7f797e59c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.59:3301/0 2023-09-30T02:06:06.748 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.747+0000 7f797e59c700 1 -- 172.21.15.59:0/60375251 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79800a60a0 con 0x7f79800b13e0 2023-09-30T02:06:06.749 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.749+0000 7f797e59c700 1 --2- 172.21.15.59:0/60375251 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f79800b13e0 0x7f79800b17b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f79700098e0 tx=0x7f79700091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f79221dc57956f52 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:06.753 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.753+0000 7f797d59a700 1 -- 172.21.15.59:0/60375251 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f797000b640 con 0x7f79800b13e0 2023-09-30T02:06:06.753 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.753+0000 7f797d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:06.753 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.753+0000 7f797d59a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:06:06.753 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.753+0000 7f797d59a700 10 monclient: dump: 2023-09-30T02:06:06.753 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2023-09-30T02:06:06.753 INFO:teuthology.misc.health.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:06.753 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:06.753 INFO:teuthology.misc.health.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:06.753 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:06.753 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:06.754 INFO:teuthology.misc.health.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:06.754 INFO:teuthology.misc.health.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:06.754 INFO:teuthology.misc.health.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:06.754 INFO:teuthology.misc.health.smithi059.stderr: 2023-09-30T02:06:06.754 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.753+0000 7f797d59a700 10 monclient: _finish_auth 0 2023-09-30T02:06:06.754 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.753+0000 7f797d59a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:06.754 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.753+0000 7f797d59a700 1 -- 172.21.15.59:0/60375251 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f797000b7a0 con 0x7f79800b13e0 2023-09-30T02:06:06.754 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.753+0000 7f7984a80700 5 monclient: authenticate success, global_id 4214 2023-09-30T02:06:06.754 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.753+0000 7f7984a80700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:06:06.754 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.753+0000 7f797d59a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:06.755 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.753+0000 7f7984a80700 10 monclient: get_monmap_and_config success 2023-09-30T02:06:06.755 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.753+0000 7f797d59a700 1 -- 172.21.15.59:0/60375251 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f797000bb40 con 0x7f79800b13e0 2023-09-30T02:06:06.755 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.753+0000 7f7984a80700 4 set_mon_vals no callback set 2023-09-30T02:06:06.755 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.753+0000 7f797d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:06.755 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.753+0000 7f797d59a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:06:06.755 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.753+0000 7f797d59a700 10 monclient: dump: 2023-09-30T02:06:06.755 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2023-09-30T02:06:06.755 INFO:teuthology.misc.health.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:06.755 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:06.755 INFO:teuthology.misc.health.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:06.755 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:06.755 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:06.755 INFO:teuthology.misc.health.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:06.756 INFO:teuthology.misc.health.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:06.756 INFO:teuthology.misc.health.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:06.756 INFO:teuthology.misc.health.smithi059.stderr: 2023-09-30T02:06:06.758 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.754+0000 7f7984a80700 10 monclient: shutdown 2023-09-30T02:06:06.758 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.754+0000 7f7984a80700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:06.759 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.754+0000 7f7984a80700 1 -- 172.21.15.59:0/60375251 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f79800b13e0 msgr2=0x7f79800b17b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:06.759 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.754+0000 7f7984a80700 1 --2- 172.21.15.59:0/60375251 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f79800b13e0 0x7f79800b17b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f79700098e0 tx=0x7f79700091a0 comp rx=0 tx=0).stop 2023-09-30T02:06:06.759 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.754+0000 7f7984a80700 1 -- 172.21.15.59:0/60375251 shutdown_connections 2023-09-30T02:06:06.759 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.754+0000 7f7984a80700 1 --2- 172.21.15.59:0/60375251 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f79800b13e0 0x7f79800b17b0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:06.759 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.754+0000 7f7984a80700 1 --2- 172.21.15.59:0/60375251 >> v2:172.21.15.59:3300/0 conn(0x7f79800a56f0 0x7f7980141740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:06.759 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.754+0000 7f7984a80700 1 -- 172.21.15.59:0/60375251 >> 172.21.15.59:0/60375251 conn(0x7f7980063280 msgr2=0x7f7980063680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:06.759 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.754+0000 7f7984a80700 1 -- 172.21.15.59:0/60375251 shutdown_connections 2023-09-30T02:06:06.759 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.754+0000 7f7984a80700 1 -- 172.21.15.59:0/60375251 wait complete. 2023-09-30T02:06:06.759 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.755+0000 7f7984a80700 10 monclient: build_initial_monmap 2023-09-30T02:06:06.759 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.755+0000 7f7984a80700 10 monclient: monmap: 2023-09-30T02:06:06.759 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2023-09-30T02:06:06.759 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:06:06.760 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2023-09-30T02:06:06.760 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2023-09-30T02:06:06.760 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:06.760 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2023-09-30T02:06:06.760 INFO:teuthology.misc.health.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:06:06.760 INFO:teuthology.misc.health.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:06:06.760 INFO:teuthology.misc.health.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:06:06.760 INFO:teuthology.misc.health.smithi059.stderr: 2023-09-30T02:06:06.760 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.755+0000 7f7984a80700 1 librados: starting msgr at 2023-09-30T02:06:06.760 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.755+0000 7f7984a80700 1 librados: starting objecter 2023-09-30T02:06:06.760 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.756+0000 7f7984a80700 1 Processor -- start 2023-09-30T02:06:06.761 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.756+0000 7f7984a80700 1 -- start start 2023-09-30T02:06:06.761 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.756+0000 7f7984a80700 1 librados: setting wanted keys 2023-09-30T02:06:06.761 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.756+0000 7f7984a80700 1 librados: calling monclient init 2023-09-30T02:06:06.761 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.756+0000 7f7984a80700 10 monclient: init 2023-09-30T02:06:06.761 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.756+0000 7f7984a80700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:06.761 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.756+0000 7f7984a80700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T02:06:06.761 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.756+0000 7f7984a80700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f79800a56f0 0x7f79800a93c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:06.761 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.756+0000 7f7984a80700 10 monclient(hunting): picked mon.noname-b con 0x7f79800a56f0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:06:06.761 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.756+0000 7f7984a80700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f79800b13e0 0x7f798013de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:06.761 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.756+0000 7f7984a80700 10 monclient(hunting): picked mon.noname-c con 0x7f79800b13e0 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:06.761 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.756+0000 7f7984a80700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7980141c80 0x7f79800a9e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:06.761 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.756+0000 7f7984a80700 10 monclient(hunting): picked mon.noname-a con 0x7f7980141c80 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:06:06.762 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.756+0000 7f7984a80700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:06.762 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.756+0000 7f7984a80700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79800ab840 con 0x7f7980141c80 2023-09-30T02:06:06.762 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.756+0000 7f7984a80700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:06.762 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.756+0000 7f7984a80700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f798013e390 con 0x7f79800a56f0 2023-09-30T02:06:06.762 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.756+0000 7f7984a80700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:06.762 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.756+0000 7f7984a80700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f798013ed20 con 0x7f79800b13e0 2023-09-30T02:06:06.762 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.756+0000 7f7984a80700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:06.762 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.756+0000 7f7984a80700 10 monclient(hunting): authenticate will time out at 1812.058840s 2023-09-30T02:06:06.762 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.757+0000 7f797e59c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f79800a56f0 0x7f79800a93c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:06.762 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.757+0000 7f797e59c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f79800a56f0 0x7f79800a93c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:59290/0 (socket says 172.21.15.59:59290) 2023-09-30T02:06:06.762 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.757+0000 7f797e59c700 1 -- 172.21.15.59:0/919998704 learned_addr learned my addr 172.21.15.59:0/919998704 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:06:06.762 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.757+0000 7f797e59c700 10 monclient(hunting): get_auth_request con 0x7f79800a56f0 auth_method 0 2023-09-30T02:06:06.762 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.757+0000 7f797e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:06.763 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.757+0000 7f797e59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:06.763 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.757+0000 7f797e59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:06.763 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.758+0000 7f797dd9b700 1 --2- 172.21.15.59:0/919998704 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f79800b13e0 0x7f798013de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:06.763 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.758+0000 7f797e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:06.763 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.758+0000 7f797e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:06.763 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.758+0000 7f797e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:06.763 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.758+0000 7f797dd9b700 10 monclient(hunting): get_auth_request con 0x7f79800b13e0 auth_method 0 2023-09-30T02:06:06.763 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.758+0000 7f797dd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:06.763 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.758+0000 7f797dd9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:06.763 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.758+0000 7f797dd9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:06.763 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.759+0000 7f797dd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:06.763 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.759+0000 7f797dd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:06.764 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.759+0000 7f797dd9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:06.764 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.761+0000 7f797e59c700 10 monclient(hunting): handle_auth_done global_id 4258 payload 824 2023-09-30T02:06:06.764 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.761+0000 7f797e59c700 1 -- 172.21.15.59:0/919998704 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f79800b13e0 msgr2=0x7f798013de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:06.765 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.761+0000 7f797e59c700 1 --2- 172.21.15.59:0/919998704 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f79800b13e0 0x7f798013de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:06.765 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.761+0000 7f797e59c700 1 -- 172.21.15.59:0/919998704 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7980141c80 msgr2=0x7f79800a9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:06.765 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.761+0000 7f797e59c700 1 --2- 172.21.15.59:0/919998704 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7980141c80 0x7f79800a9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:06.765 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.761+0000 7f797e59c700 10 monclient: _finish_hunting 0 2023-09-30T02:06:06.765 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.761+0000 7f797e59c700 1 monclient: found mon.noname-b 2023-09-30T02:06:06.765 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.761+0000 7f797e59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:06.765 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.761+0000 7f797e59c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:06:06.765 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.761+0000 7f797e59c700 1 -- 172.21.15.59:0/919998704 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7970009420 con 0x7f79800a56f0 2023-09-30T02:06:06.765 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.764+0000 7f797e59c700 1 --2- 172.21.15.59:0/919998704 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f79800a56f0 0x7f79800a93c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f79700093f0 tx=0x7f7970017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:06.765 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.764+0000 7f796e7fc700 10 client.?.objecter ms_handle_connect 0x7f79800a56f0 2023-09-30T02:06:06.765 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.764+0000 7f796e7fc700 10 client.?.objecter resend_mon_ops 2023-09-30T02:06:06.769 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.768+0000 7f796e7fc700 1 -- 172.21.15.59:0/919998704 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f797001bcb0 con 0x7f79800a56f0 2023-09-30T02:06:06.769 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.768+0000 7f796e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:06.769 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.768+0000 7f796e7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:06:06.770 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.768+0000 7f796e7fc700 10 monclient: dump: 2023-09-30T02:06:06.770 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2023-09-30T02:06:06.770 INFO:teuthology.misc.health.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:06.770 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:06.770 INFO:teuthology.misc.health.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:06.770 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:06.770 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:06.770 INFO:teuthology.misc.health.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:06.770 INFO:teuthology.misc.health.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:06.770 INFO:teuthology.misc.health.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:06.770 INFO:teuthology.misc.health.smithi059.stderr: 2023-09-30T02:06:06.770 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.768+0000 7f796e7fc700 10 monclient: _finish_auth 0 2023-09-30T02:06:06.771 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.768+0000 7f796e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:06.771 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.768+0000 7f796e7fc700 1 -- 172.21.15.59:0/919998704 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f797001be30 con 0x7f79800a56f0 2023-09-30T02:06:06.771 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.768+0000 7f7984a80700 5 monclient: authenticate success, global_id 4258 2023-09-30T02:06:06.771 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.768+0000 7f7984a80700 10 monclient: _renew_subs 2023-09-30T02:06:06.771 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.768+0000 7f7984a80700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:06.771 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.768+0000 7f7984a80700 1 -- 172.21.15.59:0/919998704 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79800a79f0 con 0x7f79800a56f0 2023-09-30T02:06:06.771 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.768+0000 7f796e7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:06.771 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.768+0000 7f796e7fc700 1 -- 172.21.15.59:0/919998704 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7970011650 con 0x7f79800a56f0 2023-09-30T02:06:06.771 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.768+0000 7f796e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:06.771 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.768+0000 7f796e7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:06:06.771 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.768+0000 7f796e7fc700 10 monclient: dump: 2023-09-30T02:06:06.771 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2023-09-30T02:06:06.772 INFO:teuthology.misc.health.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:06.772 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:06.772 INFO:teuthology.misc.health.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:06.772 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:06.772 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:06.772 INFO:teuthology.misc.health.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:06.772 INFO:teuthology.misc.health.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:06.772 INFO:teuthology.misc.health.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:06.772 INFO:teuthology.misc.health.smithi059.stderr: 2023-09-30T02:06:06.772 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.768+0000 7f797cd99700 4 set_mon_vals no callback set 2023-09-30T02:06:06.772 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.768+0000 7f7984a80700 10 client.4258.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:06:06.772 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.768+0000 7f7984a80700 10 monclient: _renew_subs 2023-09-30T02:06:06.772 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.768+0000 7f7984a80700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:06.773 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.768+0000 7f7984a80700 1 -- 172.21.15.59:0/919998704 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79800aa3d0 con 0x7f79800a56f0 2023-09-30T02:06:06.773 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.768+0000 7f7984a80700 1 librados: init done 2023-09-30T02:06:06.773 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.769+0000 7f7984a80700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:06.773 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.769+0000 7f7984a80700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:06.773 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.769+0000 7f7984a80700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:06.773 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.769+0000 7f7984a80700 1 -- 172.21.15.59:0/919998704 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79800a9b10 con 0x7f79800a56f0 2023-09-30T02:06:06.849 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.849+0000 7f796e7fc700 1 -- 172.21.15.59:0/919998704 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 4) v1 ==== 84649+0+0 (secure 0 0 0) 0x7f7970011a80 con 0x7f79800a56f0 2023-09-30T02:06:06.850 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.850+0000 7f796e7fc700 1 --2- 172.21.15.59:0/919998704 >> v2:172.21.15.110:6800/115870 conn(0x7f7960068880 0x7f796006ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:06.850 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.850+0000 7f796e7fc700 1 -- 172.21.15.59:0/919998704 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 2592+0+0 (secure 0 0 0) 0x7f7970085150 con 0x7f79800a56f0 2023-09-30T02:06:06.850 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.850+0000 7f796e7fc700 10 client.4258.objecter ms_dispatch 0x7f7980001550 osd_map(15..15 src has 1..15) v4 2023-09-30T02:06:06.850 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.850+0000 7f796e7fc700 3 client.4258.objecter handle_osd_map got epochs [15,15] > 0 2023-09-30T02:06:06.850 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.850+0000 7f796e7fc700 3 client.4258.objecter handle_osd_map decoding full epoch 15 2023-09-30T02:06:06.850 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.850+0000 7f796e7fc700 20 client.4258.objecter dump_active .. 0 homeless 2023-09-30T02:06:06.850 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.850+0000 7f797dd9b700 1 --2- 172.21.15.59:0/919998704 >> v2:172.21.15.110:6800/115870 conn(0x7f7960068880 0x7f796006ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:06.851 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.850+0000 7f797dd9b700 10 monclient: get_auth_request con 0x7f7960068880 auth_method 0 2023-09-30T02:06:06.851 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.850+0000 7f796e7fc700 10 client.4258.objecter ms_handle_connect 0x7f7960068880 2023-09-30T02:06:06.851 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:06.850+0000 7f797dd9b700 1 --2- 172.21.15.59:0/919998704 >> v2:172.21.15.110:6800/115870 conn(0x7f7960068880 0x7f796006ad30 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f79800aae60 tx=0x7f79740091a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:07.047 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:07.047+0000 7f796e7fc700 1 -- 172.21.15.59:0/919998704 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7970053150 con 0x7f79800a56f0 2023-09-30T02:06:07.047 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:07.047+0000 7f796e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:07.047 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:07.047+0000 7f796e7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:06:07.306 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:07.307+0000 7f7984a80700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-09-30T02:06:07.306 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:07.307+0000 7f7984a80700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T02:06:07.306 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:07.307+0000 7f7984a80700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:07.306 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:07.307+0000 7f7984a80700 1 -- 172.21.15.59:0/919998704 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f79800a9ce0 con 0x7f79800a56f0 2023-09-30T02:06:07.401 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:07.402+0000 7f796e7fc700 1 -- 172.21.15.59:0/919998704 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+9519 (secure 0 0 0) 0x7f7970056770 con 0x7f79800a56f0 2023-09-30T02:06:07.401 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:07.402+0000 7f796e7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T02:06:07.402 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:07.402+0000 7f796e7fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:06:07.402 INFO:teuthology.misc.health.smithi059.stdout: 2023-09-30T02:06:07.402 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":16,"fsid":"623052cb-4f82-4b2b-b72b-3ed5be614613","created":"2023-09-30T02:05:11.679291+0000","modified":"2023-09-30T02:06:06.666529+0000","last_up_change":"2023-09-30T02:06:05.635936+0000","last_in_change":"2023-09-30T02:05:18.270595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":7,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-30T02:06:06.187725+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{}}],"osds":[{"osd":0,"uuid":"efdab44b-be57-4b81-8567-ff2deec1d3e3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eff9786d-8c39-4044-91d2-a31f33e3b16e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b276ed4b-317b-4ddf-96ab-7ad71b8e3ff7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"48bfaf43-5846-4385-b9c6-6bc424cd5acd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"76fa5443-1571-4e72-9d22-6ab19e59839a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6801","nonce":116560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":116560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":116560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6803","nonce":116560}]},"public_addr":"172.21.15.110:6801/116560","cluster_addr":"172.21.15.110:6802/116560","heartbeat_back_addr":"172.21.15.110:6804/116560","heartbeat_front_addr":"172.21.15.110:6803/116560","state":["exists","up"]},{"osd":5,"uuid":"e9436478-4140-4fd6-ba2e-04fa6f6cf367","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6809","nonce":116562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":116562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":116562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6811","nonce":116562}]},"public_addr":"172.21.15.110:6809/116562","cluster_addr":"172.21.15.110:6810/116562","heartbeat_back_addr":"172.21.15.110:6812/116562","heartbeat_front_addr":"172.21.15.110:6811/116562","state":["exists","up"]},{"osd":6,"uuid":"76676042-2074-4cf8-9ef9-5c87bf604944","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6813","nonce":116563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":116563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":116563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6815","nonce":116563}]},"public_addr":"172.21.15.110:6813/116563","cluster_addr":"172.21.15.110:6814/116563","heartbeat_back_addr":"172.21.15.110:6816/116563","heartbeat_front_addr":"172.21.15.110:6815/116563","state":["exists","up"]},{"osd":7,"uuid":"045479f4-f7f7-4b95-bc38-a2c52ac31b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6805","nonce":116561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":116561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":116561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6807","nonce":116561}]},"public_addr":"172.21.15.110:6805/116561","cluster_addr":"172.21.15.110:6806/116561","heartbeat_back_addr":"172.21.15.110:6808/116561","heartbeat_front_addr":"172.21.15.110:6807/116561","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:06:02.158984+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:06:01.902487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:06:01.310184+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:06:01.669641+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T02:06:07.403 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:07.404+0000 7f7984a80700 10 librados: watch_flush enter 2023-09-30T02:06:07.404 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:07.404+0000 7f7984a80700 10 librados: watch_flush exit 2023-09-30T02:06:07.404 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:07.404+0000 7f7984a80700 20 client.4258.objecter shutdown clearing up homeless session... 2023-09-30T02:06:07.404 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:07.404+0000 7f7984a80700 10 client.4258.objecter successfully canceled tick 2023-09-30T02:06:07.405 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:07.404+0000 7f7984a80700 1 -- 172.21.15.59:0/919998704 >> v2:172.21.15.110:6800/115870 conn(0x7f7960068880 msgr2=0x7f796006ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:07.405 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:07.404+0000 7f7984a80700 1 --2- 172.21.15.59:0/919998704 >> v2:172.21.15.110:6800/115870 conn(0x7f7960068880 0x7f796006ad30 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f79800aae60 tx=0x7f79740091a0 comp rx=0 tx=0).stop 2023-09-30T02:06:07.405 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:07.404+0000 7f7984a80700 10 monclient: shutdown 2023-09-30T02:06:07.405 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:07.404+0000 7f7984a80700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:07.405 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:07.404+0000 7f7984a80700 1 -- 172.21.15.59:0/919998704 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f79800a56f0 msgr2=0x7f79800a93c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:07.405 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:07.404+0000 7f7984a80700 1 --2- 172.21.15.59:0/919998704 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f79800a56f0 0x7f79800a93c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f79700093f0 tx=0x7f7970017000 comp rx=0 tx=0).stop 2023-09-30T02:06:07.405 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:07.405+0000 7f7984a80700 1 -- 172.21.15.59:0/919998704 shutdown_connections 2023-09-30T02:06:07.405 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:07.405+0000 7f7984a80700 1 --2- 172.21.15.59:0/919998704 >> v2:172.21.15.110:6800/115870 conn(0x7f7960068880 0x7f796006ad30 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:07.406 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:07.405+0000 7f7984a80700 1 --2- 172.21.15.59:0/919998704 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f79800a56f0 0x7f79800a93c0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:07.406 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:07.405+0000 7f7984a80700 1 --2- 172.21.15.59:0/919998704 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7980141c80 0x7f79800a9e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:07.406 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:07.405+0000 7f7984a80700 1 --2- 172.21.15.59:0/919998704 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f79800b13e0 0x7f798013de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:07.406 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:07.405+0000 7f7984a80700 1 -- 172.21.15.59:0/919998704 >> 172.21.15.59:0/919998704 conn(0x7f7980063280 msgr2=0x7f79800b4220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:07.406 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:07.405+0000 7f7984a80700 1 -- 172.21.15.59:0/919998704 shutdown_connections 2023-09-30T02:06:07.406 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:07.405+0000 7f7984a80700 1 -- 172.21.15.59:0/919998704 wait complete. 2023-09-30T02:06:07.406 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:07.405+0000 7f7984a80700 1 librados: shutdown 2023-09-30T02:06:07.423 DEBUG:teuthology.misc:4 of 8 OSDs are up 2023-09-30T02:06:07.611 INFO:tasks.ceph.osd.0.smithi059.stderr:2023-09-30T02:06:07.604+0000 e689700 -1 osd.0 0 waiting for initial osdmap 2023-09-30T02:06:07.644 INFO:tasks.ceph.osd.1.smithi059.stderr:2023-09-30T02:06:07.639+0000 fe8c700 -1 osd.1 0 waiting for initial osdmap 2023-09-30T02:06:07.691 INFO:tasks.ceph.osd.2.smithi059.stderr:2023-09-30T02:06:07.687+0000 e689700 -1 osd.2 0 waiting for initial osdmap 2023-09-30T02:06:08.347 INFO:tasks.ceph.osd.0.smithi059.stderr:2023-09-30T02:06:08.329+0000 14890700 -1 osd.0 16 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T02:06:08.407 INFO:tasks.ceph.osd.1.smithi059.stderr:2023-09-30T02:06:08.388+0000 15091700 -1 osd.1 16 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T02:06:08.452 INFO:tasks.ceph.osd.2.smithi059.stderr:2023-09-30T02:06:08.433+0000 14890700 -1 osd.2 16 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T02:06:08.640 INFO:tasks.ceph.osd.3.smithi059.stderr:2023-09-30T02:06:08.637+0000 e689700 -1 osd.3 0 waiting for initial osdmap 2023-09-30T02:06:09.372 INFO:tasks.ceph.osd.3.smithi059.stderr:2023-09-30T02:06:09.355+0000 14890700 -1 osd.3 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T02:06:13.030 INFO:tasks.ceph.mon.a.smithi059.stderr:==00:00:01:22.896 116256== Warning: unimplemented fcntl command: 1036 2023-09-30T02:06:13.088 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:01:22.954 116257== Warning: unimplemented fcntl command: 1036 2023-09-30T02:06:13.111 INFO:tasks.ceph.mon.a.smithi059.stderr:==00:00:01:22.977 116256== Warning: unimplemented fcntl command: 1036 2023-09-30T02:06:13.130 INFO:tasks.ceph.mon.b.smithi110.stderr:==00:00:01:22.978 115869== Warning: unimplemented fcntl command: 1036 2023-09-30T02:06:13.182 INFO:tasks.ceph.mon.b.smithi110.stderr:==00:00:01:23.029 115869== Warning: unimplemented fcntl command: 1036 2023-09-30T02:06:13.217 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:01:23.083 116257== Warning: unimplemented fcntl command: 1036 2023-09-30T02:06:13.456 INFO:tasks.ceph.mon.a.smithi059.stderr:==00:00:01:23.321 116256== Warning: unimplemented fcntl command: 1036 2023-09-30T02:06:13.465 INFO:tasks.ceph.mon.b.smithi110.stderr:==00:00:01:23.312 115869== Warning: unimplemented fcntl command: 1036 2023-09-30T02:06:13.542 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:01:23.407 116257== Warning: unimplemented fcntl command: 1036 2023-09-30T02:06:14.229 DEBUG:teuthology.orchestra.run.smithi059:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-09-30T02:06:14.356 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.357+0000 7fae69d51700 10 monclient: get_monmap_and_config 2023-09-30T02:06:14.357 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.357+0000 7fae69d51700 10 monclient: build_initial_monmap 2023-09-30T02:06:14.357 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.357+0000 7fae69d51700 10 monclient: monmap: 2023-09-30T02:06:14.357 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2023-09-30T02:06:14.357 INFO:teuthology.misc.health.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:14.357 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2023-09-30T02:06:14.357628+0000 2023-09-30T02:06:14.357 INFO:teuthology.misc.health.smithi059.stderr:created 2023-09-30T02:06:14.357628+0000 2023-09-30T02:06:14.357 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:14.357 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:14.357 INFO:teuthology.misc.health.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:06:14.357 INFO:teuthology.misc.health.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:06:14.357 INFO:teuthology.misc.health.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:06:14.357 INFO:teuthology.misc.health.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:06:14.358 INFO:teuthology.misc.health.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:06:14.358 INFO:teuthology.misc.health.smithi059.stderr: 2023-09-30T02:06:14.359 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.358+0000 7fae69d51700 1 Processor -- start 2023-09-30T02:06:14.359 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.358+0000 7fae69d51700 1 -- start start 2023-09-30T02:06:14.360 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.358+0000 7fae69d51700 10 monclient: init 2023-09-30T02:06:14.360 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.358+0000 7fae69d51700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:14.360 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.358+0000 7fae69d51700 10 monclient: _add_conns ranks=[3,2,0,4,1] 2023-09-30T02:06:14.360 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.358+0000 7fae69d51700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fae640b1660 0x7fae640a54e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:14.360 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.358+0000 7fae69d51700 10 monclient(hunting): picked mon.noname-a con 0x7fae640b1660 addr v2:172.21.15.59:3300/0 2023-09-30T02:06:14.360 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.358+0000 7fae69d51700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fae640a7630 0x7fae640a7a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:14.360 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.358+0000 7fae69d51700 10 monclient(hunting): picked mon.noname-b con 0x7fae640a7630 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:14.360 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.358+0000 7fae69d51700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fae640a7f80 addr v1:172.21.15.59:6789/0 2023-09-30T02:06:14.360 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.358+0000 7fae69d51700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae640a5e40 con 0x7fae640a7f80 2023-09-30T02:06:14.360 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.358+0000 7fae69d51700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:14.360 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.358+0000 7fae69d51700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae640b1a30 con 0x7fae640b1660 2023-09-30T02:06:14.360 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.358+0000 7fae69d51700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:14.361 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.358+0000 7fae69d51700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae640a83e0 con 0x7fae640a7630 2023-09-30T02:06:14.361 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.358+0000 7fae69d51700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:14.361 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.358+0000 7fae69d51700 10 monclient(hunting): authenticate will time out at 1819.660852s 2023-09-30T02:06:14.361 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.359+0000 7fae637fe700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fae640b1660 0x7fae640a54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:14.361 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.359+0000 7fae62ffd700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fae640a7630 0x7fae640a7a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:14.361 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.360+0000 7fae62ffd700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fae640a7630 0x7fae640a7a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:42922/0 (socket says 172.21.15.59:42922) 2023-09-30T02:06:14.361 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.360+0000 7fae62ffd700 1 -- 172.21.15.59:0/303653273 learned_addr learned my addr 172.21.15.59:0/303653273 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:06:14.361 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.360+0000 7fae62ffd700 10 monclient(hunting): get_auth_request con 0x7fae640a7630 auth_method 0 2023-09-30T02:06:14.361 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.360+0000 7fae62ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:14.361 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.360+0000 7fae62ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:14.361 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.360+0000 7fae62ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:14.362 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.360+0000 7fae637fe700 10 monclient(hunting): get_auth_request con 0x7fae640b1660 auth_method 0 2023-09-30T02:06:14.362 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.360+0000 7fae637fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:14.362 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.360+0000 7fae637fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:14.362 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.360+0000 7fae637fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:14.362 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.361+0000 7fae62ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:14.363 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.361+0000 7fae62ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:14.363 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.361+0000 7fae62ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:14.363 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.361+0000 7fae637fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:14.363 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.362+0000 7fae637fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:14.363 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.362+0000 7fae637fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:14.364 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.364+0000 7fae62ffd700 10 monclient(hunting): handle_auth_done global_id 4220 payload 274 2023-09-30T02:06:14.364 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.364+0000 7fae62ffd700 1 -- 172.21.15.59:0/303653273 >> v2:172.21.15.59:3300/0 conn(0x7fae640b1660 msgr2=0x7fae640a54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:06:14.364 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.364+0000 7fae62ffd700 1 --2- 172.21.15.59:0/303653273 >> v2:172.21.15.59:3300/0 conn(0x7fae640b1660 0x7fae640a54e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:14.364 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.364+0000 7fae62ffd700 1 -- 172.21.15.59:0/303653273 >> v1:172.21.15.59:6789/0 conn(0x7fae640a7f80 legacy=0x7fae640af430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:06:14.364 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.364+0000 7fae62ffd700 10 monclient: _finish_hunting 0 2023-09-30T02:06:14.364 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.364+0000 7fae62ffd700 1 monclient: found mon.noname-b 2023-09-30T02:06:14.364 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.364+0000 7fae62ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:14.364 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.364+0000 7fae62ffd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.59:3301/0 2023-09-30T02:06:14.364 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.364+0000 7fae62ffd700 1 -- 172.21.15.59:0/303653273 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae640a6f30 con 0x7fae640a7630 2023-09-30T02:06:14.366 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.367+0000 7fae62ffd700 1 --2- 172.21.15.59:0/303653273 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fae640a7630 0x7fae640a7a40 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fae580099b0 tx=0x7fae58009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e825ff1c8ca5bf7 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:14.372 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.372+0000 7fae627fc700 1 -- 172.21.15.59:0/303653273 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae58009ea0 con 0x7fae640a7630 2023-09-30T02:06:14.372 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.373+0000 7fae627fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:14.372 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.373+0000 7fae627fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:06:14.372 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.373+0000 7fae627fc700 10 monclient: dump: 2023-09-30T02:06:14.372 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2023-09-30T02:06:14.372 INFO:teuthology.misc.health.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:14.373 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:14.373 INFO:teuthology.misc.health.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:14.373 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:14.373 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:14.373 INFO:teuthology.misc.health.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:14.373 INFO:teuthology.misc.health.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:14.373 INFO:teuthology.misc.health.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:14.373 INFO:teuthology.misc.health.smithi059.stderr: 2023-09-30T02:06:14.373 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.373+0000 7fae627fc700 10 monclient: _finish_auth 0 2023-09-30T02:06:14.373 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.373+0000 7fae627fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:14.373 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.373+0000 7fae627fc700 1 -- 172.21.15.59:0/303653273 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae5800b820 con 0x7fae640a7630 2023-09-30T02:06:14.374 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.373+0000 7fae69d51700 5 monclient: authenticate success, global_id 4220 2023-09-30T02:06:14.374 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.373+0000 7fae627fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:14.374 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.373+0000 7fae69d51700 10 monclient: get_monmap_and_config success 2023-09-30T02:06:14.374 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.373+0000 7fae627fc700 1 -- 172.21.15.59:0/303653273 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae5800bbc0 con 0x7fae640a7630 2023-09-30T02:06:14.374 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.373+0000 7fae69d51700 4 set_mon_vals no callback set 2023-09-30T02:06:14.374 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.373+0000 7fae627fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:14.374 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.373+0000 7fae627fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:06:14.374 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.373+0000 7fae627fc700 10 monclient: dump: 2023-09-30T02:06:14.374 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2023-09-30T02:06:14.374 INFO:teuthology.misc.health.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:14.374 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:14.374 INFO:teuthology.misc.health.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:14.375 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:14.375 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:14.375 INFO:teuthology.misc.health.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:14.375 INFO:teuthology.misc.health.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:14.375 INFO:teuthology.misc.health.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:14.375 INFO:teuthology.misc.health.smithi059.stderr: 2023-09-30T02:06:14.378 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.373+0000 7fae69d51700 10 monclient: shutdown 2023-09-30T02:06:14.378 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.373+0000 7fae69d51700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:14.378 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.373+0000 7fae69d51700 1 -- 172.21.15.59:0/303653273 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fae640a7630 msgr2=0x7fae640a7a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:14.378 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.373+0000 7fae69d51700 1 --2- 172.21.15.59:0/303653273 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fae640a7630 0x7fae640a7a40 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fae580099b0 tx=0x7fae58009250 comp rx=0 tx=0).stop 2023-09-30T02:06:14.378 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.373+0000 7fae69d51700 1 -- 172.21.15.59:0/303653273 shutdown_connections 2023-09-30T02:06:14.378 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.373+0000 7fae69d51700 1 --2- 172.21.15.59:0/303653273 >> v2:172.21.15.59:3300/0 conn(0x7fae640b1660 0x7fae640a54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:14.378 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.373+0000 7fae69d51700 1 --2- 172.21.15.59:0/303653273 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fae640a7630 0x7fae640a7a40 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:14.379 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.373+0000 7fae69d51700 1 -- 172.21.15.59:0/303653273 >> 172.21.15.59:0/303653273 conn(0x7fae64069130 msgr2=0x7fae64063280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:14.379 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.373+0000 7fae69d51700 1 -- 172.21.15.59:0/303653273 shutdown_connections 2023-09-30T02:06:14.379 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.374+0000 7fae69d51700 1 -- 172.21.15.59:0/303653273 wait complete. 2023-09-30T02:06:14.379 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.374+0000 7fae69d51700 10 monclient: build_initial_monmap 2023-09-30T02:06:14.379 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.374+0000 7fae69d51700 10 monclient: monmap: 2023-09-30T02:06:14.379 INFO:teuthology.misc.health.smithi059.stderr:epoch 0 2023-09-30T02:06:14.379 INFO:teuthology.misc.health.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:06:14.379 INFO:teuthology.misc.health.smithi059.stderr:last_changed 0.000000 2023-09-30T02:06:14.379 INFO:teuthology.misc.health.smithi059.stderr:created 0.000000 2023-09-30T02:06:14.379 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:14.379 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 1 2023-09-30T02:06:14.379 INFO:teuthology.misc.health.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:06:14.380 INFO:teuthology.misc.health.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:06:14.380 INFO:teuthology.misc.health.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:06:14.380 INFO:teuthology.misc.health.smithi059.stderr: 2023-09-30T02:06:14.380 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.374+0000 7fae69d51700 1 librados: starting msgr at 2023-09-30T02:06:14.380 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.374+0000 7fae69d51700 1 librados: starting objecter 2023-09-30T02:06:14.380 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.375+0000 7fae69d51700 1 Processor -- start 2023-09-30T02:06:14.380 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.375+0000 7fae69d51700 1 -- start start 2023-09-30T02:06:14.380 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.375+0000 7fae69d51700 1 librados: setting wanted keys 2023-09-30T02:06:14.380 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.375+0000 7fae69d51700 1 librados: calling monclient init 2023-09-30T02:06:14.380 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.375+0000 7fae69d51700 10 monclient: init 2023-09-30T02:06:14.380 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.376+0000 7fae69d51700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:14.380 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.376+0000 7fae69d51700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T02:06:14.380 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.376+0000 7fae69d51700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fae640a7630 0x7fae64140ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:14.381 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.376+0000 7fae69d51700 10 monclient(hunting): picked mon.noname-b con 0x7fae640a7630 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:06:14.381 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.376+0000 7fae69d51700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fae640a7f80 0x7fae6413de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:14.381 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.376+0000 7fae69d51700 10 monclient(hunting): picked mon.noname-a con 0x7fae640a7f80 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:06:14.381 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.376+0000 7fae69d51700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fae640b1660 0x7fae64135c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:14.381 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.376+0000 7fae69d51700 10 monclient(hunting): picked mon.noname-c con 0x7fae640b1660 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:14.381 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.376+0000 7fae69d51700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:14.381 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.376+0000 7fae69d51700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae641417b0 con 0x7fae640a7f80 2023-09-30T02:06:14.381 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.376+0000 7fae69d51700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:14.381 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.376+0000 7fae69d51700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae6413f030 con 0x7fae640a7630 2023-09-30T02:06:14.381 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.376+0000 7fae69d51700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:14.381 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.376+0000 7fae69d51700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae6413eab0 con 0x7fae640b1660 2023-09-30T02:06:14.382 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.376+0000 7fae69d51700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:14.382 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.376+0000 7fae69d51700 10 monclient(hunting): authenticate will time out at 1819.678170s 2023-09-30T02:06:14.382 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.376+0000 7fae637fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fae640a7630 0x7fae64140ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:14.382 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.376+0000 7fae63fff700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fae640b1660 0x7fae64135c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:14.382 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.376+0000 7fae62ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fae640a7f80 0x7fae6413de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:14.382 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.377+0000 7fae62ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fae640a7f80 0x7fae6413de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38640/0 (socket says 172.21.15.59:38640) 2023-09-30T02:06:14.382 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.377+0000 7fae62ffd700 1 -- 172.21.15.59:0/3537140622 learned_addr learned my addr 172.21.15.59:0/3537140622 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:06:14.382 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.377+0000 7fae62ffd700 10 monclient(hunting): get_auth_request con 0x7fae640a7f80 auth_method 0 2023-09-30T02:06:14.382 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.377+0000 7fae62ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:14.382 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.377+0000 7fae62ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:14.382 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.377+0000 7fae62ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:14.382 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.377+0000 7fae637fe700 10 monclient(hunting): get_auth_request con 0x7fae640a7630 auth_method 0 2023-09-30T02:06:14.383 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.377+0000 7fae637fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:14.383 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.377+0000 7fae637fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:14.383 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.377+0000 7fae637fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:14.383 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.377+0000 7fae63fff700 10 monclient(hunting): get_auth_request con 0x7fae640b1660 auth_method 0 2023-09-30T02:06:14.383 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.377+0000 7fae63fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:14.383 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.377+0000 7fae63fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:14.383 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.377+0000 7fae63fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:14.383 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.378+0000 7fae62ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:14.383 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.378+0000 7fae62ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:14.383 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.378+0000 7fae62ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:14.383 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.378+0000 7fae637fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:14.384 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.378+0000 7fae637fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:14.384 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.378+0000 7fae637fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:14.384 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.378+0000 7fae63fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:14.384 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.378+0000 7fae63fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:14.384 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.378+0000 7fae63fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:14.385 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.383+0000 7fae637fe700 10 monclient(hunting): handle_auth_done global_id 4261 payload 824 2023-09-30T02:06:14.385 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.383+0000 7fae637fe700 1 -- 172.21.15.59:0/3537140622 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fae640b1660 msgr2=0x7fae64135c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:14.385 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.383+0000 7fae637fe700 1 --2- 172.21.15.59:0/3537140622 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fae640b1660 0x7fae64135c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:14.385 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.383+0000 7fae637fe700 1 -- 172.21.15.59:0/3537140622 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fae640a7f80 msgr2=0x7fae6413de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:14.385 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.383+0000 7fae637fe700 1 --2- 172.21.15.59:0/3537140622 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fae640a7f80 0x7fae6413de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:14.385 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.383+0000 7fae637fe700 10 monclient: _finish_hunting 0 2023-09-30T02:06:14.385 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.383+0000 7fae637fe700 1 monclient: found mon.noname-b 2023-09-30T02:06:14.385 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.383+0000 7fae637fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:14.385 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.383+0000 7fae637fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:06:14.385 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.383+0000 7fae637fe700 1 -- 172.21.15.59:0/3537140622 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae580094f0 con 0x7fae640a7630 2023-09-30T02:06:14.386 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.386+0000 7fae637fe700 1 --2- 172.21.15.59:0/3537140622 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fae640a7630 0x7fae64140ab0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fae5400e900 tx=0x7fae540091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:14.386 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.386+0000 7fae4b7fe700 10 client.?.objecter ms_handle_connect 0x7fae640a7630 2023-09-30T02:06:14.386 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.386+0000 7fae4b7fe700 10 client.?.objecter resend_mon_ops 2023-09-30T02:06:14.391 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.391+0000 7fae4b7fe700 1 -- 172.21.15.59:0/3537140622 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae54013680 con 0x7fae640a7630 2023-09-30T02:06:14.391 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.391+0000 7fae4b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:14.392 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.391+0000 7fae4b7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:06:14.392 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.391+0000 7fae4b7fe700 10 monclient: dump: 2023-09-30T02:06:14.392 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2023-09-30T02:06:14.392 INFO:teuthology.misc.health.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:14.392 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:14.392 INFO:teuthology.misc.health.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:14.392 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:14.392 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:14.392 INFO:teuthology.misc.health.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:14.392 INFO:teuthology.misc.health.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:14.392 INFO:teuthology.misc.health.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:14.393 INFO:teuthology.misc.health.smithi059.stderr: 2023-09-30T02:06:14.393 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.391+0000 7fae4b7fe700 10 monclient: _finish_auth 0 2023-09-30T02:06:14.393 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.391+0000 7fae4b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:14.393 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.391+0000 7fae4b7fe700 1 -- 172.21.15.59:0/3537140622 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae540137e0 con 0x7fae640a7630 2023-09-30T02:06:14.393 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.391+0000 7fae69d51700 5 monclient: authenticate success, global_id 4261 2023-09-30T02:06:14.393 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.392+0000 7fae4b7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:14.393 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.392+0000 7fae4b7fe700 1 -- 172.21.15.59:0/3537140622 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae54013b80 con 0x7fae640a7630 2023-09-30T02:06:14.393 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.392+0000 7fae69d51700 10 monclient: _renew_subs 2023-09-30T02:06:14.393 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.392+0000 7fae69d51700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:14.393 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.392+0000 7fae69d51700 1 -- 172.21.15.59:0/3537140622 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae64136140 con 0x7fae640a7630 2023-09-30T02:06:14.393 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.392+0000 7fae4b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:14.393 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.392+0000 7fae4b7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:06:14.394 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.392+0000 7fae61ffb700 4 set_mon_vals no callback set 2023-09-30T02:06:14.394 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.392+0000 7fae4b7fe700 10 monclient: dump: 2023-09-30T02:06:14.394 INFO:teuthology.misc.health.smithi059.stderr:epoch 1 2023-09-30T02:06:14.394 INFO:teuthology.misc.health.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:14.394 INFO:teuthology.misc.health.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:14.394 INFO:teuthology.misc.health.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:14.394 INFO:teuthology.misc.health.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:14.394 INFO:teuthology.misc.health.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:14.394 INFO:teuthology.misc.health.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:14.394 INFO:teuthology.misc.health.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:14.394 INFO:teuthology.misc.health.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:14.394 INFO:teuthology.misc.health.smithi059.stderr: 2023-09-30T02:06:14.395 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.392+0000 7fae69d51700 10 client.4261.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:06:14.395 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.392+0000 7fae69d51700 10 monclient: _renew_subs 2023-09-30T02:06:14.395 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.392+0000 7fae69d51700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:14.395 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.392+0000 7fae69d51700 1 -- 172.21.15.59:0/3537140622 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae64136600 con 0x7fae640a7630 2023-09-30T02:06:14.395 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.392+0000 7fae69d51700 1 librados: init done 2023-09-30T02:06:14.395 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.393+0000 7fae69d51700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:14.395 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.393+0000 7fae69d51700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:14.396 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.393+0000 7fae69d51700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:14.396 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.393+0000 7fae69d51700 1 -- 172.21.15.59:0/3537140622 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae64141a90 con 0x7fae640a7630 2023-09-30T02:06:14.424 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.425+0000 7fae4b7fe700 1 -- 172.21.15.59:0/3537140622 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 4) v1 ==== 84649+0+0 (secure 0 0 0) 0x7fae540253d0 con 0x7fae640a7630 2023-09-30T02:06:14.426 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.426+0000 7fae4b7fe700 1 --2- 172.21.15.59:0/3537140622 >> v2:172.21.15.110:6800/115870 conn(0x7fae50068950 0x7fae5006ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:14.426 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.426+0000 7fae4b7fe700 1 -- 172.21.15.59:0/3537140622 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 3733+0+0 (secure 0 0 0) 0x7fae540843f0 con 0x7fae640a7630 2023-09-30T02:06:14.426 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.426+0000 7fae4b7fe700 10 client.4261.objecter ms_dispatch 0x7fae64001550 osd_map(21..21 src has 1..21) v4 2023-09-30T02:06:14.426 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.426+0000 7fae4b7fe700 3 client.4261.objecter handle_osd_map got epochs [21,21] > 0 2023-09-30T02:06:14.426 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.426+0000 7fae4b7fe700 3 client.4261.objecter handle_osd_map decoding full epoch 21 2023-09-30T02:06:14.426 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.426+0000 7fae4b7fe700 20 client.4261.objecter dump_active .. 0 homeless 2023-09-30T02:06:14.426 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.426+0000 7fae62ffd700 1 --2- 172.21.15.59:0/3537140622 >> v2:172.21.15.110:6800/115870 conn(0x7fae50068950 0x7fae5006ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:14.426 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.426+0000 7fae62ffd700 10 monclient: get_auth_request con 0x7fae50068950 auth_method 0 2023-09-30T02:06:14.426 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.426+0000 7fae62ffd700 1 --2- 172.21.15.59:0/3537140622 >> v2:172.21.15.110:6800/115870 conn(0x7fae50068950 0x7fae5006ae00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fae6413e740 tx=0x7fae58016000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:14.426 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.426+0000 7fae4b7fe700 10 client.4261.objecter ms_handle_connect 0x7fae50068950 2023-09-30T02:06:14.675 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.676+0000 7fae4b7fe700 1 -- 172.21.15.59:0/3537140622 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fae54051ed0 con 0x7fae640a7630 2023-09-30T02:06:14.675 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.676+0000 7fae4b7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:14.675 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.676+0000 7fae4b7fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:06:14.942 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.943+0000 7fae69d51700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-09-30T02:06:14.942 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.943+0000 7fae69d51700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T02:06:14.942 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.943+0000 7fae69d51700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:14.943 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.943+0000 7fae69d51700 1 -- 172.21.15.59:0/3537140622 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fae640b2da0 con 0x7fae640a7630 2023-09-30T02:06:14.973 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.973+0000 7fae4b7fe700 1 -- 172.21.15.59:0/3537140622 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+10416 (secure 0 0 0) 0x7fae54021090 con 0x7fae640a7630 2023-09-30T02:06:14.973 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.973+0000 7fae4b7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T02:06:14.973 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.973+0000 7fae4b7fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:06:14.974 INFO:teuthology.misc.health.smithi059.stdout: 2023-09-30T02:06:14.974 INFO:teuthology.misc.health.smithi059.stdout:{"epoch":21,"fsid":"623052cb-4f82-4b2b-b72b-3ed5be614613","created":"2023-09-30T02:05:11.679291+0000","modified":"2023-09-30T02:06:12.209821+0000","last_up_change":"2023-09-30T02:06:10.164005+0000","last_in_change":"2023-09-30T02:05:18.270595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-30T02:06:06.187725+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"efdab44b-be57-4b81-8567-ff2deec1d3e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":116664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6801","nonce":116664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6803","nonce":116664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":116664}]},"public_addr":"172.21.15.59:6800/116664","cluster_addr":"172.21.15.59:6801/116664","heartbeat_back_addr":"172.21.15.59:6803/116664","heartbeat_front_addr":"172.21.15.59:6802/116664","state":["exists","up"]},{"osd":1,"uuid":"eff9786d-8c39-4044-91d2-a31f33e3b16e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":116665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6809","nonce":116665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6811","nonce":116665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":116665}]},"public_addr":"172.21.15.59:6808/116665","cluster_addr":"172.21.15.59:6809/116665","heartbeat_back_addr":"172.21.15.59:6811/116665","heartbeat_front_addr":"172.21.15.59:6810/116665","state":["exists","up"]},{"osd":2,"uuid":"b276ed4b-317b-4ddf-96ab-7ad71b8e3ff7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":116666}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6805","nonce":116666}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6807","nonce":116666}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":116666}]},"public_addr":"172.21.15.59:6804/116666","cluster_addr":"172.21.15.59:6805/116666","heartbeat_back_addr":"172.21.15.59:6807/116666","heartbeat_front_addr":"172.21.15.59:6806/116666","state":["exists","up"]},{"osd":3,"uuid":"48bfaf43-5846-4385-b9c6-6bc424cd5acd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":116667}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6813","nonce":116667}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6815","nonce":116667}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":116667}]},"public_addr":"172.21.15.59:6812/116667","cluster_addr":"172.21.15.59:6813/116667","heartbeat_back_addr":"172.21.15.59:6815/116667","heartbeat_front_addr":"172.21.15.59:6814/116667","state":["exists","up"]},{"osd":4,"uuid":"76fa5443-1571-4e72-9d22-6ab19e59839a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6801","nonce":116560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":116560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":116560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6803","nonce":116560}]},"public_addr":"172.21.15.110:6801/116560","cluster_addr":"172.21.15.110:6802/116560","heartbeat_back_addr":"172.21.15.110:6804/116560","heartbeat_front_addr":"172.21.15.110:6803/116560","state":["exists","up"]},{"osd":5,"uuid":"e9436478-4140-4fd6-ba2e-04fa6f6cf367","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6809","nonce":116562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":116562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":116562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6811","nonce":116562}]},"public_addr":"172.21.15.110:6809/116562","cluster_addr":"172.21.15.110:6810/116562","heartbeat_back_addr":"172.21.15.110:6812/116562","heartbeat_front_addr":"172.21.15.110:6811/116562","state":["exists","up"]},{"osd":6,"uuid":"76676042-2074-4cf8-9ef9-5c87bf604944","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6813","nonce":116563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":116563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":116563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6815","nonce":116563}]},"public_addr":"172.21.15.110:6813/116563","cluster_addr":"172.21.15.110:6814/116563","heartbeat_back_addr":"172.21.15.110:6816/116563","heartbeat_front_addr":"172.21.15.110:6815/116563","state":["exists","up"]},{"osd":7,"uuid":"045479f4-f7f7-4b95-bc38-a2c52ac31b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6805","nonce":116561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":116561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":116561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6807","nonce":116561}]},"public_addr":"172.21.15.110:6805/116561","cluster_addr":"172.21.15.110:6806/116561","heartbeat_back_addr":"172.21.15.110:6808/116561","heartbeat_front_addr":"172.21.15.110:6807/116561","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:06:05.267167+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:06:06.183162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:06:05.646500+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:06:06.546668+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:06:02.158984+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:06:01.902487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:06:01.310184+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:06:01.669641+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T02:06:14.976 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.976+0000 7fae69d51700 10 librados: watch_flush enter 2023-09-30T02:06:14.976 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.976+0000 7fae69d51700 10 librados: watch_flush exit 2023-09-30T02:06:14.976 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.976+0000 7fae69d51700 20 client.4261.objecter shutdown clearing up homeless session... 2023-09-30T02:06:14.976 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.976+0000 7fae69d51700 10 client.4261.objecter successfully canceled tick 2023-09-30T02:06:14.976 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.976+0000 7fae69d51700 1 -- 172.21.15.59:0/3537140622 >> v2:172.21.15.110:6800/115870 conn(0x7fae50068950 msgr2=0x7fae5006ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:14.976 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.976+0000 7fae69d51700 1 --2- 172.21.15.59:0/3537140622 >> v2:172.21.15.110:6800/115870 conn(0x7fae50068950 0x7fae5006ae00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fae6413e740 tx=0x7fae58016000 comp rx=0 tx=0).stop 2023-09-30T02:06:14.976 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.976+0000 7fae69d51700 10 monclient: shutdown 2023-09-30T02:06:14.976 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.976+0000 7fae69d51700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:14.977 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.976+0000 7fae69d51700 1 -- 172.21.15.59:0/3537140622 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fae640a7630 msgr2=0x7fae64140ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:14.977 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.976+0000 7fae69d51700 1 --2- 172.21.15.59:0/3537140622 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fae640a7630 0x7fae64140ab0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fae5400e900 tx=0x7fae540091a0 comp rx=0 tx=0).stop 2023-09-30T02:06:14.977 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.976+0000 7fae69d51700 1 -- 172.21.15.59:0/3537140622 shutdown_connections 2023-09-30T02:06:14.977 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.976+0000 7fae69d51700 1 --2- 172.21.15.59:0/3537140622 >> v2:172.21.15.110:6800/115870 conn(0x7fae50068950 0x7fae5006ae00 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:14.977 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.976+0000 7fae69d51700 1 --2- 172.21.15.59:0/3537140622 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fae640a7630 0x7fae64140ab0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:14.977 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.976+0000 7fae69d51700 1 --2- 172.21.15.59:0/3537140622 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fae640b1660 0x7fae64135c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:14.977 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.976+0000 7fae69d51700 1 --2- 172.21.15.59:0/3537140622 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fae640a7f80 0x7fae6413de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:14.977 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.976+0000 7fae69d51700 1 -- 172.21.15.59:0/3537140622 >> 172.21.15.59:0/3537140622 conn(0x7fae64069130 msgr2=0x7fae640b4110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:14.977 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.976+0000 7fae69d51700 1 -- 172.21.15.59:0/3537140622 shutdown_connections 2023-09-30T02:06:14.977 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.976+0000 7fae69d51700 1 -- 172.21.15.59:0/3537140622 wait complete. 2023-09-30T02:06:14.977 INFO:teuthology.misc.health.smithi059.stderr:2023-09-30T02:06:14.976+0000 7fae69d51700 1 librados: shutdown 2023-09-30T02:06:14.988 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-09-30T02:06:14.989 INFO:tasks.ceph:Creating RBD pool 2023-09-30T02:06:14.989 DEBUG:teuthology.orchestra.run.smithi059:> sudo ceph --cluster ceph osd pool create rbd 8 2023-09-30T02:06:15.131 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.132+0000 7fd2b0c98700 10 monclient: get_monmap_and_config 2023-09-30T02:06:15.131 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.132+0000 7fd2b0c98700 10 monclient: build_initial_monmap 2023-09-30T02:06:15.132 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.132+0000 7fd2b0c98700 10 monclient: monmap: 2023-09-30T02:06:15.132 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:15.132 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:15.132 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:06:15.132662+0000 2023-09-30T02:06:15.132 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:06:15.132662+0000 2023-09-30T02:06:15.132 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:15.132 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:15.132 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:06:15.132 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:06:15.132 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:06:15.132 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:06:15.132 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:06:15.132 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:15.133 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.133+0000 7fd2b0c98700 1 Processor -- start 2023-09-30T02:06:15.133 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.133+0000 7fd2b0c98700 1 -- start start 2023-09-30T02:06:15.134 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.133+0000 7fd2b0c98700 10 monclient: init 2023-09-30T02:06:15.134 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.134+0000 7fd2b0c98700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:15.134 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.134+0000 7fd2b0c98700 10 monclient: _add_conns ranks=[3,2,0,1,4] 2023-09-30T02:06:15.134 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.134+0000 7fd2b0c98700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fd2ac0a5b70 0x7fd2ac0b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:15.134 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.134+0000 7fd2b0c98700 10 monclient(hunting): picked mon.noname-a con 0x7fd2ac0a5b70 addr v2:172.21.15.59:3300/0 2023-09-30T02:06:15.134 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.134+0000 7fd2b0c98700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd2ac0a5f40 0x7fd2ac141b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:15.134 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.134+0000 7fd2b0c98700 10 monclient(hunting): picked mon.noname-b con 0x7fd2ac0a5f40 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:15.134 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.134+0000 7fd2b0c98700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd2ac1420c0 addr v1:172.21.15.59:6789/0 2023-09-30T02:06:15.134 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.134+0000 7fd2b0c98700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2ac0a63b0 con 0x7fd2ac1420c0 2023-09-30T02:06:15.134 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.134+0000 7fd2b0c98700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:15.134 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.134+0000 7fd2b0c98700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2ac0a75a0 con 0x7fd2ac0a5b70 2023-09-30T02:06:15.135 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.134+0000 7fd2b0c98700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:15.135 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.134+0000 7fd2b0c98700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2ac0af3f0 con 0x7fd2ac0a5f40 2023-09-30T02:06:15.135 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.134+0000 7fd2b0c98700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:15.135 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.134+0000 7fd2b0c98700 10 monclient(hunting): authenticate will time out at 1820.436532s 2023-09-30T02:06:15.136 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.135+0000 7fd2a9d9b700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd2ac0a5f40 0x7fd2ac141b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:15.136 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.135+0000 7fd2aad9d700 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fd2ac1420c0 0x7fd2ac13e230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:49636/0 (socket says 172.21.15.59:49636) 2023-09-30T02:06:15.136 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.135+0000 7fd2aad9d700 1 -- 172.21.15.59:0/2881777025 learned_addr learned my addr 172.21.15.59:0/2881777025 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:06:15.136 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.135+0000 7fd2a9d9b700 10 monclient(hunting): get_auth_request con 0x7fd2ac0a5f40 auth_method 0 2023-09-30T02:06:15.136 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.135+0000 7fd2a9d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:15.136 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.135+0000 7fd2a9d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:15.136 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.135+0000 7fd2a9d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:15.136 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.136+0000 7fd2aa59c700 1 --2- 172.21.15.59:0/2881777025 >> v2:172.21.15.59:3300/0 conn(0x7fd2ac0a5b70 0x7fd2ac0b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:15.137 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.136+0000 7fd2a9d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:15.137 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.136+0000 7fd2a9d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:15.137 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.136+0000 7fd2a9d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:15.137 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.137+0000 7fd2aa59c700 10 monclient(hunting): get_auth_request con 0x7fd2ac0a5b70 auth_method 0 2023-09-30T02:06:15.137 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.137+0000 7fd2aa59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:15.137 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.137+0000 7fd2aa59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:15.137 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.137+0000 7fd2aa59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:15.137 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.137+0000 7fd2aa59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:15.138 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.137+0000 7fd2aa59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:15.138 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.137+0000 7fd2aa59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:15.138 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.139+0000 7fd2a9d9b700 10 monclient(hunting): handle_auth_done global_id 4232 payload 274 2023-09-30T02:06:15.138 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.139+0000 7fd2a9d9b700 1 -- 172.21.15.59:0/2881777025 >> v2:172.21.15.59:3300/0 conn(0x7fd2ac0a5b70 msgr2=0x7fd2ac0b1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:06:15.138 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.139+0000 7fd2a9d9b700 1 --2- 172.21.15.59:0/2881777025 >> v2:172.21.15.59:3300/0 conn(0x7fd2ac0a5b70 0x7fd2ac0b1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:15.139 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.139+0000 7fd2a9d9b700 1 -- 172.21.15.59:0/2881777025 >> v1:172.21.15.59:6789/0 conn(0x7fd2ac1420c0 legacy=0x7fd2ac13e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:15.139 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.139+0000 7fd2a9d9b700 10 monclient: _finish_hunting 0 2023-09-30T02:06:15.139 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.139+0000 7fd2a9d9b700 1 monclient: found mon.noname-b 2023-09-30T02:06:15.139 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.139+0000 7fd2a9d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:15.139 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.139+0000 7fd2a9d9b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.59:3301/0 2023-09-30T02:06:15.139 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.139+0000 7fd2a9d9b700 1 -- 172.21.15.59:0/2881777025 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2ac142500 con 0x7fd2ac0a5f40 2023-09-30T02:06:15.140 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.141+0000 7fd2a9d9b700 1 --2- 172.21.15.59:0/2881777025 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd2ac0a5f40 0x7fd2ac141b80 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd2940098e0 tx=0x7fd2940091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9336dc7420ad7814 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:15.145 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.146+0000 7fd2a959a700 1 -- 172.21.15.59:0/2881777025 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd29400b640 con 0x7fd2ac0a5f40 2023-09-30T02:06:15.146 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.146+0000 7fd2a959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:15.146 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.146+0000 7fd2a959a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:06:15.146 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.146+0000 7fd2a959a700 10 monclient: dump: 2023-09-30T02:06:15.146 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:15.146 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:15.146 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:15.146 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:15.146 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:15.146 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:15.146 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:15.146 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:15.147 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:15.147 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:15.147 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.146+0000 7fd2a959a700 10 monclient: _finish_auth 0 2023-09-30T02:06:15.147 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.146+0000 7fd2a959a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:15.147 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.146+0000 7fd2a959a700 1 -- 172.21.15.59:0/2881777025 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd29400b7a0 con 0x7fd2ac0a5f40 2023-09-30T02:06:15.147 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.146+0000 7fd2b0c98700 5 monclient: authenticate success, global_id 4232 2023-09-30T02:06:15.147 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.146+0000 7fd2a959a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:15.147 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.146+0000 7fd2b0c98700 10 monclient: get_monmap_and_config success 2023-09-30T02:06:15.147 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.146+0000 7fd2a959a700 1 -- 172.21.15.59:0/2881777025 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd29400bb40 con 0x7fd2ac0a5f40 2023-09-30T02:06:15.147 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.146+0000 7fd2a959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:15.147 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.146+0000 7fd2b0c98700 4 set_mon_vals no callback set 2023-09-30T02:06:15.147 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.146+0000 7fd2a959a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:06:15.147 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.146+0000 7fd2a959a700 10 monclient: dump: 2023-09-30T02:06:15.148 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:15.148 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:15.148 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:15.148 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:15.148 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:15.148 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:15.148 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:15.148 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:15.148 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:15.148 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:15.151 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.147+0000 7fd2b0c98700 10 monclient: shutdown 2023-09-30T02:06:15.151 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.147+0000 7fd2b0c98700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:15.152 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.147+0000 7fd2b0c98700 1 -- 172.21.15.59:0/2881777025 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd2ac0a5f40 msgr2=0x7fd2ac141b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:15.152 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.147+0000 7fd2b0c98700 1 --2- 172.21.15.59:0/2881777025 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd2ac0a5f40 0x7fd2ac141b80 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd2940098e0 tx=0x7fd2940091a0 comp rx=0 tx=0).stop 2023-09-30T02:06:15.152 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.147+0000 7fd2b0c98700 1 -- 172.21.15.59:0/2881777025 shutdown_connections 2023-09-30T02:06:15.152 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.147+0000 7fd2b0c98700 1 --2- 172.21.15.59:0/2881777025 >> v2:172.21.15.59:3300/0 conn(0x7fd2ac0a5b70 0x7fd2ac0b1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:15.152 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.147+0000 7fd2b0c98700 1 --2- 172.21.15.59:0/2881777025 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd2ac0a5f40 0x7fd2ac141b80 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:15.152 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.147+0000 7fd2b0c98700 1 -- 172.21.15.59:0/2881777025 >> 172.21.15.59:0/2881777025 conn(0x7fd2ac0631f0 msgr2=0x7fd2ac0635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:15.152 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.147+0000 7fd2b0c98700 1 -- 172.21.15.59:0/2881777025 shutdown_connections 2023-09-30T02:06:15.152 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.147+0000 7fd2b0c98700 1 -- 172.21.15.59:0/2881777025 wait complete. 2023-09-30T02:06:15.152 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.148+0000 7fd2b0c98700 10 monclient: build_initial_monmap 2023-09-30T02:06:15.152 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.148+0000 7fd2b0c98700 10 monclient: monmap: 2023-09-30T02:06:15.152 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:15.153 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:06:15.153 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:06:15.153 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:06:15.153 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:15.153 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:06:15.153 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:06:15.153 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:06:15.153 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:06:15.153 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:15.153 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.148+0000 7fd2b0c98700 1 librados: starting msgr at 2023-09-30T02:06:15.153 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.148+0000 7fd2b0c98700 1 librados: starting objecter 2023-09-30T02:06:15.153 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.148+0000 7fd2b0c98700 1 Processor -- start 2023-09-30T02:06:15.154 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.148+0000 7fd2b0c98700 1 -- start start 2023-09-30T02:06:15.154 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.148+0000 7fd2b0c98700 1 librados: setting wanted keys 2023-09-30T02:06:15.154 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.148+0000 7fd2b0c98700 1 librados: calling monclient init 2023-09-30T02:06:15.154 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.148+0000 7fd2b0c98700 10 monclient: init 2023-09-30T02:06:15.154 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.149+0000 7fd2b0c98700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:15.154 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.149+0000 7fd2b0c98700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T02:06:15.154 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.149+0000 7fd2b0c98700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd2ac0a5b70 0x7fd2ac0a7af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:15.154 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.149+0000 7fd2b0c98700 10 monclient(hunting): picked mon.noname-b con 0x7fd2ac0a5b70 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:06:15.154 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.149+0000 7fd2b0c98700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd2ac0a5f40 0x7fd2ac0a8030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:15.154 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.149+0000 7fd2b0c98700 10 monclient(hunting): picked mon.noname-c con 0x7fd2ac0a5f40 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:15.154 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.149+0000 7fd2b0c98700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd2ac1420c0 0x7fd2ac0b2f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:15.154 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.149+0000 7fd2b0c98700 10 monclient(hunting): picked mon.noname-a con 0x7fd2ac1420c0 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:06:15.155 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.149+0000 7fd2b0c98700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:15.155 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.149+0000 7fd2b0c98700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2ac0b2af0 con 0x7fd2ac1420c0 2023-09-30T02:06:15.155 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.149+0000 7fd2b0c98700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:15.155 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.149+0000 7fd2b0c98700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2ac0a9480 con 0x7fd2ac0a5b70 2023-09-30T02:06:15.155 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.149+0000 7fd2b0c98700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:15.155 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.149+0000 7fd2b0c98700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2ac0a8570 con 0x7fd2ac0a5f40 2023-09-30T02:06:15.155 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.149+0000 7fd2b0c98700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:15.155 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.149+0000 7fd2b0c98700 10 monclient(hunting): authenticate will time out at 1820.451372s 2023-09-30T02:06:15.155 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.150+0000 7fd2aad9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd2ac1420c0 0x7fd2ac0b2f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:15.155 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.150+0000 7fd2aa59c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd2ac0a5b70 0x7fd2ac0a7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:15.155 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.150+0000 7fd2aa59c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd2ac0a5b70 0x7fd2ac0a7af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:54938/0 (socket says 172.21.15.59:54938) 2023-09-30T02:06:15.155 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.150+0000 7fd2aa59c700 1 -- 172.21.15.59:0/338681324 learned_addr learned my addr 172.21.15.59:0/338681324 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:06:15.155 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.150+0000 7fd2aa59c700 10 monclient(hunting): get_auth_request con 0x7fd2ac0a5b70 auth_method 0 2023-09-30T02:06:15.156 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.150+0000 7fd2aa59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:15.156 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.150+0000 7fd2aa59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:15.156 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.150+0000 7fd2aa59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:15.156 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.150+0000 7fd2aad9d700 10 monclient(hunting): get_auth_request con 0x7fd2ac1420c0 auth_method 0 2023-09-30T02:06:15.156 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.150+0000 7fd2aad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:15.156 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.150+0000 7fd2aad9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:15.156 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.150+0000 7fd2aad9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:15.156 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.151+0000 7fd2aa59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:15.156 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.151+0000 7fd2aa59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:15.156 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.151+0000 7fd2aa59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:15.156 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.151+0000 7fd2aad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:15.156 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.151+0000 7fd2aad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:15.157 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.151+0000 7fd2aad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:15.157 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.151+0000 7fd2a9d9b700 1 --2- 172.21.15.59:0/338681324 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd2ac0a5f40 0x7fd2ac0a8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:15.157 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.152+0000 7fd2a9d9b700 10 monclient(hunting): get_auth_request con 0x7fd2ac0a5f40 auth_method 0 2023-09-30T02:06:15.157 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.152+0000 7fd2a9d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:15.157 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.152+0000 7fd2a9d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:15.157 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.152+0000 7fd2a9d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:15.158 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.153+0000 7fd2a9d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:15.158 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.153+0000 7fd2a9d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:15.158 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.153+0000 7fd2a9d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:15.158 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.154+0000 7fd2aa59c700 10 monclient(hunting): handle_auth_done global_id 4264 payload 824 2023-09-30T02:06:15.158 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.155+0000 7fd2aa59c700 1 -- 172.21.15.59:0/338681324 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd2ac0a5f40 msgr2=0x7fd2ac0a8030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:15.158 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.155+0000 7fd2aa59c700 1 --2- 172.21.15.59:0/338681324 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd2ac0a5f40 0x7fd2ac0a8030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:15.159 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.155+0000 7fd2aa59c700 1 -- 172.21.15.59:0/338681324 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd2ac1420c0 msgr2=0x7fd2ac0b2f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:15.159 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.155+0000 7fd2aa59c700 1 --2- 172.21.15.59:0/338681324 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd2ac1420c0 0x7fd2ac0b2f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:15.159 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.155+0000 7fd2aa59c700 10 monclient: _finish_hunting 0 2023-09-30T02:06:15.159 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.155+0000 7fd2aa59c700 1 monclient: found mon.noname-b 2023-09-30T02:06:15.159 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.155+0000 7fd2aa59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:15.159 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.155+0000 7fd2aa59c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:06:15.159 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.155+0000 7fd2aa59c700 1 -- 172.21.15.59:0/338681324 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd294009420 con 0x7fd2ac0a5b70 2023-09-30T02:06:15.159 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.157+0000 7fd2aa59c700 1 --2- 172.21.15.59:0/338681324 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd2ac0a5b70 0x7fd2ac0a7af0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd29c00ecb0 tx=0x7fd29c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:15.159 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.157+0000 7fd2a27fc700 10 client.?.objecter ms_handle_connect 0x7fd2ac0a5b70 2023-09-30T02:06:15.159 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.157+0000 7fd2a27fc700 10 client.?.objecter resend_mon_ops 2023-09-30T02:06:15.161 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.161+0000 7fd2a27fc700 1 -- 172.21.15.59:0/338681324 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd29c00c850 con 0x7fd2ac0a5b70 2023-09-30T02:06:15.161 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.161+0000 7fd2a27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:15.161 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.161+0000 7fd2a27fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:06:15.161 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.161+0000 7fd2a27fc700 10 monclient: dump: 2023-09-30T02:06:15.161 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:15.161 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:15.161 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:15.161 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:15.162 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:15.162 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:15.162 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:15.162 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:15.162 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:15.162 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:15.162 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.161+0000 7fd2a27fc700 10 monclient: _finish_auth 0 2023-09-30T02:06:15.162 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.161+0000 7fd2a27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:15.162 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.161+0000 7fd2a27fc700 1 -- 172.21.15.59:0/338681324 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd29c00c9b0 con 0x7fd2ac0a5b70 2023-09-30T02:06:15.162 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.161+0000 7fd2a27fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:15.162 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.161+0000 7fd2a27fc700 1 -- 172.21.15.59:0/338681324 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd29c00cd50 con 0x7fd2ac0a5b70 2023-09-30T02:06:15.162 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.161+0000 7fd2b0c98700 5 monclient: authenticate success, global_id 4264 2023-09-30T02:06:15.163 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.161+0000 7fd2b0c98700 10 monclient: _renew_subs 2023-09-30T02:06:15.163 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.161+0000 7fd2b0c98700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:15.163 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.161+0000 7fd2b0c98700 1 -- 172.21.15.59:0/338681324 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2ac0b34c0 con 0x7fd2ac0a5b70 2023-09-30T02:06:15.163 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.161+0000 7fd2a8d99700 4 set_mon_vals no callback set 2023-09-30T02:06:15.163 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.161+0000 7fd2a27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:15.163 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.161+0000 7fd2a27fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:06:15.163 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.161+0000 7fd2a27fc700 10 monclient: dump: 2023-09-30T02:06:15.163 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:15.163 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:15.163 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:15.163 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:15.163 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:15.164 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:15.164 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:15.164 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:15.164 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:15.164 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:15.164 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.161+0000 7fd2b0c98700 10 client.4264.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:06:15.164 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.161+0000 7fd2b0c98700 10 monclient: _renew_subs 2023-09-30T02:06:15.164 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.161+0000 7fd2b0c98700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:15.164 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.161+0000 7fd2b0c98700 1 -- 172.21.15.59:0/338681324 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2ac0b3a00 con 0x7fd2ac0a5b70 2023-09-30T02:06:15.164 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.161+0000 7fd2b0c98700 1 librados: init done 2023-09-30T02:06:15.165 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.162+0000 7fd2b0c98700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:15.165 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.162+0000 7fd2b0c98700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:15.165 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.162+0000 7fd2b0c98700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:15.165 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.162+0000 7fd2b0c98700 1 -- 172.21.15.59:0/338681324 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2ac0b28c0 con 0x7fd2ac0a5b70 2023-09-30T02:06:15.412 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.412+0000 7fd2a27fc700 1 -- 172.21.15.59:0/338681324 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 4) v1 ==== 84649+0+0 (secure 0 0 0) 0x7fd29c013680 con 0x7fd2ac0a5b70 2023-09-30T02:06:15.412 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.413+0000 7fd2a27fc700 1 --2- 172.21.15.59:0/338681324 >> v2:172.21.15.110:6800/115870 conn(0x7fd28c068800 0x7fd28c06acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:15.413 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.413+0000 7fd2a27fc700 1 -- 172.21.15.59:0/338681324 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 3733+0+0 (secure 0 0 0) 0x7fd29c084540 con 0x7fd2ac0a5b70 2023-09-30T02:06:15.413 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.413+0000 7fd2a27fc700 10 client.4264.objecter ms_dispatch 0x7fd2ac001550 osd_map(21..21 src has 1..21) v4 2023-09-30T02:06:15.413 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.413+0000 7fd2a27fc700 3 client.4264.objecter handle_osd_map got epochs [21,21] > 0 2023-09-30T02:06:15.413 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.413+0000 7fd2a27fc700 3 client.4264.objecter handle_osd_map decoding full epoch 21 2023-09-30T02:06:15.413 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.413+0000 7fd2a27fc700 20 client.4264.objecter dump_active .. 0 homeless 2023-09-30T02:06:15.413 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.413+0000 7fd2a9d9b700 1 --2- 172.21.15.59:0/338681324 >> v2:172.21.15.110:6800/115870 conn(0x7fd28c068800 0x7fd28c06acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:15.413 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.413+0000 7fd2a9d9b700 10 monclient: get_auth_request con 0x7fd28c068800 auth_method 0 2023-09-30T02:06:15.413 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.414+0000 7fd2a27fc700 10 client.4264.objecter ms_handle_connect 0x7fd28c068800 2023-09-30T02:06:15.414 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.414+0000 7fd2a9d9b700 1 --2- 172.21.15.59:0/338681324 >> v2:172.21.15.110:6800/115870 conn(0x7fd28c068800 0x7fd28c06acb0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fd2940093f0 tx=0x7fd294017000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:15.420 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.421+0000 7fd2a27fc700 1 -- 172.21.15.59:0/338681324 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd29c052020 con 0x7fd2ac0a5b70 2023-09-30T02:06:15.420 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.421+0000 7fd2a27fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:15.420 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.421+0000 7fd2a27fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:06:15.666 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.666+0000 7fd2b0c98700 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2023-09-30T02:06:15.666 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.666+0000 7fd2b0c98700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2023-09-30T02:06:15.666 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.666+0000 7fd2b0c98700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:15.666 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:15.666+0000 7fd2b0c98700 1 -- 172.21.15.59:0/338681324 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7fd2ac0b28c0 con 0x7fd2ac0a5b70 2023-09-30T02:06:16.322 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.323+0000 7fd2a27fc700 1 -- 172.21.15.59:0/338681324 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fd29c051ad0 con 0x7fd2ac0a5b70 2023-09-30T02:06:16.349 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.350+0000 7fd2a27fc700 1 -- 172.21.15.59:0/338681324 <== mon.1 v2:172.21.15.110:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v22) v1 ==== 109+0+0 (secure 0 0 0) 0x7fd29c051e50 con 0x7fd2ac0a5b70 2023-09-30T02:06:16.349 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.350+0000 7fd2a27fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2023-09-30T02:06:16.349 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.350+0000 7fd2a27fc700 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2023-09-30T02:06:16.353 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.353+0000 7fd2b0c98700 10 librados: watch_flush enter 2023-09-30T02:06:16.353 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.353+0000 7fd2b0c98700 10 librados: watch_flush exit 2023-09-30T02:06:16.353 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.353+0000 7fd2b0c98700 20 client.4264.objecter shutdown clearing up homeless session... 2023-09-30T02:06:16.353 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.353+0000 7fd2b0c98700 10 client.4264.objecter successfully canceled tick 2023-09-30T02:06:16.353 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.353+0000 7fd2b0c98700 1 -- 172.21.15.59:0/338681324 >> v2:172.21.15.110:6800/115870 conn(0x7fd28c068800 msgr2=0x7fd28c06acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:16.353 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.353+0000 7fd2b0c98700 1 --2- 172.21.15.59:0/338681324 >> v2:172.21.15.110:6800/115870 conn(0x7fd28c068800 0x7fd28c06acb0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fd2940093f0 tx=0x7fd294017000 comp rx=0 tx=0).stop 2023-09-30T02:06:16.353 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.353+0000 7fd2b0c98700 10 monclient: shutdown 2023-09-30T02:06:16.354 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.353+0000 7fd2b0c98700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:16.354 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.353+0000 7fd2b0c98700 1 -- 172.21.15.59:0/338681324 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd2ac0a5b70 msgr2=0x7fd2ac0a7af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:16.354 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.353+0000 7fd2b0c98700 1 --2- 172.21.15.59:0/338681324 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd2ac0a5b70 0x7fd2ac0a7af0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd29c00ecb0 tx=0x7fd29c0091a0 comp rx=0 tx=0).stop 2023-09-30T02:06:16.354 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.353+0000 7fd2b0c98700 1 -- 172.21.15.59:0/338681324 shutdown_connections 2023-09-30T02:06:16.354 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.353+0000 7fd2b0c98700 1 --2- 172.21.15.59:0/338681324 >> v2:172.21.15.110:6800/115870 conn(0x7fd28c068800 0x7fd28c06acb0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:16.354 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.353+0000 7fd2b0c98700 1 --2- 172.21.15.59:0/338681324 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd2ac0a5b70 0x7fd2ac0a7af0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:16.354 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.353+0000 7fd2b0c98700 1 --2- 172.21.15.59:0/338681324 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd2ac1420c0 0x7fd2ac0b2f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:16.354 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.353+0000 7fd2b0c98700 1 --2- 172.21.15.59:0/338681324 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd2ac0a5f40 0x7fd2ac0a8030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:16.354 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.353+0000 7fd2b0c98700 1 -- 172.21.15.59:0/338681324 >> 172.21.15.59:0/338681324 conn(0x7fd2ac0631f0 msgr2=0x7fd2ac0af560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:16.354 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.353+0000 7fd2b0c98700 1 -- 172.21.15.59:0/338681324 shutdown_connections 2023-09-30T02:06:16.355 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.353+0000 7fd2b0c98700 1 -- 172.21.15.59:0/338681324 wait complete. 2023-09-30T02:06:16.355 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.353+0000 7fd2b0c98700 1 librados: shutdown 2023-09-30T02:06:16.355 INFO:teuthology.orchestra.run.smithi059.stderr:pool 'rbd' created 2023-09-30T02:06:16.378 DEBUG:teuthology.orchestra.run.smithi059:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-09-30T02:06:16.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.526+0000 7fc4d39d4700 10 monclient: get_monmap_and_config 2023-09-30T02:06:16.526 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.526+0000 7fc4d39d4700 10 monclient: build_initial_monmap 2023-09-30T02:06:16.526 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.526+0000 7fc4d39d4700 10 monclient: monmap: 2023-09-30T02:06:16.526 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:16.526 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:16.526 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:06:16.526745+0000 2023-09-30T02:06:16.526 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:06:16.526745+0000 2023-09-30T02:06:16.526 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:16.526 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:16.526 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:06:16.527 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:06:16.527 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:06:16.527 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:06:16.527 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:06:16.527 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:16.527 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.526+0000 7fc4d39d4700 1 Processor -- start 2023-09-30T02:06:16.529 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.527+0000 7fc4d39d4700 1 -- start start 2023-09-30T02:06:16.529 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.527+0000 7fc4d39d4700 10 monclient: init 2023-09-30T02:06:16.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.527+0000 7fc4d39d4700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:16.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.527+0000 7fc4d39d4700 10 monclient: _add_conns ranks=[4,3,2,1,0] 2023-09-30T02:06:16.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.527+0000 7fc4d39d4700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fc4cc0a5730 0x7fc4cc0a7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:16.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.527+0000 7fc4d39d4700 10 monclient(hunting): picked mon.noname-c con 0x7fc4cc0a5730 addr v2:172.21.15.110:3300/0 2023-09-30T02:06:16.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.527+0000 7fc4d39d4700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc4cc1418e0 0x7fc4cc13ae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:16.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.527+0000 7fc4d39d4700 10 monclient(hunting): picked mon.noname-a con 0x7fc4cc1418e0 addr v2:172.21.15.59:3300/0 2023-09-30T02:06:16.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.527+0000 7fc4d39d4700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc4cc13b360 0x7fc4cc13b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:16.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.527+0000 7fc4d39d4700 10 monclient(hunting): picked mon.noname-b con 0x7fc4cc13b360 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:16.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.527+0000 7fc4d39d4700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:16.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.527+0000 7fc4d39d4700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4cc0a7410 con 0x7fc4cc1418e0 2023-09-30T02:06:16.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.527+0000 7fc4d39d4700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:16.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.527+0000 7fc4d39d4700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4cc0a84e0 con 0x7fc4cc0a5730 2023-09-30T02:06:16.531 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.527+0000 7fc4d39d4700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:16.531 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.527+0000 7fc4d39d4700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4cc0a7b10 con 0x7fc4cc13b360 2023-09-30T02:06:16.531 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.527+0000 7fc4d39d4700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:16.531 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.527+0000 7fc4d39d4700 10 monclient(hunting): authenticate will time out at 1821.829975s 2023-09-30T02:06:16.531 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.528+0000 7fc4d1770700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fc4cc0a5730 0x7fc4cc0a7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:16.531 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.528+0000 7fc4d0f6f700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc4cc1418e0 0x7fc4cc13ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:16.531 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.528+0000 7fc4d1f71700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc4cc13b360 0x7fc4cc13b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:16.531 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.529+0000 7fc4d1770700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fc4cc0a5730 0x7fc4cc0a7fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:54940/0 (socket says 172.21.15.59:54940) 2023-09-30T02:06:16.531 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.529+0000 7fc4d1770700 1 -- 172.21.15.59:0/3102319175 learned_addr learned my addr 172.21.15.59:0/3102319175 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:06:16.531 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.529+0000 7fc4d1770700 10 monclient(hunting): get_auth_request con 0x7fc4cc0a5730 auth_method 0 2023-09-30T02:06:16.531 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.529+0000 7fc4d1770700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:16.532 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.529+0000 7fc4d1770700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:16.532 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.529+0000 7fc4d1770700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:16.532 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.529+0000 7fc4d1f71700 10 monclient(hunting): get_auth_request con 0x7fc4cc13b360 auth_method 0 2023-09-30T02:06:16.532 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.529+0000 7fc4d1f71700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:16.532 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.529+0000 7fc4d1f71700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:16.532 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.529+0000 7fc4d1f71700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:16.532 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.529+0000 7fc4d0f6f700 10 monclient(hunting): get_auth_request con 0x7fc4cc1418e0 auth_method 0 2023-09-30T02:06:16.532 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.529+0000 7fc4d0f6f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:16.532 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.529+0000 7fc4d0f6f700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:16.532 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.529+0000 7fc4d0f6f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:16.532 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.529+0000 7fc4d1770700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:16.532 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.529+0000 7fc4d1770700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:16.533 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.529+0000 7fc4d1770700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:16.533 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.530+0000 7fc4d1f71700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:16.533 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.530+0000 7fc4d1f71700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:16.533 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.530+0000 7fc4d1f71700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:16.533 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.530+0000 7fc4d0f6f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:16.533 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.530+0000 7fc4d0f6f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:16.533 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.530+0000 7fc4d0f6f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:16.534 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.532+0000 7fc4d1770700 10 monclient(hunting): handle_auth_done global_id 4267 payload 274 2023-09-30T02:06:16.534 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.532+0000 7fc4d1770700 1 -- 172.21.15.59:0/3102319175 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc4cc13b360 msgr2=0x7fc4cc13b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:06:16.534 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.532+0000 7fc4d1770700 1 --2- 172.21.15.59:0/3102319175 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc4cc13b360 0x7fc4cc13b7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:16.534 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.532+0000 7fc4d1770700 1 -- 172.21.15.59:0/3102319175 >> v2:172.21.15.59:3300/0 conn(0x7fc4cc1418e0 msgr2=0x7fc4cc13ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:06:16.534 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.532+0000 7fc4d1770700 1 --2- 172.21.15.59:0/3102319175 >> v2:172.21.15.59:3300/0 conn(0x7fc4cc1418e0 0x7fc4cc13ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:16.534 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.532+0000 7fc4d1770700 10 monclient: _finish_hunting 0 2023-09-30T02:06:16.534 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.532+0000 7fc4d1770700 1 monclient: found mon.noname-c 2023-09-30T02:06:16.534 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.532+0000 7fc4d1770700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:16.535 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.532+0000 7fc4d1770700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3300/0 2023-09-30T02:06:16.535 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.532+0000 7fc4d1770700 1 -- 172.21.15.59:0/3102319175 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4cc0a5b00 con 0x7fc4cc0a5730 2023-09-30T02:06:16.535 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.534+0000 7fc4d1770700 1 --2- 172.21.15.59:0/3102319175 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc4cc0a5730 0x7fc4cc0a7fa0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc4cc0b4b90 tx=0x7fc4bc009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bbc65c5cc89c947 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:16.539 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.539+0000 7fc4cbfff700 1 -- 172.21.15.59:0/3102319175 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4bc00e070 con 0x7fc4cc0a5730 2023-09-30T02:06:16.539 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.539+0000 7fc4cbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:16.539 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.539+0000 7fc4cbfff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:06:16.539 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.539+0000 7fc4cbfff700 10 monclient: dump: 2023-09-30T02:06:16.539 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:16.539 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:16.539 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:16.539 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:16.540 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:16.540 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:16.540 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:16.540 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:16.540 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:16.540 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:16.540 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.539+0000 7fc4cbfff700 10 monclient: _finish_auth 0 2023-09-30T02:06:16.540 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.539+0000 7fc4cbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:16.540 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.539+0000 7fc4cbfff700 1 -- 172.21.15.59:0/3102319175 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4bc00b970 con 0x7fc4cc0a5730 2023-09-30T02:06:16.540 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.539+0000 7fc4d39d4700 5 monclient: authenticate success, global_id 4267 2023-09-30T02:06:16.540 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.539+0000 7fc4d39d4700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:06:16.541 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.539+0000 7fc4cbfff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:16.541 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.539+0000 7fc4d39d4700 10 monclient: get_monmap_and_config success 2023-09-30T02:06:16.541 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.539+0000 7fc4cbfff700 1 -- 172.21.15.59:0/3102319175 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4bc0092e0 con 0x7fc4cc0a5730 2023-09-30T02:06:16.541 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.539+0000 7fc4d39d4700 4 set_mon_vals no callback set 2023-09-30T02:06:16.541 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.539+0000 7fc4cbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:16.541 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.539+0000 7fc4cbfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:06:16.541 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.539+0000 7fc4cbfff700 10 monclient: dump: 2023-09-30T02:06:16.541 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:16.541 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:16.541 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:16.541 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:16.541 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:16.542 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:16.542 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:16.542 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:16.542 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:16.542 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:16.542 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.539+0000 7fc4d39d4700 10 monclient: shutdown 2023-09-30T02:06:16.542 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.539+0000 7fc4d39d4700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:16.542 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.539+0000 7fc4d39d4700 1 -- 172.21.15.59:0/3102319175 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc4cc0a5730 msgr2=0x7fc4cc0a7fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:16.542 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.539+0000 7fc4d39d4700 1 --2- 172.21.15.59:0/3102319175 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc4cc0a5730 0x7fc4cc0a7fa0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc4cc0b4b90 tx=0x7fc4bc009840 comp rx=0 tx=0).stop 2023-09-30T02:06:16.542 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.539+0000 7fc4d39d4700 1 -- 172.21.15.59:0/3102319175 shutdown_connections 2023-09-30T02:06:16.542 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.539+0000 7fc4d39d4700 1 --2- 172.21.15.59:0/3102319175 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc4cc13b360 0x7fc4cc13b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:16.542 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.539+0000 7fc4d39d4700 1 --2- 172.21.15.59:0/3102319175 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc4cc0a5730 0x7fc4cc0a7fa0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:16.543 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.539+0000 7fc4d39d4700 1 --2- 172.21.15.59:0/3102319175 >> v2:172.21.15.59:3300/0 conn(0x7fc4cc1418e0 0x7fc4cc13ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:16.543 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.539+0000 7fc4d39d4700 1 -- 172.21.15.59:0/3102319175 >> 172.21.15.59:0/3102319175 conn(0x7fc4cc0631f0 msgr2=0x7fc4cc0635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:16.543 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.540+0000 7fc4d39d4700 1 -- 172.21.15.59:0/3102319175 shutdown_connections 2023-09-30T02:06:16.543 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.540+0000 7fc4d39d4700 1 -- 172.21.15.59:0/3102319175 wait complete. 2023-09-30T02:06:16.545 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.540+0000 7fc4d39d4700 10 monclient: build_initial_monmap 2023-09-30T02:06:16.545 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.540+0000 7fc4d39d4700 10 monclient: monmap: 2023-09-30T02:06:16.545 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:16.546 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:06:16.546 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:06:16.546 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:06:16.546 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:16.546 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:06:16.546 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:06:16.546 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:06:16.546 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:06:16.546 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:16.546 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.541+0000 7fc4d39d4700 1 librados: starting msgr at 2023-09-30T02:06:16.546 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.541+0000 7fc4d39d4700 1 librados: starting objecter 2023-09-30T02:06:16.546 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.541+0000 7fc4d39d4700 1 Processor -- start 2023-09-30T02:06:16.547 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.541+0000 7fc4d39d4700 1 -- start start 2023-09-30T02:06:16.547 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.541+0000 7fc4d39d4700 1 librados: setting wanted keys 2023-09-30T02:06:16.547 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.541+0000 7fc4d39d4700 1 librados: calling monclient init 2023-09-30T02:06:16.547 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.541+0000 7fc4d39d4700 10 monclient: init 2023-09-30T02:06:16.547 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.542+0000 7fc4d39d4700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:16.547 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.542+0000 7fc4d39d4700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T02:06:16.547 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.542+0000 7fc4d39d4700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4cc0a5730 0x7fc4cc14e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:16.547 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.542+0000 7fc4d39d4700 10 monclient(hunting): picked mon.noname-a con 0x7fc4cc0a5730 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:06:16.547 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.542+0000 7fc4d39d4700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc4cc13b360 0x7fc4cc14e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:16.547 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.542+0000 7fc4d39d4700 10 monclient(hunting): picked mon.noname-b con 0x7fc4cc13b360 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:06:16.547 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.542+0000 7fc4d39d4700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc4cc1418e0 0x7fc4cc13ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:16.547 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.542+0000 7fc4d39d4700 10 monclient(hunting): picked mon.noname-c con 0x7fc4cc1418e0 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:16.548 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.542+0000 7fc4d39d4700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:16.548 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.542+0000 7fc4d39d4700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4cc13c300 con 0x7fc4cc0a5730 2023-09-30T02:06:16.548 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.542+0000 7fc4d39d4700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:16.548 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.542+0000 7fc4d39d4700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4cc13c5e0 con 0x7fc4cc13b360 2023-09-30T02:06:16.548 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.542+0000 7fc4d39d4700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:16.548 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.542+0000 7fc4d39d4700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4cc13e6f0 con 0x7fc4cc1418e0 2023-09-30T02:06:16.548 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.542+0000 7fc4d39d4700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:16.548 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.542+0000 7fc4d39d4700 10 monclient(hunting): authenticate will time out at 1821.844661s 2023-09-30T02:06:16.548 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.543+0000 7fc4d0f6f700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc4cc13b360 0x7fc4cc14e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:16.548 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.543+0000 7fc4d1f71700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc4cc1418e0 0x7fc4cc13ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:16.548 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.543+0000 7fc4d1770700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4cc0a5730 0x7fc4cc14e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:16.548 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.543+0000 7fc4d1f71700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc4cc1418e0 0x7fc4cc13ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:42968/0 (socket says 172.21.15.59:42968) 2023-09-30T02:06:16.549 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.543+0000 7fc4d0f6f700 10 monclient(hunting): get_auth_request con 0x7fc4cc13b360 auth_method 0 2023-09-30T02:06:16.549 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.543+0000 7fc4d1f71700 1 -- 172.21.15.59:0/952677086 learned_addr learned my addr 172.21.15.59:0/952677086 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:06:16.549 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.543+0000 7fc4d0f6f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:16.549 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.543+0000 7fc4d0f6f700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:16.549 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.543+0000 7fc4d0f6f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:16.549 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.543+0000 7fc4d1f71700 10 monclient(hunting): get_auth_request con 0x7fc4cc1418e0 auth_method 0 2023-09-30T02:06:16.549 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.543+0000 7fc4d1f71700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:16.549 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.543+0000 7fc4d1f71700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:16.549 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.543+0000 7fc4d1f71700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:16.549 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.543+0000 7fc4d1770700 10 monclient(hunting): get_auth_request con 0x7fc4cc0a5730 auth_method 0 2023-09-30T02:06:16.549 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.543+0000 7fc4d1770700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:16.549 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.543+0000 7fc4d1770700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:16.550 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.543+0000 7fc4d1770700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:16.550 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.544+0000 7fc4d0f6f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:16.550 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.544+0000 7fc4d0f6f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:16.550 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.544+0000 7fc4d0f6f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:16.550 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.544+0000 7fc4d1770700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:16.550 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.544+0000 7fc4d1770700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:16.550 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.544+0000 7fc4d1770700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:16.550 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.544+0000 7fc4d1f71700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:16.550 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.544+0000 7fc4d1f71700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:16.550 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.544+0000 7fc4d1f71700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:16.551 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.547+0000 7fc4d0f6f700 10 monclient(hunting): handle_auth_done global_id 4270 payload 824 2023-09-30T02:06:16.551 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.547+0000 7fc4d0f6f700 1 -- 172.21.15.59:0/952677086 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc4cc1418e0 msgr2=0x7fc4cc13ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:16.551 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.547+0000 7fc4d0f6f700 1 --2- 172.21.15.59:0/952677086 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc4cc1418e0 0x7fc4cc13ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:16.551 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.547+0000 7fc4d0f6f700 1 -- 172.21.15.59:0/952677086 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4cc0a5730 msgr2=0x7fc4cc14e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:16.551 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.547+0000 7fc4d0f6f700 1 --2- 172.21.15.59:0/952677086 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4cc0a5730 0x7fc4cc14e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:16.552 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.547+0000 7fc4d0f6f700 10 monclient: _finish_hunting 0 2023-09-30T02:06:16.552 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.547+0000 7fc4d0f6f700 1 monclient: found mon.noname-b 2023-09-30T02:06:16.552 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.547+0000 7fc4d0f6f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:16.552 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.547+0000 7fc4d0f6f700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:06:16.552 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.547+0000 7fc4d0f6f700 1 -- 172.21.15.59:0/952677086 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4bc009ae0 con 0x7fc4cc13b360 2023-09-30T02:06:16.552 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.551+0000 7fc4d0f6f700 1 --2- 172.21.15.59:0/952677086 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc4cc13b360 0x7fc4cc14e860 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc4c000e930 tx=0x7fc4c00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:16.552 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.551+0000 7fc4c97fa700 10 client.?.objecter ms_handle_connect 0x7fc4cc13b360 2023-09-30T02:06:16.552 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.551+0000 7fc4c97fa700 10 client.?.objecter resend_mon_ops 2023-09-30T02:06:16.553 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.553+0000 7fc4c97fa700 1 -- 172.21.15.59:0/952677086 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4c0013680 con 0x7fc4cc13b360 2023-09-30T02:06:16.553 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.553+0000 7fc4c97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:16.553 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.553+0000 7fc4c97fa700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:06:16.553 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.553+0000 7fc4c97fa700 10 monclient: dump: 2023-09-30T02:06:16.553 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:16.553 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:16.553 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:16.554 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:16.554 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:16.554 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:16.554 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:16.554 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:16.554 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:16.554 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:16.554 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.553+0000 7fc4c97fa700 10 monclient: _finish_auth 0 2023-09-30T02:06:16.554 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.553+0000 7fc4c97fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:16.554 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.553+0000 7fc4d39d4700 5 monclient: authenticate success, global_id 4270 2023-09-30T02:06:16.554 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.553+0000 7fc4d39d4700 10 monclient: _renew_subs 2023-09-30T02:06:16.554 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.553+0000 7fc4d39d4700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:16.555 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.553+0000 7fc4d39d4700 1 -- 172.21.15.59:0/952677086 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4cc14eda0 con 0x7fc4cc13b360 2023-09-30T02:06:16.555 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.553+0000 7fc4d39d4700 10 client.4270.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:06:16.555 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.553+0000 7fc4d39d4700 10 monclient: _renew_subs 2023-09-30T02:06:16.555 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.553+0000 7fc4d39d4700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:16.555 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.553+0000 7fc4d39d4700 1 -- 172.21.15.59:0/952677086 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4cc13d240 con 0x7fc4cc13b360 2023-09-30T02:06:16.555 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.553+0000 7fc4d39d4700 1 librados: init done 2023-09-30T02:06:16.556 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.554+0000 7fc4d39d4700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:16.556 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.554+0000 7fc4d39d4700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:16.556 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.554+0000 7fc4d39d4700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:16.556 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.554+0000 7fc4d39d4700 1 -- 172.21.15.59:0/952677086 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4cc0a8000 con 0x7fc4cc13b360 2023-09-30T02:06:16.575 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.575+0000 7fc4c97fa700 1 -- 172.21.15.59:0/952677086 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4c0005a00 con 0x7fc4cc13b360 2023-09-30T02:06:16.575 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.575+0000 7fc4c97fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:16.575 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.575+0000 7fc4c97fa700 1 -- 172.21.15.59:0/952677086 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4c0005b90 con 0x7fc4cc13b360 2023-09-30T02:06:16.575 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.575+0000 7fc4c97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:16.575 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.575+0000 7fc4c97fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:06:16.575 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.575+0000 7fc4c97fa700 10 monclient: dump: 2023-09-30T02:06:16.575 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:16.576 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:16.576 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:16.576 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:16.576 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:16.576 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:16.576 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:16.576 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:16.576 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:16.576 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:16.576 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.575+0000 7fc4caffd700 4 set_mon_vals no callback set 2023-09-30T02:06:16.577 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.577+0000 7fc4c97fa700 1 -- 172.21.15.59:0/952677086 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fc4c00139f0 con 0x7fc4cc13b360 2023-09-30T02:06:16.577 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.577+0000 7fc4c97fa700 1 --2- 172.21.15.59:0/952677086 >> v2:172.21.15.110:6800/115870 conn(0x7fc4b80688c0 0x7fc4b806ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:16.577 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.577+0000 7fc4d1770700 1 --2- 172.21.15.59:0/952677086 >> v2:172.21.15.110:6800/115870 conn(0x7fc4b80688c0 0x7fc4b806ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:16.577 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.577+0000 7fc4d1770700 10 monclient: get_auth_request con 0x7fc4b80688c0 auth_method 0 2023-09-30T02:06:16.577 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.578+0000 7fc4c97fa700 10 client.4270.objecter ms_handle_connect 0x7fc4b80688c0 2023-09-30T02:06:16.577 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.578+0000 7fc4d1770700 1 --2- 172.21.15.59:0/952677086 >> v2:172.21.15.110:6800/115870 conn(0x7fc4b80688c0 0x7fc4b806ad70 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc4bc009ab0 tx=0x7fc4bc018000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:16.676 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.677+0000 7fc4c97fa700 1 -- 172.21.15.59:0/952677086 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 4075+0+0 (secure 0 0 0) 0x7fc4c00560d0 con 0x7fc4cc13b360 2023-09-30T02:06:16.677 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.677+0000 7fc4c97fa700 10 client.4270.objecter ms_dispatch 0x7fc4cc001550 osd_map(22..22 src has 1..22) v4 2023-09-30T02:06:16.677 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.677+0000 7fc4c97fa700 3 client.4270.objecter handle_osd_map got epochs [22,22] > 0 2023-09-30T02:06:16.677 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.677+0000 7fc4c97fa700 3 client.4270.objecter handle_osd_map decoding full epoch 22 2023-09-30T02:06:16.677 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.677+0000 7fc4c97fa700 20 client.4270.objecter dump_active .. 0 homeless 2023-09-30T02:06:16.838 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.838+0000 7fc4c97fa700 1 -- 172.21.15.59:0/952677086 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc4c0051b40 con 0x7fc4cc13b360 2023-09-30T02:06:16.838 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.838+0000 7fc4c97fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:16.838 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:16.838+0000 7fc4c97fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:06:17.081 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:17.082+0000 7fc4d39d4700 10 start_mon_command cmd=[{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2023-09-30T02:06:17.082 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:17.082+0000 7fc4d39d4700 10 monclient: _send_command 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2023-09-30T02:06:17.082 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:17.082+0000 7fc4d39d4700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:17.082 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:17.082+0000 7fc4d39d4700 1 -- 172.21.15.59:0/952677086 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fc4cc13c770 con 0x7fc4cc13b360 2023-09-30T02:06:18.316 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.317+0000 7fc4c97fa700 1 -- 172.21.15.59:0/952677086 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v24) v1 ==== 173+0+0 (secure 0 0 0) 0x7fc4c00210c0 con 0x7fc4cc13b360 2023-09-30T02:06:18.316 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.317+0000 7fc4c97fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2023-09-30T02:06:18.316 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.317+0000 7fc4c97fa700 10 monclient: _finish_command 2 = system:0 enabled application 'rbd' on pool 'rbd' 2023-09-30T02:06:18.319 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.319+0000 7fc4d39d4700 10 librados: watch_flush enter 2023-09-30T02:06:18.319 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.319+0000 7fc4d39d4700 10 librados: watch_flush exit 2023-09-30T02:06:18.319 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.319+0000 7fc4d39d4700 20 client.4270.objecter shutdown clearing up homeless session... 2023-09-30T02:06:18.319 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.319+0000 7fc4d39d4700 10 client.4270.objecter successfully canceled tick 2023-09-30T02:06:18.319 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.319+0000 7fc4d39d4700 1 -- 172.21.15.59:0/952677086 >> v2:172.21.15.110:6800/115870 conn(0x7fc4b80688c0 msgr2=0x7fc4b806ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:18.319 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.319+0000 7fc4d39d4700 1 --2- 172.21.15.59:0/952677086 >> v2:172.21.15.110:6800/115870 conn(0x7fc4b80688c0 0x7fc4b806ad70 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc4bc009ab0 tx=0x7fc4bc018000 comp rx=0 tx=0).stop 2023-09-30T02:06:18.319 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.319+0000 7fc4d39d4700 10 monclient: shutdown 2023-09-30T02:06:18.320 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.319+0000 7fc4d39d4700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:18.320 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.319+0000 7fc4d39d4700 1 -- 172.21.15.59:0/952677086 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc4cc13b360 msgr2=0x7fc4cc14e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:18.320 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.319+0000 7fc4d39d4700 1 --2- 172.21.15.59:0/952677086 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc4cc13b360 0x7fc4cc14e860 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc4c000e930 tx=0x7fc4c00091a0 comp rx=0 tx=0).stop 2023-09-30T02:06:18.320 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.319+0000 7fc4d39d4700 1 -- 172.21.15.59:0/952677086 shutdown_connections 2023-09-30T02:06:18.320 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.319+0000 7fc4d39d4700 1 --2- 172.21.15.59:0/952677086 >> v2:172.21.15.110:6800/115870 conn(0x7fc4b80688c0 0x7fc4b806ad70 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:18.320 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.319+0000 7fc4d39d4700 1 --2- 172.21.15.59:0/952677086 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc4cc1418e0 0x7fc4cc13ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:18.320 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.319+0000 7fc4d39d4700 1 --2- 172.21.15.59:0/952677086 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4cc0a5730 0x7fc4cc14e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:18.320 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.319+0000 7fc4d39d4700 1 --2- 172.21.15.59:0/952677086 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc4cc13b360 0x7fc4cc14e860 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:18.320 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.319+0000 7fc4d39d4700 1 -- 172.21.15.59:0/952677086 >> 172.21.15.59:0/952677086 conn(0x7fc4cc0631f0 msgr2=0x7fc4cc0a5cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:18.320 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.319+0000 7fc4d39d4700 1 -- 172.21.15.59:0/952677086 shutdown_connections 2023-09-30T02:06:18.320 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.319+0000 7fc4d39d4700 1 -- 172.21.15.59:0/952677086 wait complete. 2023-09-30T02:06:18.321 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.319+0000 7fc4d39d4700 1 librados: shutdown 2023-09-30T02:06:18.321 INFO:teuthology.orchestra.run.smithi059.stderr:enabled application 'rbd' on pool 'rbd' 2023-09-30T02:06:18.333 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-09-30T02:06:18.334 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-09-30T02:06:18.334 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-09-30T02:06:18.334 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-09-30T02:06:18.334 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-09-30T02:06:18.467 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.468+0000 7f0fe76ca700 10 monclient: get_monmap_and_config 2023-09-30T02:06:18.467 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.468+0000 7f0fe76ca700 10 monclient: build_initial_monmap 2023-09-30T02:06:18.468 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.468+0000 7f0fe76ca700 10 monclient: monmap: 2023-09-30T02:06:18.468 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:18.468 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:18.468 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:06:18.468704+0000 2023-09-30T02:06:18.468 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:06:18.468704+0000 2023-09-30T02:06:18.468 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:18.468 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:18.468 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:06:18.468 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:06:18.468 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:06:18.468 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:06:18.468 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:06:18.469 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:18.470 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.468+0000 7f0fe76ca700 1 Processor -- start 2023-09-30T02:06:18.470 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.469+0000 7f0fe76ca700 1 -- start start 2023-09-30T02:06:18.470 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.469+0000 7f0fe76ca700 10 monclient: init 2023-09-30T02:06:18.470 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.469+0000 7f0fe76ca700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:18.470 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.469+0000 7f0fe76ca700 10 monclient: _add_conns ranks=[0,2,1,3,4] 2023-09-30T02:06:18.470 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.469+0000 7f0fe76ca700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0fe00a5b70 addr v1:172.21.15.59:6789/0 2023-09-30T02:06:18.471 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.469+0000 7f0fe76ca700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0fe0142290 0x7f0fe00b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:18.471 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.469+0000 7f0fe76ca700 10 monclient(hunting): picked mon.noname-b con 0x7f0fe0142290 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:18.471 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.469+0000 7f0fe76ca700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0fe00a5f40 addr v1:172.21.15.110:6789/0 2023-09-30T02:06:18.471 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.469+0000 7f0fe76ca700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fe00a6380 con 0x7f0fe00a5b70 2023-09-30T02:06:18.471 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.469+0000 7f0fe76ca700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fe00a75a0 con 0x7f0fe00a5f40 2023-09-30T02:06:18.471 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.469+0000 7f0fe76ca700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:18.471 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.469+0000 7f0fe76ca700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe00af3f0 con 0x7f0fe0142290 2023-09-30T02:06:18.471 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.469+0000 7f0fe76ca700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:18.471 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.469+0000 7f0fe76ca700 10 monclient(hunting): authenticate will time out at 1823.771916s 2023-09-30T02:06:18.471 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.470+0000 7f0fe4c65700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0fe0142290 0x7f0fe00b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:18.471 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.470+0000 7f0fe5466700 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f0fe00a5b70 0x7f0fe0141b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:34950/0 (socket says 172.21.15.59:34950) 2023-09-30T02:06:18.471 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.470+0000 7f0fe5466700 1 -- 172.21.15.59:0/1523728206 learned_addr learned my addr 172.21.15.59:0/1523728206 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:06:18.472 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.471+0000 7f0fe4c65700 10 monclient(hunting): get_auth_request con 0x7f0fe0142290 auth_method 0 2023-09-30T02:06:18.472 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.471+0000 7f0fe4c65700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:18.472 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.471+0000 7f0fe4c65700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:18.472 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.471+0000 7f0fe4c65700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:18.472 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.471+0000 7f0fe4c65700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:18.472 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.471+0000 7f0fe4c65700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:18.472 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.471+0000 7f0fe4c65700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:18.473 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.474+0000 7f0fd7fff700 1 -- 172.21.15.59:0/1523728206 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3500889816 0 0) 0x7f0fe00a75a0 con 0x7f0fe00a5f40 2023-09-30T02:06:18.474 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.474+0000 7f0fd7fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:18.474 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.474+0000 7f0fd7fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:18.474 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.474+0000 7f0fd7fff700 10 monclient(hunting): my global_id is 4273 2023-09-30T02:06:18.474 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.474+0000 7f0fd7fff700 1 -- 172.21.15.59:0/1523728206 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fcc003570 con 0x7f0fe00a5f40 2023-09-30T02:06:18.474 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.474+0000 7f0fd7fff700 1 -- 172.21.15.59:0/1523728206 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 693213760 0 0) 0x7f0fe00a6380 con 0x7f0fe00a5b70 2023-09-30T02:06:18.474 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.474+0000 7f0fd7fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:18.474 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.474+0000 7f0fd7fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:18.474 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.474+0000 7f0fd7fff700 10 monclient(hunting): my global_id is 4263 2023-09-30T02:06:18.474 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.474+0000 7f0fd7fff700 1 -- 172.21.15.59:0/1523728206 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fe00a75a0 con 0x7f0fe00a5b70 2023-09-30T02:06:18.474 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.474+0000 7f0fe4c65700 10 monclient(hunting): handle_auth_done global_id 4244 payload 274 2023-09-30T02:06:18.474 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.474+0000 7f0fe4c65700 1 -- 172.21.15.59:0/1523728206 >> v1:172.21.15.110:6789/0 conn(0x7f0fe00a5f40 legacy=0x7f0fe013af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:18.475 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.474+0000 7f0fe4c65700 1 -- 172.21.15.59:0/1523728206 >> v1:172.21.15.59:6789/0 conn(0x7f0fe00a5b70 legacy=0x7f0fe0141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:18.475 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.474+0000 7f0fe4c65700 10 monclient: _finish_hunting 0 2023-09-30T02:06:18.475 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.474+0000 7f0fe4c65700 1 monclient: found mon.noname-b 2023-09-30T02:06:18.475 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.474+0000 7f0fe4c65700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:18.475 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.474+0000 7f0fe4c65700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.59:3301/0 2023-09-30T02:06:18.475 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.474+0000 7f0fe4c65700 1 -- 172.21.15.59:0/1523728206 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fe013b640 con 0x7f0fe0142290 2023-09-30T02:06:18.475 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.476+0000 7f0fe4c65700 1 --2- 172.21.15.59:0/1523728206 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0fe0142290 0x7f0fe00b1500 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f0fd0009910 tx=0x7f0fd00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d0bc77edc7dcdb66 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:18.480 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.480+0000 7f0fd7fff700 1 -- 172.21.15.59:0/1523728206 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fd0011630 con 0x7f0fe0142290 2023-09-30T02:06:18.480 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.480+0000 7f0fd7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:18.480 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.480+0000 7f0fd7fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:06:18.480 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.480+0000 7f0fd7fff700 10 monclient: dump: 2023-09-30T02:06:18.480 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:18.480 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:18.480 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:18.480 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:18.480 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:18.480 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:18.481 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:18.481 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:18.481 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:18.481 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:18.481 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.480+0000 7f0fd7fff700 10 monclient: _finish_auth 0 2023-09-30T02:06:18.481 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.480+0000 7f0fd7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:18.481 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.480+0000 7f0fd7fff700 1 -- 172.21.15.59:0/1523728206 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fd0011790 con 0x7f0fe0142290 2023-09-30T02:06:18.481 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.480+0000 7f0fe76ca700 5 monclient: authenticate success, global_id 4244 2023-09-30T02:06:18.481 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.480+0000 7f0fe76ca700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:06:18.481 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.480+0000 7f0fd7fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:18.481 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.480+0000 7f0fe76ca700 10 monclient: get_monmap_and_config success 2023-09-30T02:06:18.481 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.480+0000 7f0fe76ca700 4 set_mon_vals no callback set 2023-09-30T02:06:18.482 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.480+0000 7f0fd7fff700 1 -- 172.21.15.59:0/1523728206 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fd0011b30 con 0x7f0fe0142290 2023-09-30T02:06:18.482 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.480+0000 7f0fd7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:18.482 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.480+0000 7f0fd7fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:06:18.482 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.480+0000 7f0fd7fff700 10 monclient: dump: 2023-09-30T02:06:18.482 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:18.482 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:18.482 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:18.482 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:18.482 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:18.482 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:18.482 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:18.482 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:18.483 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:18.483 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:18.485 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.481+0000 7f0fe76ca700 10 monclient: shutdown 2023-09-30T02:06:18.486 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.481+0000 7f0fe76ca700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:18.486 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.481+0000 7f0fe76ca700 1 -- 172.21.15.59:0/1523728206 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0fe0142290 msgr2=0x7f0fe00b1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:18.486 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.481+0000 7f0fe76ca700 1 --2- 172.21.15.59:0/1523728206 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0fe0142290 0x7f0fe00b1500 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f0fd0009910 tx=0x7f0fd00091a0 comp rx=0 tx=0).stop 2023-09-30T02:06:18.486 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.481+0000 7f0fe76ca700 1 -- 172.21.15.59:0/1523728206 shutdown_connections 2023-09-30T02:06:18.486 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.481+0000 7f0fe76ca700 1 --2- 172.21.15.59:0/1523728206 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0fe0142290 0x7f0fe00b1500 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:18.486 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.481+0000 7f0fe76ca700 1 -- 172.21.15.59:0/1523728206 >> 172.21.15.59:0/1523728206 conn(0x7f0fe00631f0 msgr2=0x7f0fe00635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:18.486 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.481+0000 7f0fe76ca700 1 -- 172.21.15.59:0/1523728206 shutdown_connections 2023-09-30T02:06:18.486 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.481+0000 7f0fe76ca700 1 -- 172.21.15.59:0/1523728206 wait complete. 2023-09-30T02:06:18.486 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.482+0000 7f0fe76ca700 10 monclient: build_initial_monmap 2023-09-30T02:06:18.486 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.482+0000 7f0fe76ca700 10 monclient: monmap: 2023-09-30T02:06:18.486 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:18.487 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:06:18.487 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:06:18.487 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:06:18.487 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:18.487 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:06:18.487 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:06:18.487 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:06:18.487 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:06:18.487 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:18.487 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.482+0000 7f0fe76ca700 1 librados: starting msgr at 2023-09-30T02:06:18.487 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.482+0000 7f0fe76ca700 1 librados: starting objecter 2023-09-30T02:06:18.487 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.482+0000 7f0fe76ca700 1 Processor -- start 2023-09-30T02:06:18.487 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.482+0000 7f0fe76ca700 1 -- start start 2023-09-30T02:06:18.488 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.482+0000 7f0fe76ca700 1 librados: setting wanted keys 2023-09-30T02:06:18.488 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.482+0000 7f0fe76ca700 1 librados: calling monclient init 2023-09-30T02:06:18.488 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.482+0000 7f0fe76ca700 10 monclient: init 2023-09-30T02:06:18.488 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.483+0000 7f0fe76ca700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:18.488 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.483+0000 7f0fe76ca700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T02:06:18.488 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.483+0000 7f0fe76ca700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0fe00a5b70 0x7f0fe01516a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:18.488 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.483+0000 7f0fe76ca700 10 monclient(hunting): picked mon.noname-a con 0x7f0fe00a5b70 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:06:18.488 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.483+0000 7f0fe76ca700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0fe00a5f40 0x7f0fe013ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:18.488 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.483+0000 7f0fe76ca700 10 monclient(hunting): picked mon.noname-c con 0x7f0fe00a5f40 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:18.488 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.483+0000 7f0fe76ca700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0fe0142290 0x7f0fe00a7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:18.488 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.483+0000 7f0fe76ca700 10 monclient(hunting): picked mon.noname-b con 0x7f0fe0142290 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:06:18.488 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.483+0000 7f0fe76ca700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:18.489 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.483+0000 7f0fe76ca700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe0152170 con 0x7f0fe00a5b70 2023-09-30T02:06:18.489 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.483+0000 7f0fe76ca700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:18.489 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.483+0000 7f0fe76ca700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe01522e0 con 0x7f0fe0142290 2023-09-30T02:06:18.489 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.483+0000 7f0fe76ca700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:18.489 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.483+0000 7f0fe76ca700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe00a7fe0 con 0x7f0fe00a5f40 2023-09-30T02:06:18.489 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.483+0000 7f0fe76ca700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:18.489 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.483+0000 7f0fe76ca700 10 monclient(hunting): authenticate will time out at 1823.785816s 2023-09-30T02:06:18.489 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.484+0000 7f0fe5c67700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0fe0142290 0x7f0fe00a7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:18.489 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.484+0000 7f0fe4c65700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0fe00a5f40 0x7f0fe013ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:18.489 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.484+0000 7f0fe5466700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0fe00a5b70 0x7f0fe01516a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:18.489 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.484+0000 7f0fe5c67700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0fe0142290 0x7f0fe00a7aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:38830/0 (socket says 172.21.15.59:38830) 2023-09-30T02:06:18.489 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.484+0000 7f0fe5c67700 1 -- 172.21.15.59:0/1070859968 learned_addr learned my addr 172.21.15.59:0/1070859968 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:06:18.489 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.484+0000 7f0fe5c67700 10 monclient(hunting): get_auth_request con 0x7f0fe0142290 auth_method 0 2023-09-30T02:06:18.490 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.484+0000 7f0fe5c67700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:18.490 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.484+0000 7f0fe5c67700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:18.490 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.484+0000 7f0fe5c67700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:18.490 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.484+0000 7f0fe4c65700 10 monclient(hunting): get_auth_request con 0x7f0fe00a5f40 auth_method 0 2023-09-30T02:06:18.490 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.484+0000 7f0fe4c65700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:18.490 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.484+0000 7f0fe4c65700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:18.490 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.484+0000 7f0fe4c65700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:18.490 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.485+0000 7f0fe5466700 10 monclient(hunting): get_auth_request con 0x7f0fe00a5b70 auth_method 0 2023-09-30T02:06:18.490 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.485+0000 7f0fe5466700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:18.490 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.485+0000 7f0fe5466700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:18.490 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.485+0000 7f0fe5466700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:18.490 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.485+0000 7f0fe5c67700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:18.490 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.485+0000 7f0fe5c67700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:18.491 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.485+0000 7f0fe5c67700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:18.491 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.485+0000 7f0fe4c65700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:18.491 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.485+0000 7f0fe4c65700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:18.491 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.485+0000 7f0fe4c65700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:18.491 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.485+0000 7f0fe5466700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:18.491 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.485+0000 7f0fe5466700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:18.491 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.485+0000 7f0fe5466700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:18.492 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.488+0000 7f0fe5c67700 10 monclient(hunting): handle_auth_done global_id 4276 payload 824 2023-09-30T02:06:18.492 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.488+0000 7f0fe5c67700 1 -- 172.21.15.59:0/1070859968 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0fe00a5f40 msgr2=0x7f0fe013ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:18.492 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.488+0000 7f0fe5c67700 1 --2- 172.21.15.59:0/1070859968 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0fe00a5f40 0x7f0fe013ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:18.492 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.488+0000 7f0fe5c67700 1 -- 172.21.15.59:0/1070859968 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0fe00a5b70 msgr2=0x7f0fe01516a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:18.492 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.488+0000 7f0fe5c67700 1 --2- 172.21.15.59:0/1070859968 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0fe00a5b70 0x7f0fe01516a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:18.492 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.488+0000 7f0fe5c67700 10 monclient: _finish_hunting 0 2023-09-30T02:06:18.492 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.488+0000 7f0fe5c67700 1 monclient: found mon.noname-b 2023-09-30T02:06:18.492 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.488+0000 7f0fe5c67700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:18.492 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.488+0000 7f0fe5c67700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:06:18.493 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.488+0000 7f0fe5c67700 1 -- 172.21.15.59:0/1070859968 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fd0009420 con 0x7f0fe0142290 2023-09-30T02:06:18.493 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.491+0000 7f0fe5c67700 1 --2- 172.21.15.59:0/1070859968 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0fe0142290 0x7f0fe00a7aa0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0fd800c910 tx=0x7f0fd80027a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:18.493 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.491+0000 7f0fd57fa700 10 client.?.objecter ms_handle_connect 0x7f0fe0142290 2023-09-30T02:06:18.493 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.491+0000 7f0fd57fa700 10 client.?.objecter resend_mon_ops 2023-09-30T02:06:18.495 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.495+0000 7f0fd57fa700 1 -- 172.21.15.59:0/1070859968 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fd8012630 con 0x7f0fe0142290 2023-09-30T02:06:18.495 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.495+0000 7f0fd57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:18.495 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.495+0000 7f0fd57fa700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:06:18.495 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.495+0000 7f0fd57fa700 10 monclient: dump: 2023-09-30T02:06:18.495 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:18.496 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:18.496 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:18.496 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:18.496 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:18.496 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:18.496 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:18.496 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:18.496 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:18.496 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:18.496 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.495+0000 7f0fd57fa700 10 monclient: _finish_auth 0 2023-09-30T02:06:18.496 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.495+0000 7f0fd57fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:18.496 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.495+0000 7f0fd57fa700 1 -- 172.21.15.59:0/1070859968 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fd80127b0 con 0x7f0fe0142290 2023-09-30T02:06:18.496 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.495+0000 7f0fd57fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:18.497 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.495+0000 7f0fd57fa700 1 -- 172.21.15.59:0/1070859968 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fd8012b70 con 0x7f0fe0142290 2023-09-30T02:06:18.497 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.495+0000 7f0fd57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:18.497 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.495+0000 7f0fd57fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:06:18.497 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.495+0000 7f0fd57fa700 10 monclient: dump: 2023-09-30T02:06:18.497 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:18.497 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:18.497 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:18.497 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:18.497 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:18.497 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:18.497 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:18.497 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:18.498 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:18.498 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:18.498 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.496+0000 7f0fe76ca700 5 monclient: authenticate success, global_id 4276 2023-09-30T02:06:18.498 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.496+0000 7f0fe76ca700 10 monclient: _renew_subs 2023-09-30T02:06:18.498 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.496+0000 7f0fe76ca700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:18.498 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.496+0000 7f0fe76ca700 1 -- 172.21.15.59:0/1070859968 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fe00a81a0 con 0x7f0fe0142290 2023-09-30T02:06:18.498 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.495+0000 7f0fd77fe700 4 set_mon_vals no callback set 2023-09-30T02:06:18.498 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.496+0000 7f0fe76ca700 10 client.4276.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:06:18.498 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.496+0000 7f0fe76ca700 10 monclient: _renew_subs 2023-09-30T02:06:18.498 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.496+0000 7f0fe76ca700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:18.498 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.496+0000 7f0fe76ca700 1 -- 172.21.15.59:0/1070859968 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fe00a8ba0 con 0x7f0fe0142290 2023-09-30T02:06:18.498 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.496+0000 7f0fe76ca700 1 librados: init done 2023-09-30T02:06:18.499 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.496+0000 7f0fe76ca700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:18.499 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.496+0000 7f0fe76ca700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:18.499 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.496+0000 7f0fe76ca700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:18.499 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.496+0000 7f0fe76ca700 1 -- 172.21.15.59:0/1070859968 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fe0141d10 con 0x7f0fe0142290 2023-09-30T02:06:18.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.524+0000 7f0fd57fa700 1 -- 172.21.15.59:0/1070859968 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f0fd801b400 con 0x7f0fe0142290 2023-09-30T02:06:18.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.525+0000 7f0fd57fa700 1 --2- 172.21.15.59:0/1070859968 >> v2:172.21.15.110:6800/115870 conn(0x7f0fc40689e0 0x7f0fc406ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:18.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.525+0000 7f0fd57fa700 1 -- 172.21.15.59:0/1070859968 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f0fd8084780 con 0x7f0fe0142290 2023-09-30T02:06:18.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.525+0000 7f0fd57fa700 10 client.4276.objecter ms_dispatch 0x7f0fe0001550 osd_map(24..24 src has 1..24) v4 2023-09-30T02:06:18.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.525+0000 7f0fd57fa700 3 client.4276.objecter handle_osd_map got epochs [24,24] > 0 2023-09-30T02:06:18.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.525+0000 7f0fd57fa700 3 client.4276.objecter handle_osd_map decoding full epoch 24 2023-09-30T02:06:18.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.525+0000 7f0fd57fa700 20 client.4276.objecter dump_active .. 0 homeless 2023-09-30T02:06:18.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.525+0000 7f0fe5466700 1 --2- 172.21.15.59:0/1070859968 >> v2:172.21.15.110:6800/115870 conn(0x7f0fc40689e0 0x7f0fc406ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:18.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.525+0000 7f0fe5466700 10 monclient: get_auth_request con 0x7f0fc40689e0 auth_method 0 2023-09-30T02:06:18.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.525+0000 7f0fe5466700 1 --2- 172.21.15.59:0/1070859968 >> v2:172.21.15.110:6800/115870 conn(0x7f0fc40689e0 0x7f0fc406ae90 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0fdc007fd0 tx=0x7f0fdc009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:18.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.525+0000 7f0fd57fa700 10 client.4276.objecter ms_handle_connect 0x7f0fc40689e0 2023-09-30T02:06:18.782 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.782+0000 7f0fd57fa700 1 -- 172.21.15.59:0/1070859968 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0fd80520a0 con 0x7f0fe0142290 2023-09-30T02:06:18.782 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.782+0000 7f0fd57fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:18.782 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:18.782+0000 7f0fd57fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:06:19.042 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.043+0000 7f0fe76ca700 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2023-09-30T02:06:19.042 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.043+0000 7f0fe76ca700 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2023-09-30T02:06:19.043 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.043+0000 7f0fe76ca700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:19.043 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.043+0000 7f0fe76ca700 1 -- 172.21.15.59:0/1070859968 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f0fe0064a00 con 0x7f0fe0142290 2023-09-30T02:06:19.199 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.200+0000 7f0fd57fa700 1 -- 172.21.15.59:0/1070859968 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+163649 (secure 0 0 0) 0x7f0fd8050f60 con 0x7f0fe0142290 2023-09-30T02:06:19.200 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.200+0000 7f0fd57fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2023-09-30T02:06:19.200 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.200+0000 7f0fd57fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:06:19.202 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:06:19.204 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.203+0000 7f0fe76ca700 10 librados: watch_flush enter 2023-09-30T02:06:19.204 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.203+0000 7f0fe76ca700 10 librados: watch_flush exit 2023-09-30T02:06:19.204 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.203+0000 7f0fe76ca700 20 client.4276.objecter shutdown clearing up homeless session... 2023-09-30T02:06:19.204 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.203+0000 7f0fe76ca700 10 client.4276.objecter successfully canceled tick 2023-09-30T02:06:19.204 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.203+0000 7f0fe76ca700 1 -- 172.21.15.59:0/1070859968 >> v2:172.21.15.110:6800/115870 conn(0x7f0fc40689e0 msgr2=0x7f0fc406ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:19.204 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.203+0000 7f0fe76ca700 1 --2- 172.21.15.59:0/1070859968 >> v2:172.21.15.110:6800/115870 conn(0x7f0fc40689e0 0x7f0fc406ae90 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0fdc007fd0 tx=0x7f0fdc009040 comp rx=0 tx=0).stop 2023-09-30T02:06:19.204 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.203+0000 7f0fe76ca700 10 monclient: shutdown 2023-09-30T02:06:19.204 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.203+0000 7f0fe76ca700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:19.204 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.203+0000 7f0fe76ca700 1 -- 172.21.15.59:0/1070859968 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0fe0142290 msgr2=0x7f0fe00a7aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:19.204 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.203+0000 7f0fe76ca700 1 --2- 172.21.15.59:0/1070859968 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0fe0142290 0x7f0fe00a7aa0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0fd800c910 tx=0x7f0fd80027a0 comp rx=0 tx=0).stop 2023-09-30T02:06:19.205 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.203+0000 7f0fe76ca700 1 -- 172.21.15.59:0/1070859968 shutdown_connections 2023-09-30T02:06:19.205 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.203+0000 7f0fe76ca700 1 --2- 172.21.15.59:0/1070859968 >> v2:172.21.15.110:6800/115870 conn(0x7f0fc40689e0 0x7f0fc406ae90 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:19.205 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.203+0000 7f0fe76ca700 1 --2- 172.21.15.59:0/1070859968 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0fe0142290 0x7f0fe00a7aa0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:19.205 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.203+0000 7f0fe76ca700 1 --2- 172.21.15.59:0/1070859968 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0fe00a5b70 0x7f0fe01516a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:19.205 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.203+0000 7f0fe76ca700 1 --2- 172.21.15.59:0/1070859968 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0fe00a5f40 0x7f0fe013ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:19.205 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.203+0000 7f0fe76ca700 1 -- 172.21.15.59:0/1070859968 >> 172.21.15.59:0/1070859968 conn(0x7f0fe00631f0 msgr2=0x7f0fe00635c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:19.205 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.203+0000 7f0fe76ca700 1 -- 172.21.15.59:0/1070859968 shutdown_connections 2023-09-30T02:06:19.205 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.203+0000 7f0fe76ca700 1 -- 172.21.15.59:0/1070859968 wait complete. 2023-09-30T02:06:19.205 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.203+0000 7f0fe76ca700 1 librados: shutdown 2023-09-30T02:06:19.217 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":5,"active_gid":4108,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6800","nonce":115870}]},"active_addr":"172.21.15.110:6800/115870","active_change":"2023-09-30T02:05:19.246284+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":4104,"name":"y","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.110:0","nonce":1057117874}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.110:0","nonce":1256202835}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.110:0","nonce":342312274}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.110:0","nonce":31193585}]}]} 2023-09-30T02:06:19.223 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-09-30T02:06:19.223 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-09-30T02:06:19.223 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-30T02:06:19.372 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.372+0000 7fd2f3e7a700 10 monclient: get_monmap_and_config 2023-09-30T02:06:19.372 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.372+0000 7fd2f3e7a700 10 monclient: build_initial_monmap 2023-09-30T02:06:19.372 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.372+0000 7fd2f3e7a700 10 monclient: monmap: 2023-09-30T02:06:19.372 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:19.372 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:19.372 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:06:19.373286+0000 2023-09-30T02:06:19.373 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:06:19.373286+0000 2023-09-30T02:06:19.373 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:19.373 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:19.373 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:06:19.373 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:06:19.373 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:06:19.373 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:06:19.373 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:06:19.373 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:19.374 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.373+0000 7fd2f3e7a700 1 Processor -- start 2023-09-30T02:06:19.375 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.373+0000 7fd2f3e7a700 1 -- start start 2023-09-30T02:06:19.375 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.373+0000 7fd2f3e7a700 10 monclient: init 2023-09-30T02:06:19.375 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.374+0000 7fd2f3e7a700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:19.375 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.374+0000 7fd2f3e7a700 10 monclient: _add_conns ranks=[1,3,4,0,2] 2023-09-30T02:06:19.375 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.374+0000 7fd2f3e7a700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd2ec0a5b70 addr v1:172.21.15.110:6789/0 2023-09-30T02:06:19.375 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.374+0000 7fd2f3e7a700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fd2ec142290 0x7fd2ec0b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:19.375 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.374+0000 7fd2f3e7a700 10 monclient(hunting): picked mon.noname-a con 0x7fd2ec142290 addr v2:172.21.15.59:3300/0 2023-09-30T02:06:19.375 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.374+0000 7fd2f3e7a700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd2ec0a5f40 0x7fd2ec13af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:19.375 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.374+0000 7fd2f3e7a700 10 monclient(hunting): picked mon.noname-c con 0x7fd2ec0a5f40 addr v2:172.21.15.110:3300/0 2023-09-30T02:06:19.375 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.374+0000 7fd2f3e7a700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2ec0a75a0 con 0x7fd2ec0a5b70 2023-09-30T02:06:19.375 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.374+0000 7fd2f3e7a700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:19.375 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.374+0000 7fd2f3e7a700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2ec0a6360 con 0x7fd2ec142290 2023-09-30T02:06:19.376 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.374+0000 7fd2f3e7a700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:19.376 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.374+0000 7fd2f3e7a700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2ec0af3f0 con 0x7fd2ec0a5f40 2023-09-30T02:06:19.376 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.374+0000 7fd2f3e7a700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:19.376 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.374+0000 7fd2f3e7a700 10 monclient(hunting): authenticate will time out at 1824.676644s 2023-09-30T02:06:19.408 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.408+0000 7fd2f2417700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd2ec0a5f40 0x7fd2ec13af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:19.408 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.408+0000 7fd2f2417700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd2ec0a5f40 0x7fd2ec13af30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:38836/0 (socket says 172.21.15.59:38836) 2023-09-30T02:06:19.408 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.408+0000 7fd2f2417700 1 -- 172.21.15.59:0/3242943927 learned_addr learned my addr 172.21.15.59:0/3242943927 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:06:19.408 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.409+0000 7fd2f2417700 10 monclient(hunting): get_auth_request con 0x7fd2ec0a5f40 auth_method 0 2023-09-30T02:06:19.408 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.409+0000 7fd2f2417700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:19.408 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.409+0000 7fd2f2417700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:19.408 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.409+0000 7fd2f2417700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:19.413 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.413+0000 7fd2f1415700 1 --2- 172.21.15.59:0/3242943927 >> v2:172.21.15.59:3300/0 conn(0x7fd2ec142290 0x7fd2ec0b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:19.414 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.415+0000 7fd2f1415700 10 monclient(hunting): get_auth_request con 0x7fd2ec142290 auth_method 0 2023-09-30T02:06:19.414 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.415+0000 7fd2f1415700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:19.414 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.415+0000 7fd2f1415700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:19.414 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.415+0000 7fd2f1415700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:19.414 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.415+0000 7fd2f1415700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:19.415 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.415+0000 7fd2f1415700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:19.415 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.415+0000 7fd2f1415700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:19.417 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.418+0000 7fd2f1415700 10 monclient(hunting): handle_auth_done global_id 4269 payload 274 2023-09-30T02:06:19.418 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.418+0000 7fd2f1415700 1 -- 172.21.15.59:0/3242943927 >> v2:172.21.15.110:3300/0 conn(0x7fd2ec0a5f40 msgr2=0x7fd2ec13af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:06:19.418 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.418+0000 7fd2f1415700 1 --2- 172.21.15.59:0/3242943927 >> v2:172.21.15.110:3300/0 conn(0x7fd2ec0a5f40 0x7fd2ec13af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:19.418 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.418+0000 7fd2f1415700 1 -- 172.21.15.59:0/3242943927 >> v1:172.21.15.110:6789/0 conn(0x7fd2ec0a5b70 legacy=0x7fd2ec141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:19.418 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.418+0000 7fd2f1415700 10 monclient: _finish_hunting 0 2023-09-30T02:06:19.418 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.418+0000 7fd2f1415700 1 monclient: found mon.noname-a 2023-09-30T02:06:19.418 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.418+0000 7fd2f1415700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:19.418 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.418+0000 7fd2f1415700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.59:3300/0 2023-09-30T02:06:19.418 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.418+0000 7fd2f1415700 1 -- 172.21.15.59:0/3242943927 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2ec13b470 con 0x7fd2ec142290 2023-09-30T02:06:19.425 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.425+0000 7fd2f1415700 1 --2- 172.21.15.59:0/3242943927 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd2ec142290 0x7fd2ec0b1500 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec0b0640 tx=0x7fd2dc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8ee8c7c66db0bf6 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:19.431 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.430+0000 7fd2f0c14700 1 -- 172.21.15.59:0/3242943927 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2dc00b6b0 con 0x7fd2ec142290 2023-09-30T02:06:19.431 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.430+0000 7fd2f0c14700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:19.431 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.430+0000 7fd2f0c14700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T02:06:19.431 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.430+0000 7fd2f0c14700 10 monclient: dump: 2023-09-30T02:06:19.431 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:19.431 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:19.431 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:19.432 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:19.432 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:19.432 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:19.432 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:19.432 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:19.432 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:19.432 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:19.432 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.431+0000 7fd2f0c14700 10 monclient: _finish_auth 0 2023-09-30T02:06:19.432 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.431+0000 7fd2f0c14700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:19.432 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.431+0000 7fd2f0c14700 1 -- 172.21.15.59:0/3242943927 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2dc00b810 con 0x7fd2ec142290 2023-09-30T02:06:19.432 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.431+0000 7fd2f3e7a700 5 monclient: authenticate success, global_id 4269 2023-09-30T02:06:19.432 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.431+0000 7fd2f3e7a700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:06:19.433 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.431+0000 7fd2f0c14700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:19.433 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.431+0000 7fd2f3e7a700 10 monclient: get_monmap_and_config success 2023-09-30T02:06:19.433 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.431+0000 7fd2f0c14700 1 -- 172.21.15.59:0/3242943927 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2dc00bbb0 con 0x7fd2ec142290 2023-09-30T02:06:19.433 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.431+0000 7fd2f3e7a700 4 set_mon_vals no callback set 2023-09-30T02:06:19.433 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.431+0000 7fd2f0c14700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:19.433 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.431+0000 7fd2f0c14700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T02:06:19.433 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.431+0000 7fd2f0c14700 10 monclient: dump: 2023-09-30T02:06:19.433 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:19.433 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:19.433 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:19.433 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:19.433 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:19.434 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:19.434 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:19.434 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:19.434 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:19.434 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:19.434 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.431+0000 7fd2f3e7a700 10 monclient: shutdown 2023-09-30T02:06:19.434 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.431+0000 7fd2f3e7a700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:19.434 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.431+0000 7fd2f3e7a700 1 -- 172.21.15.59:0/3242943927 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd2ec142290 msgr2=0x7fd2ec0b1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:19.434 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.431+0000 7fd2f3e7a700 1 --2- 172.21.15.59:0/3242943927 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd2ec142290 0x7fd2ec0b1500 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec0b0640 tx=0x7fd2dc0091a0 comp rx=0 tx=0).stop 2023-09-30T02:06:19.434 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.431+0000 7fd2f3e7a700 1 -- 172.21.15.59:0/3242943927 shutdown_connections 2023-09-30T02:06:19.434 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.431+0000 7fd2f3e7a700 1 --2- 172.21.15.59:0/3242943927 >> v2:172.21.15.110:3300/0 conn(0x7fd2ec0a5f40 0x7fd2ec13af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:19.434 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.431+0000 7fd2f3e7a700 1 --2- 172.21.15.59:0/3242943927 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd2ec142290 0x7fd2ec0b1500 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:19.435 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.431+0000 7fd2f3e7a700 1 -- 172.21.15.59:0/3242943927 >> 172.21.15.59:0/3242943927 conn(0x7fd2ec0631f0 msgr2=0x7fd2ec0635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:19.435 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.432+0000 7fd2f3e7a700 1 -- 172.21.15.59:0/3242943927 shutdown_connections 2023-09-30T02:06:19.435 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.432+0000 7fd2f3e7a700 1 -- 172.21.15.59:0/3242943927 wait complete. 2023-09-30T02:06:19.436 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.432+0000 7fd2f3e7a700 10 monclient: build_initial_monmap 2023-09-30T02:06:19.436 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.432+0000 7fd2f3e7a700 10 monclient: monmap: 2023-09-30T02:06:19.437 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:19.437 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:06:19.437 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:06:19.437 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:06:19.437 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:19.437 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:06:19.437 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:06:19.437 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:06:19.437 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:06:19.437 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:19.437 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.433+0000 7fd2f3e7a700 1 librados: starting msgr at 2023-09-30T02:06:19.437 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.433+0000 7fd2f3e7a700 1 librados: starting objecter 2023-09-30T02:06:19.438 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.433+0000 7fd2f3e7a700 1 Processor -- start 2023-09-30T02:06:19.438 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.433+0000 7fd2f3e7a700 1 -- start start 2023-09-30T02:06:19.438 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.433+0000 7fd2f3e7a700 1 librados: setting wanted keys 2023-09-30T02:06:19.438 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.433+0000 7fd2f3e7a700 1 librados: calling monclient init 2023-09-30T02:06:19.438 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.433+0000 7fd2f3e7a700 10 monclient: init 2023-09-30T02:06:19.438 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.433+0000 7fd2f3e7a700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:19.438 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.433+0000 7fd2f3e7a700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T02:06:19.438 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.433+0000 7fd2f3e7a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd2ec0a5b70 0x7fd2ec13ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:19.438 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.433+0000 7fd2f3e7a700 10 monclient(hunting): picked mon.noname-a con 0x7fd2ec0a5b70 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:06:19.438 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.433+0000 7fd2f3e7a700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd2ec0a5f40 0x7fd2ec151710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:19.438 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.433+0000 7fd2f3e7a700 10 monclient(hunting): picked mon.noname-b con 0x7fd2ec0a5f40 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:06:19.439 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.433+0000 7fd2f3e7a700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd2ec142290 0x7fd2ec151c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:19.439 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.433+0000 7fd2f3e7a700 10 monclient(hunting): picked mon.noname-c con 0x7fd2ec142290 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:19.439 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.433+0000 7fd2f3e7a700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:19.439 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.433+0000 7fd2f3e7a700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2ec13a8f0 con 0x7fd2ec0a5b70 2023-09-30T02:06:19.439 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.433+0000 7fd2f3e7a700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:19.439 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.433+0000 7fd2f3e7a700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2ec13a4f0 con 0x7fd2ec0a5f40 2023-09-30T02:06:19.439 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.433+0000 7fd2f3e7a700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:19.439 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.433+0000 7fd2f3e7a700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2ec152190 con 0x7fd2ec142290 2023-09-30T02:06:19.439 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.433+0000 7fd2f3e7a700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:19.439 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.433+0000 7fd2f3e7a700 10 monclient(hunting): authenticate will time out at 1824.735906s 2023-09-30T02:06:19.439 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.434+0000 7fd2f2417700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd2ec142290 0x7fd2ec151c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:19.439 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.434+0000 7fd2f2417700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd2ec142290 0x7fd2ec151c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:47444/0 (socket says 172.21.15.59:47444) 2023-09-30T02:06:19.440 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.434+0000 7fd2f2417700 1 -- 172.21.15.59:0/3338145973 learned_addr learned my addr 172.21.15.59:0/3338145973 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:06:19.440 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.434+0000 7fd2f2417700 10 monclient(hunting): get_auth_request con 0x7fd2ec142290 auth_method 0 2023-09-30T02:06:19.440 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.434+0000 7fd2f2417700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:19.440 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.434+0000 7fd2f2417700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:19.440 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.434+0000 7fd2f2417700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:19.440 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.435+0000 7fd2f2417700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:19.440 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.435+0000 7fd2f2417700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:19.440 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.435+0000 7fd2f2417700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:19.441 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.438+0000 7fd2f1415700 1 --2- 172.21.15.59:0/3338145973 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd2ec0a5f40 0x7fd2ec151710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:19.441 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.438+0000 7fd2f1415700 10 monclient(hunting): get_auth_request con 0x7fd2ec0a5f40 auth_method 0 2023-09-30T02:06:19.441 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.438+0000 7fd2f1415700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:19.441 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.438+0000 7fd2f1415700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:19.442 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.438+0000 7fd2f1415700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:19.442 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.439+0000 7fd2f2417700 10 monclient(hunting): handle_auth_done global_id 4250 payload 824 2023-09-30T02:06:19.442 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.439+0000 7fd2f2417700 1 -- 172.21.15.59:0/3338145973 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd2ec0a5f40 msgr2=0x7fd2ec151710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:19.442 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.439+0000 7fd2f2417700 1 --2- 172.21.15.59:0/3338145973 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd2ec0a5f40 0x7fd2ec151710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:19.442 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.439+0000 7fd2f2417700 1 -- 172.21.15.59:0/3338145973 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd2ec0a5b70 msgr2=0x7fd2ec13ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:19.442 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.439+0000 7fd2f2417700 1 --2- 172.21.15.59:0/3338145973 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd2ec0a5b70 0x7fd2ec13ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:19.442 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.439+0000 7fd2f2417700 10 monclient: _finish_hunting 0 2023-09-30T02:06:19.442 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.439+0000 7fd2f2417700 1 monclient: found mon.noname-c 2023-09-30T02:06:19.442 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.439+0000 7fd2f2417700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:19.442 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.439+0000 7fd2f2417700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.59:3301/0 2023-09-30T02:06:19.442 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.439+0000 7fd2f2417700 1 -- 172.21.15.59:0/3338145973 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2dc009420 con 0x7fd2ec142290 2023-09-30T02:06:19.442 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.441+0000 7fd2f2417700 1 --2- 172.21.15.59:0/3338145973 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd2ec142290 0x7fd2ec151c50 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd2e400e870 tx=0x7fd2e4008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:19.443 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.441+0000 7fd2e1ffb700 10 client.?.objecter ms_handle_connect 0x7fd2ec142290 2023-09-30T02:06:19.443 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.441+0000 7fd2e1ffb700 10 client.?.objecter resend_mon_ops 2023-09-30T02:06:19.449 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.449+0000 7fd2e1ffb700 1 -- 172.21.15.59:0/3338145973 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2e4013410 con 0x7fd2ec142290 2023-09-30T02:06:19.449 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.449+0000 7fd2e1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:19.449 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.449+0000 7fd2e1ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:06:19.449 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.449+0000 7fd2e1ffb700 10 monclient: dump: 2023-09-30T02:06:19.449 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:19.449 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:19.449 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:19.450 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:19.450 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:19.450 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:19.450 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:19.450 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:19.450 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:19.450 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:19.450 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.449+0000 7fd2e1ffb700 10 monclient: _finish_auth 0 2023-09-30T02:06:19.450 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.449+0000 7fd2e1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:19.450 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.449+0000 7fd2e1ffb700 1 -- 172.21.15.59:0/3338145973 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2e4013590 con 0x7fd2ec142290 2023-09-30T02:06:19.450 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.449+0000 7fd2e1ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:19.450 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.449+0000 7fd2e1ffb700 1 -- 172.21.15.59:0/3338145973 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2e4013950 con 0x7fd2ec142290 2023-09-30T02:06:19.451 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.449+0000 7fd2f3e7a700 5 monclient: authenticate success, global_id 4250 2023-09-30T02:06:19.451 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.449+0000 7fd2f3e7a700 10 monclient: _renew_subs 2023-09-30T02:06:19.451 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.449+0000 7fd2f3e7a700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:06:19.451 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.449+0000 7fd2f3e7a700 1 -- 172.21.15.59:0/3338145973 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2ec1523b0 con 0x7fd2ec142290 2023-09-30T02:06:19.451 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.449+0000 7fd2e1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:19.451 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.449+0000 7fd2e1ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:06:19.451 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.449+0000 7fd2e3fff700 4 set_mon_vals no callback set 2023-09-30T02:06:19.451 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.449+0000 7fd2e1ffb700 10 monclient: dump: 2023-09-30T02:06:19.451 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:19.451 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:19.451 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:19.451 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:19.452 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:19.452 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:19.452 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:19.452 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:19.452 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:19.452 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:19.452 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.449+0000 7fd2f3e7a700 10 client.4250.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:06:19.452 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.449+0000 7fd2f3e7a700 10 monclient: _renew_subs 2023-09-30T02:06:19.452 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.449+0000 7fd2f3e7a700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:06:19.452 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.449+0000 7fd2f3e7a700 1 -- 172.21.15.59:0/3338145973 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2ec13e320 con 0x7fd2ec142290 2023-09-30T02:06:19.452 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.449+0000 7fd2f3e7a700 1 librados: init done 2023-09-30T02:06:19.453 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.450+0000 7fd2f3e7a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:19.453 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.450+0000 7fd2f3e7a700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:19.453 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.450+0000 7fd2f3e7a700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:06:19.453 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.450+0000 7fd2f3e7a700 1 -- 172.21.15.59:0/3338145973 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2ec0a71b0 con 0x7fd2ec142290 2023-09-30T02:06:19.490 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.490+0000 7fd2e1ffb700 1 -- 172.21.15.59:0/3338145973 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fd2e4013cc0 con 0x7fd2ec142290 2023-09-30T02:06:19.491 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.491+0000 7fd2e1ffb700 1 --2- 172.21.15.59:0/3338145973 >> v2:172.21.15.110:6800/115870 conn(0x7fd2d0068960 0x7fd2d006ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:19.491 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.491+0000 7fd2e1ffb700 1 -- 172.21.15.59:0/3338145973 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fd2e4084540 con 0x7fd2ec142290 2023-09-30T02:06:19.491 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.491+0000 7fd2e1ffb700 10 client.4250.objecter ms_dispatch 0x7fd2ec001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:06:19.491 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.491+0000 7fd2e1ffb700 3 client.4250.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:06:19.491 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.491+0000 7fd2e1ffb700 3 client.4250.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:06:19.491 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.491+0000 7fd2f1c16700 1 --2- 172.21.15.59:0/3338145973 >> v2:172.21.15.110:6800/115870 conn(0x7fd2d0068960 0x7fd2d006ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:19.491 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.491+0000 7fd2e1ffb700 20 client.4250.objecter dump_active .. 0 homeless 2023-09-30T02:06:19.491 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.491+0000 7fd2f1c16700 10 monclient: get_auth_request con 0x7fd2d0068960 auth_method 0 2023-09-30T02:06:19.491 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.492+0000 7fd2e1ffb700 10 client.4250.objecter ms_handle_connect 0x7fd2d0068960 2023-09-30T02:06:19.491 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.492+0000 7fd2f1c16700 1 --2- 172.21.15.59:0/3338145973 >> v2:172.21.15.110:6800/115870 conn(0x7fd2d0068960 0x7fd2d006ae10 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd2e8001d90 tx=0x7fd2e8007ec0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:19.763 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.764+0000 7fd2e1ffb700 1 -- 172.21.15.59:0/3338145973 <== mon.2 v2:172.21.15.59:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd2e4051e60 con 0x7fd2ec142290 2023-09-30T02:06:19.764 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.764+0000 7fd2e1ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:19.764 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.764+0000 7fd2e1ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:06:19.975 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.976+0000 7fd2f3e7a700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-09-30T02:06:19.975 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.976+0000 7fd2f3e7a700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T02:06:19.975 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.976+0000 7fd2f3e7a700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:06:19.976 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:19.976+0000 7fd2f3e7a700 1 -- 172.21.15.59:0/3338145973 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd2ec13aa60 con 0x7fd2ec142290 2023-09-30T02:06:20.026 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.027+0000 7fd2e1ffb700 1 -- 172.21.15.59:0/3338145973 <== mon.2 v2:172.21.15.59:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+11935 (secure 0 0 0) 0x7fd2e4055480 con 0x7fd2ec142290 2023-09-30T02:06:20.026 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.027+0000 7fd2e1ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T02:06:20.027 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.027+0000 7fd2e1ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:06:20.027 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:06:20.027 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":25,"fsid":"623052cb-4f82-4b2b-b72b-3ed5be614613","created":"2023-09-30T02:05:11.679291+0000","modified":"2023-09-30T02:06:19.209689+0000","last_up_change":"2023-09-30T02:06:10.164005+0000","last_in_change":"2023-09-30T02:05:18.270595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-30T02:06:06.187725+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-09-30T02:06:15.695024+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"efdab44b-be57-4b81-8567-ff2deec1d3e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":116664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6801","nonce":116664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6803","nonce":116664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":116664}]},"public_addr":"172.21.15.59:6800/116664","cluster_addr":"172.21.15.59:6801/116664","heartbeat_back_addr":"172.21.15.59:6803/116664","heartbeat_front_addr":"172.21.15.59:6802/116664","state":["exists","up"]},{"osd":1,"uuid":"eff9786d-8c39-4044-91d2-a31f33e3b16e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":116665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6809","nonce":116665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6811","nonce":116665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":116665}]},"public_addr":"172.21.15.59:6808/116665","cluster_addr":"172.21.15.59:6809/116665","heartbeat_back_addr":"172.21.15.59:6811/116665","heartbeat_front_addr":"172.21.15.59:6810/116665","state":["exists","up"]},{"osd":2,"uuid":"b276ed4b-317b-4ddf-96ab-7ad71b8e3ff7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":116666}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6805","nonce":116666}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6807","nonce":116666}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":116666}]},"public_addr":"172.21.15.59:6804/116666","cluster_addr":"172.21.15.59:6805/116666","heartbeat_back_addr":"172.21.15.59:6807/116666","heartbeat_front_addr":"172.21.15.59:6806/116666","state":["exists","up"]},{"osd":3,"uuid":"48bfaf43-5846-4385-b9c6-6bc424cd5acd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":116667}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6813","nonce":116667}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6815","nonce":116667}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":116667}]},"public_addr":"172.21.15.59:6812/116667","cluster_addr":"172.21.15.59:6813/116667","heartbeat_back_addr":"172.21.15.59:6815/116667","heartbeat_front_addr":"172.21.15.59:6814/116667","state":["exists","up"]},{"osd":4,"uuid":"76fa5443-1571-4e72-9d22-6ab19e59839a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6801","nonce":116560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":116560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":116560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6803","nonce":116560}]},"public_addr":"172.21.15.110:6801/116560","cluster_addr":"172.21.15.110:6802/116560","heartbeat_back_addr":"172.21.15.110:6804/116560","heartbeat_front_addr":"172.21.15.110:6803/116560","state":["exists","up"]},{"osd":5,"uuid":"e9436478-4140-4fd6-ba2e-04fa6f6cf367","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6809","nonce":116562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":116562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":116562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6811","nonce":116562}]},"public_addr":"172.21.15.110:6809/116562","cluster_addr":"172.21.15.110:6810/116562","heartbeat_back_addr":"172.21.15.110:6812/116562","heartbeat_front_addr":"172.21.15.110:6811/116562","state":["exists","up"]},{"osd":6,"uuid":"76676042-2074-4cf8-9ef9-5c87bf604944","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6813","nonce":116563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":116563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":116563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6815","nonce":116563}]},"public_addr":"172.21.15.110:6813/116563","cluster_addr":"172.21.15.110:6814/116563","heartbeat_back_addr":"172.21.15.110:6816/116563","heartbeat_front_addr":"172.21.15.110:6815/116563","state":["exists","up"]},{"osd":7,"uuid":"045479f4-f7f7-4b95-bc38-a2c52ac31b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6805","nonce":116561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":116561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":116561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6807","nonce":116561}]},"public_addr":"172.21.15.110:6805/116561","cluster_addr":"172.21.15.110:6806/116561","heartbeat_back_addr":"172.21.15.110:6808/116561","heartbeat_front_addr":"172.21.15.110:6807/116561","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:06:05.267167+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:06:06.183162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:06:05.646500+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:06:06.546668+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:06:02.158984+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:06:01.902487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:06:01.310184+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:06:01.669641+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T02:06:20.029 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.029+0000 7fd2f3e7a700 10 librados: watch_flush enter 2023-09-30T02:06:20.029 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.029+0000 7fd2f3e7a700 10 librados: watch_flush exit 2023-09-30T02:06:20.029 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.029+0000 7fd2f3e7a700 20 client.4250.objecter shutdown clearing up homeless session... 2023-09-30T02:06:20.029 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.029+0000 7fd2f3e7a700 10 client.4250.objecter successfully canceled tick 2023-09-30T02:06:20.029 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.029+0000 7fd2f3e7a700 1 -- 172.21.15.59:0/3338145973 >> v2:172.21.15.110:6800/115870 conn(0x7fd2d0068960 msgr2=0x7fd2d006ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:20.029 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.029+0000 7fd2f3e7a700 1 --2- 172.21.15.59:0/3338145973 >> v2:172.21.15.110:6800/115870 conn(0x7fd2d0068960 0x7fd2d006ae10 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd2e8001d90 tx=0x7fd2e8007ec0 comp rx=0 tx=0).stop 2023-09-30T02:06:20.030 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.029+0000 7fd2f3e7a700 10 monclient: shutdown 2023-09-30T02:06:20.030 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.029+0000 7fd2f3e7a700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:20.030 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.029+0000 7fd2f3e7a700 1 -- 172.21.15.59:0/3338145973 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd2ec142290 msgr2=0x7fd2ec151c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:20.030 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.029+0000 7fd2f3e7a700 1 --2- 172.21.15.59:0/3338145973 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd2ec142290 0x7fd2ec151c50 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd2e400e870 tx=0x7fd2e4008040 comp rx=0 tx=0).stop 2023-09-30T02:06:20.030 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.029+0000 7fd2f3e7a700 1 -- 172.21.15.59:0/3338145973 shutdown_connections 2023-09-30T02:06:20.030 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.029+0000 7fd2f3e7a700 1 --2- 172.21.15.59:0/3338145973 >> v2:172.21.15.110:6800/115870 conn(0x7fd2d0068960 0x7fd2d006ae10 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:20.030 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.029+0000 7fd2f3e7a700 1 --2- 172.21.15.59:0/3338145973 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd2ec142290 0x7fd2ec151c50 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:20.030 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.029+0000 7fd2f3e7a700 1 --2- 172.21.15.59:0/3338145973 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd2ec0a5b70 0x7fd2ec13ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:20.030 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.029+0000 7fd2f3e7a700 1 --2- 172.21.15.59:0/3338145973 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd2ec0a5f40 0x7fd2ec151710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:20.030 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.029+0000 7fd2f3e7a700 1 -- 172.21.15.59:0/3338145973 >> 172.21.15.59:0/3338145973 conn(0x7fd2ec0631f0 msgr2=0x7fd2ec0afdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:20.031 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.029+0000 7fd2f3e7a700 1 -- 172.21.15.59:0/3338145973 shutdown_connections 2023-09-30T02:06:20.031 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.030+0000 7fd2f3e7a700 1 -- 172.21.15.59:0/3338145973 wait complete. 2023-09-30T02:06:20.031 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.030+0000 7fd2f3e7a700 1 librados: shutdown 2023-09-30T02:06:20.043 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-09-30T02:06:20.043 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-30T02:06:20.174 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.175+0000 7f69b3864700 10 monclient: get_monmap_and_config 2023-09-30T02:06:20.174 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.175+0000 7f69b3864700 10 monclient: build_initial_monmap 2023-09-30T02:06:20.175 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.175+0000 7f69b3864700 10 monclient: monmap: 2023-09-30T02:06:20.175 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:20.175 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:20.175 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:06:20.175615+0000 2023-09-30T02:06:20.175 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:06:20.175615+0000 2023-09-30T02:06:20.175 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:20.175 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:20.175 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:06:20.175 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:06:20.175 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:06:20.175 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:06:20.175 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:06:20.176 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:20.177 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.176+0000 7f69b3864700 1 Processor -- start 2023-09-30T02:06:20.177 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.176+0000 7f69b3864700 1 -- start start 2023-09-30T02:06:20.177 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.176+0000 7f69b3864700 10 monclient: init 2023-09-30T02:06:20.177 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.177+0000 7f69b3864700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:20.178 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.177+0000 7f69b3864700 10 monclient: _add_conns ranks=[2,0,4,3,1] 2023-09-30T02:06:20.178 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.177+0000 7f69b3864700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f69ac0a5b70 0x7f69ac0b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:20.178 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.177+0000 7f69b3864700 10 monclient(hunting): picked mon.noname-b con 0x7f69ac0a5b70 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:20.178 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.177+0000 7f69b3864700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f69ac0a5f40 addr v1:172.21.15.59:6789/0 2023-09-30T02:06:20.178 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.177+0000 7f69b3864700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f69ac142290 0x7f69ac13af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:20.178 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.177+0000 7f69b3864700 10 monclient(hunting): picked mon.noname-c con 0x7f69ac142290 addr v2:172.21.15.110:3300/0 2023-09-30T02:06:20.178 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.177+0000 7f69b3864700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69ac0a63b0 con 0x7f69ac0a5f40 2023-09-30T02:06:20.178 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.177+0000 7f69b3864700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:20.178 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.177+0000 7f69b3864700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69ac1426d0 con 0x7f69ac142290 2023-09-30T02:06:20.178 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.177+0000 7f69b3864700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:20.178 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.177+0000 7f69b3864700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69ac0a75a0 con 0x7f69ac0a5b70 2023-09-30T02:06:20.179 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.177+0000 7f69b3864700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:20.179 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.177+0000 7f69b3864700 10 monclient(hunting): authenticate will time out at 1825.479217s 2023-09-30T02:06:20.179 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.177+0000 7f69b1600700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f69ac0a5b70 0x7f69ac0b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:20.179 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.177+0000 7f69b0dff700 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f69ac0a5f40 0x7f69ac141b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:34962/0 (socket says 172.21.15.59:34962) 2023-09-30T02:06:20.179 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.178+0000 7f69b0dff700 1 -- 172.21.15.59:0/1542046280 learned_addr learned my addr 172.21.15.59:0/1542046280 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:06:20.179 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.178+0000 7f69b1e01700 1 --2- 172.21.15.59:0/1542046280 >> v2:172.21.15.110:3300/0 conn(0x7f69ac142290 0x7f69ac13af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:20.179 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.178+0000 7f69b1600700 10 monclient(hunting): get_auth_request con 0x7f69ac0a5b70 auth_method 0 2023-09-30T02:06:20.179 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.178+0000 7f69b1600700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:20.179 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.178+0000 7f69b1600700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:20.179 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.178+0000 7f69b1600700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:20.181 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.178+0000 7f69b1e01700 10 monclient(hunting): get_auth_request con 0x7f69ac142290 auth_method 0 2023-09-30T02:06:20.181 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.178+0000 7f69b1e01700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:20.181 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.178+0000 7f69b1e01700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:20.181 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.178+0000 7f69b1e01700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:20.181 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.179+0000 7f69b1600700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:20.181 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.179+0000 7f69b1600700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:20.181 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.179+0000 7f69b1600700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:20.181 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.179+0000 7f69b1e01700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:20.181 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.179+0000 7f69b1e01700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:20.181 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.179+0000 7f69b1e01700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:20.181 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.180+0000 7f69a3fff700 1 -- 172.21.15.59:0/1542046280 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3222060033 0 0) 0x7f69ac0a63b0 con 0x7f69ac0a5f40 2023-09-30T02:06:20.182 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.180+0000 7f69a3fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:20.182 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.180+0000 7f69a3fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:20.182 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.180+0000 7f69a3fff700 10 monclient(hunting): my global_id is 4275 2023-09-30T02:06:20.182 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.180+0000 7f69a3fff700 1 -- 172.21.15.59:0/1542046280 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6998003570 con 0x7f69ac0a5f40 2023-09-30T02:06:20.182 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.181+0000 7f69b1600700 10 monclient(hunting): handle_auth_done global_id 4253 payload 274 2023-09-30T02:06:20.182 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.181+0000 7f69b1600700 1 -- 172.21.15.59:0/1542046280 >> v2:172.21.15.110:3300/0 conn(0x7f69ac142290 msgr2=0x7f69ac13af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:06:20.182 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.181+0000 7f69b1600700 1 --2- 172.21.15.59:0/1542046280 >> v2:172.21.15.110:3300/0 conn(0x7f69ac142290 0x7f69ac13af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:20.182 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.181+0000 7f69b1600700 1 -- 172.21.15.59:0/1542046280 >> v1:172.21.15.59:6789/0 conn(0x7f69ac0a5f40 legacy=0x7f69ac141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:20.182 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.181+0000 7f69b1600700 10 monclient: _finish_hunting 0 2023-09-30T02:06:20.182 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.181+0000 7f69b1600700 1 monclient: found mon.noname-b 2023-09-30T02:06:20.182 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.181+0000 7f69b1600700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:20.182 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.181+0000 7f69b1600700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.59:3301/0 2023-09-30T02:06:20.183 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.181+0000 7f69b1600700 1 -- 172.21.15.59:0/1542046280 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69ac0af3f0 con 0x7f69ac0a5b70 2023-09-30T02:06:20.184 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.185+0000 7f69b1600700 1 --2- 172.21.15.59:0/1542046280 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f69ac0a5b70 0x7f69ac0b1500 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f69a80098e0 tx=0x7f69a80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0c3f6cefff49b0e server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:20.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.190+0000 7f69a3fff700 1 -- 172.21.15.59:0/1542046280 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69a8011630 con 0x7f69ac0a5b70 2023-09-30T02:06:20.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.190+0000 7f69a3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:20.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.190+0000 7f69a3fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:06:20.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.190+0000 7f69a3fff700 10 monclient: dump: 2023-09-30T02:06:20.190 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:20.190 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:20.190 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:20.191 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:20.191 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:20.191 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:20.191 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:20.191 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:20.191 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:20.191 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:20.191 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.190+0000 7f69a3fff700 10 monclient: _finish_auth 0 2023-09-30T02:06:20.191 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.190+0000 7f69a3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:20.191 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.190+0000 7f69a3fff700 1 -- 172.21.15.59:0/1542046280 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69a8011790 con 0x7f69ac0a5b70 2023-09-30T02:06:20.191 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.190+0000 7f69a3fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:20.191 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.190+0000 7f69a3fff700 1 -- 172.21.15.59:0/1542046280 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69a8011b30 con 0x7f69ac0a5b70 2023-09-30T02:06:20.192 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.190+0000 7f69b3864700 5 monclient: authenticate success, global_id 4253 2023-09-30T02:06:20.192 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.190+0000 7f69b3864700 10 monclient: get_monmap_and_config success 2023-09-30T02:06:20.192 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.190+0000 7f69a3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:20.192 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.190+0000 7f69b3864700 4 set_mon_vals no callback set 2023-09-30T02:06:20.192 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.190+0000 7f69a3fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:06:20.192 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.190+0000 7f69a3fff700 10 monclient: dump: 2023-09-30T02:06:20.192 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:20.192 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:20.192 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:20.192 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:20.192 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:20.192 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:20.193 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:20.193 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:20.193 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:20.193 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:20.196 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.191+0000 7f69b3864700 10 monclient: shutdown 2023-09-30T02:06:20.196 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.191+0000 7f69b3864700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:20.196 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.191+0000 7f69b3864700 1 -- 172.21.15.59:0/1542046280 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f69ac0a5b70 msgr2=0x7f69ac0b1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:20.196 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.191+0000 7f69b3864700 1 --2- 172.21.15.59:0/1542046280 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f69ac0a5b70 0x7f69ac0b1500 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f69a80098e0 tx=0x7f69a80091a0 comp rx=0 tx=0).stop 2023-09-30T02:06:20.196 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.191+0000 7f69b3864700 1 -- 172.21.15.59:0/1542046280 shutdown_connections 2023-09-30T02:06:20.196 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.191+0000 7f69b3864700 1 --2- 172.21.15.59:0/1542046280 >> v2:172.21.15.110:3300/0 conn(0x7f69ac142290 0x7f69ac13af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:20.196 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.191+0000 7f69b3864700 1 --2- 172.21.15.59:0/1542046280 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f69ac0a5b70 0x7f69ac0b1500 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:20.196 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.191+0000 7f69b3864700 1 -- 172.21.15.59:0/1542046280 >> 172.21.15.59:0/1542046280 conn(0x7f69ac0631f0 msgr2=0x7f69ac0635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:20.196 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.191+0000 7f69b3864700 1 -- 172.21.15.59:0/1542046280 shutdown_connections 2023-09-30T02:06:20.197 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.191+0000 7f69b3864700 1 -- 172.21.15.59:0/1542046280 wait complete. 2023-09-30T02:06:20.197 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.192+0000 7f69b3864700 10 monclient: build_initial_monmap 2023-09-30T02:06:20.197 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.192+0000 7f69b3864700 10 monclient: monmap: 2023-09-30T02:06:20.197 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:20.197 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:06:20.197 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:06:20.197 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:06:20.197 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:20.197 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:06:20.197 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:06:20.197 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:06:20.197 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:06:20.198 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:20.198 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.193+0000 7f69b3864700 1 librados: starting msgr at 2023-09-30T02:06:20.198 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.193+0000 7f69b3864700 1 librados: starting objecter 2023-09-30T02:06:20.198 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.193+0000 7f69b3864700 1 Processor -- start 2023-09-30T02:06:20.198 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.193+0000 7f69b3864700 1 -- start start 2023-09-30T02:06:20.198 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.193+0000 7f69b3864700 1 librados: setting wanted keys 2023-09-30T02:06:20.198 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.193+0000 7f69b3864700 1 librados: calling monclient init 2023-09-30T02:06:20.198 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.193+0000 7f69b3864700 10 monclient: init 2023-09-30T02:06:20.198 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.194+0000 7f69b3864700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:20.198 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.194+0000 7f69b3864700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T02:06:20.198 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.194+0000 7f69b3864700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f69ac0a5b70 0x7f69ac151550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:20.198 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.194+0000 7f69b3864700 10 monclient(hunting): picked mon.noname-b con 0x7f69ac0a5b70 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:06:20.199 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.194+0000 7f69b3864700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f69ac0a5f40 0x7f69ac13c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:20.199 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.194+0000 7f69b3864700 10 monclient(hunting): picked mon.noname-a con 0x7f69ac0a5f40 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:06:20.199 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.194+0000 7f69b3864700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f69ac142290 0x7f69ac13c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:20.199 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.194+0000 7f69b3864700 10 monclient(hunting): picked mon.noname-c con 0x7f69ac142290 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:20.199 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.194+0000 7f69b3864700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:20.199 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.194+0000 7f69b3864700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69ac13cdd0 con 0x7f69ac0a5f40 2023-09-30T02:06:20.199 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.194+0000 7f69b3864700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:20.199 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.194+0000 7f69b3864700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69ac0a9bf0 con 0x7f69ac0a5b70 2023-09-30T02:06:20.199 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.194+0000 7f69b3864700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:20.199 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.194+0000 7f69b3864700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69ac151b80 con 0x7f69ac142290 2023-09-30T02:06:20.199 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.194+0000 7f69b3864700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:20.200 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.194+0000 7f69b3864700 10 monclient(hunting): authenticate will time out at 1825.496561s 2023-09-30T02:06:20.200 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.195+0000 7f69b1600700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f69ac0a5b70 0x7f69ac151550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:20.200 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.195+0000 7f69b0dff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f69ac0a5f40 0x7f69ac13c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:20.200 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.195+0000 7f69b1600700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f69ac0a5b70 0x7f69ac151550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:38856/0 (socket says 172.21.15.59:38856) 2023-09-30T02:06:20.200 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.195+0000 7f69b1600700 1 -- 172.21.15.59:0/1056085006 learned_addr learned my addr 172.21.15.59:0/1056085006 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:06:20.200 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.195+0000 7f69b1600700 10 monclient(hunting): get_auth_request con 0x7f69ac0a5b70 auth_method 0 2023-09-30T02:06:20.200 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.195+0000 7f69b1600700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:20.200 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.195+0000 7f69b1600700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:20.200 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.195+0000 7f69b1600700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:20.200 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.196+0000 7f69b0dff700 10 monclient(hunting): get_auth_request con 0x7f69ac0a5f40 auth_method 0 2023-09-30T02:06:20.200 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.196+0000 7f69b0dff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:20.200 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.196+0000 7f69b0dff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:20.201 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.196+0000 7f69b0dff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:20.201 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.196+0000 7f69b1600700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:20.201 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.196+0000 7f69b1600700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:20.201 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.196+0000 7f69b1600700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:20.201 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.196+0000 7f69b1e01700 1 --2- 172.21.15.59:0/1056085006 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f69ac142290 0x7f69ac13c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:20.201 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.196+0000 7f69b0dff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:20.201 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.196+0000 7f69b0dff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:20.201 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.196+0000 7f69b0dff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:20.201 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.196+0000 7f69b1e01700 10 monclient(hunting): get_auth_request con 0x7f69ac142290 auth_method 0 2023-09-30T02:06:20.201 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.196+0000 7f69b1e01700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:20.201 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.196+0000 7f69b1e01700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:20.201 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.196+0000 7f69b1e01700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:20.203 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.197+0000 7f69b1e01700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:20.203 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.197+0000 7f69b1e01700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:20.203 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.197+0000 7f69b1e01700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:20.203 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.201+0000 7f69b1e01700 10 monclient(hunting): handle_auth_done global_id 4256 payload 824 2023-09-30T02:06:20.203 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.201+0000 7f69b1e01700 1 -- 172.21.15.59:0/1056085006 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f69ac0a5b70 msgr2=0x7f69ac151550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:20.203 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.201+0000 7f69b1e01700 1 --2- 172.21.15.59:0/1056085006 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f69ac0a5b70 0x7f69ac151550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:20.203 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.201+0000 7f69b1e01700 1 -- 172.21.15.59:0/1056085006 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f69ac0a5f40 msgr2=0x7f69ac13c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:20.203 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.201+0000 7f69b1e01700 1 --2- 172.21.15.59:0/1056085006 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f69ac0a5f40 0x7f69ac13c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:20.204 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.201+0000 7f69b1e01700 10 monclient: _finish_hunting 0 2023-09-30T02:06:20.204 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.201+0000 7f69b1e01700 1 monclient: found mon.noname-c 2023-09-30T02:06:20.204 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.201+0000 7f69b1e01700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:20.204 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.201+0000 7f69b1e01700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.59:3301/0 2023-09-30T02:06:20.204 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.201+0000 7f69b1e01700 1 -- 172.21.15.59:0/1056085006 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69a8009420 con 0x7f69ac142290 2023-09-30T02:06:20.204 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.204+0000 7f69b1e01700 1 --2- 172.21.15.59:0/1056085006 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f69ac142290 0x7f69ac13c890 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f69a400ecc0 tx=0x7f69a400b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:20.204 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.204+0000 7f69a17fa700 10 client.?.objecter ms_handle_connect 0x7f69ac142290 2023-09-30T02:06:20.204 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.204+0000 7f69a17fa700 10 client.?.objecter resend_mon_ops 2023-09-30T02:06:20.207 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.207+0000 7f69a17fa700 1 -- 172.21.15.59:0/1056085006 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69a400c8c0 con 0x7f69ac142290 2023-09-30T02:06:20.207 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.207+0000 7f69a17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:20.207 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.207+0000 7f69a17fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:06:20.208 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.207+0000 7f69a17fa700 10 monclient: dump: 2023-09-30T02:06:20.208 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:20.208 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:20.208 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:20.208 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:20.208 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:20.208 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:20.208 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:20.208 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:20.208 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:20.208 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:20.209 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.207+0000 7f69a17fa700 10 monclient: _finish_auth 0 2023-09-30T02:06:20.209 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.207+0000 7f69a17fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:20.209 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.207+0000 7f69a17fa700 1 -- 172.21.15.59:0/1056085006 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69a400f040 con 0x7f69ac142290 2023-09-30T02:06:20.209 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.208+0000 7f69b3864700 5 monclient: authenticate success, global_id 4256 2023-09-30T02:06:20.209 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.208+0000 7f69a17fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:20.209 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.208+0000 7f69a17fa700 1 -- 172.21.15.59:0/1056085006 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69a400cc90 con 0x7f69ac142290 2023-09-30T02:06:20.209 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.208+0000 7f69a17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:20.209 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.208+0000 7f69a17fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:06:20.209 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.208+0000 7f69a17fa700 10 monclient: dump: 2023-09-30T02:06:20.209 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:20.209 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:20.209 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:20.210 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:20.210 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:20.210 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:20.210 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:20.210 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:20.210 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:20.210 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:20.210 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.208+0000 7f69b3864700 10 monclient: _renew_subs 2023-09-30T02:06:20.210 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.208+0000 7f69a37fe700 4 set_mon_vals no callback set 2023-09-30T02:06:20.210 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.208+0000 7f69b3864700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:06:20.210 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.208+0000 7f69b3864700 1 -- 172.21.15.59:0/1056085006 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69ac152100 con 0x7f69ac142290 2023-09-30T02:06:20.210 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.208+0000 7f69b3864700 10 client.4256.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:06:20.211 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.208+0000 7f69b3864700 10 monclient: _renew_subs 2023-09-30T02:06:20.211 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.208+0000 7f69b3864700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:06:20.211 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.208+0000 7f69b3864700 1 -- 172.21.15.59:0/1056085006 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69ac13ee60 con 0x7f69ac142290 2023-09-30T02:06:20.211 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.208+0000 7f69b3864700 1 librados: init done 2023-09-30T02:06:20.211 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.209+0000 7f69b3864700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:20.211 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.209+0000 7f69b3864700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:20.211 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.209+0000 7f69b3864700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:06:20.212 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.209+0000 7f69b3864700 1 -- 172.21.15.59:0/1056085006 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69ac151da0 con 0x7f69ac142290 2023-09-30T02:06:20.523 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.524+0000 7f69a17fa700 1 -- 172.21.15.59:0/1056085006 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f69a4014690 con 0x7f69ac142290 2023-09-30T02:06:20.523 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.524+0000 7f69a17fa700 1 --2- 172.21.15.59:0/1056085006 >> v2:172.21.15.110:6800/115870 conn(0x7f6990068960 0x7f699006ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:20.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.524+0000 7f69a17fa700 1 -- 172.21.15.59:0/1056085006 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f69a4084e20 con 0x7f69ac142290 2023-09-30T02:06:20.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.524+0000 7f69a17fa700 10 client.4256.objecter ms_dispatch 0x7f69ac001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:06:20.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.524+0000 7f69a17fa700 3 client.4256.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:06:20.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.524+0000 7f69a17fa700 3 client.4256.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:06:20.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.524+0000 7f69b1600700 1 --2- 172.21.15.59:0/1056085006 >> v2:172.21.15.110:6800/115870 conn(0x7f6990068960 0x7f699006ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:20.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.524+0000 7f69b1600700 10 monclient: get_auth_request con 0x7f6990068960 auth_method 0 2023-09-30T02:06:20.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.524+0000 7f69a17fa700 20 client.4256.objecter dump_active .. 0 homeless 2023-09-30T02:06:20.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.524+0000 7f69a17fa700 1 -- 172.21.15.59:0/1056085006 <== mon.2 v2:172.21.15.59:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f69a4085270 con 0x7f69ac142290 2023-09-30T02:06:20.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.524+0000 7f69a17fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:20.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.524+0000 7f69a17fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:06:20.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.525+0000 7f69a17fa700 10 client.4256.objecter ms_handle_connect 0x7f6990068960 2023-09-30T02:06:20.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.525+0000 7f69b1600700 1 --2- 172.21.15.59:0/1056085006 >> v2:172.21.15.110:6800/115870 conn(0x7f6990068960 0x7f699006ae10 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f69a8011fd0 tx=0x7f69a8017000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:20.762 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.763+0000 7f69b3864700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-09-30T02:06:20.762 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.763+0000 7f69b3864700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T02:06:20.763 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.763+0000 7f69b3864700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:06:20.763 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:20.763+0000 7f69b3864700 1 -- 172.21.15.59:0/1056085006 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f69ac064a00 con 0x7f69ac142290 2023-09-30T02:06:21.103 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.104+0000 7f69a17fa700 1 -- 172.21.15.59:0/1056085006 <== mon.2 v2:172.21.15.59:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+11935 (secure 0 0 0) 0x7f69a4052740 con 0x7f69ac142290 2023-09-30T02:06:21.104 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.104+0000 7f69a17fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T02:06:21.104 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.104+0000 7f69a17fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:06:21.104 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:06:21.104 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":25,"fsid":"623052cb-4f82-4b2b-b72b-3ed5be614613","created":"2023-09-30T02:05:11.679291+0000","modified":"2023-09-30T02:06:19.209689+0000","last_up_change":"2023-09-30T02:06:10.164005+0000","last_in_change":"2023-09-30T02:05:18.270595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-30T02:06:06.187725+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-09-30T02:06:15.695024+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"efdab44b-be57-4b81-8567-ff2deec1d3e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":116664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6801","nonce":116664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6803","nonce":116664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":116664}]},"public_addr":"172.21.15.59:6800/116664","cluster_addr":"172.21.15.59:6801/116664","heartbeat_back_addr":"172.21.15.59:6803/116664","heartbeat_front_addr":"172.21.15.59:6802/116664","state":["exists","up"]},{"osd":1,"uuid":"eff9786d-8c39-4044-91d2-a31f33e3b16e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":116665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6809","nonce":116665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6811","nonce":116665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":116665}]},"public_addr":"172.21.15.59:6808/116665","cluster_addr":"172.21.15.59:6809/116665","heartbeat_back_addr":"172.21.15.59:6811/116665","heartbeat_front_addr":"172.21.15.59:6810/116665","state":["exists","up"]},{"osd":2,"uuid":"b276ed4b-317b-4ddf-96ab-7ad71b8e3ff7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":116666}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6805","nonce":116666}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6807","nonce":116666}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":116666}]},"public_addr":"172.21.15.59:6804/116666","cluster_addr":"172.21.15.59:6805/116666","heartbeat_back_addr":"172.21.15.59:6807/116666","heartbeat_front_addr":"172.21.15.59:6806/116666","state":["exists","up"]},{"osd":3,"uuid":"48bfaf43-5846-4385-b9c6-6bc424cd5acd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":116667}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6813","nonce":116667}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6815","nonce":116667}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":116667}]},"public_addr":"172.21.15.59:6812/116667","cluster_addr":"172.21.15.59:6813/116667","heartbeat_back_addr":"172.21.15.59:6815/116667","heartbeat_front_addr":"172.21.15.59:6814/116667","state":["exists","up"]},{"osd":4,"uuid":"76fa5443-1571-4e72-9d22-6ab19e59839a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6801","nonce":116560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":116560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":116560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6803","nonce":116560}]},"public_addr":"172.21.15.110:6801/116560","cluster_addr":"172.21.15.110:6802/116560","heartbeat_back_addr":"172.21.15.110:6804/116560","heartbeat_front_addr":"172.21.15.110:6803/116560","state":["exists","up"]},{"osd":5,"uuid":"e9436478-4140-4fd6-ba2e-04fa6f6cf367","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6809","nonce":116562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":116562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":116562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6811","nonce":116562}]},"public_addr":"172.21.15.110:6809/116562","cluster_addr":"172.21.15.110:6810/116562","heartbeat_back_addr":"172.21.15.110:6812/116562","heartbeat_front_addr":"172.21.15.110:6811/116562","state":["exists","up"]},{"osd":6,"uuid":"76676042-2074-4cf8-9ef9-5c87bf604944","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6813","nonce":116563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":116563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":116563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6815","nonce":116563}]},"public_addr":"172.21.15.110:6813/116563","cluster_addr":"172.21.15.110:6814/116563","heartbeat_back_addr":"172.21.15.110:6816/116563","heartbeat_front_addr":"172.21.15.110:6815/116563","state":["exists","up"]},{"osd":7,"uuid":"045479f4-f7f7-4b95-bc38-a2c52ac31b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6805","nonce":116561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":116561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":116561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6807","nonce":116561}]},"public_addr":"172.21.15.110:6805/116561","cluster_addr":"172.21.15.110:6806/116561","heartbeat_back_addr":"172.21.15.110:6808/116561","heartbeat_front_addr":"172.21.15.110:6807/116561","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:06:05.267167+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:06:06.183162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:06:05.646500+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:06:06.546668+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:06:02.158984+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:06:01.902487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:06:01.310184+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:06:01.669641+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T02:06:21.106 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.107+0000 7f69b3864700 10 librados: watch_flush enter 2023-09-30T02:06:21.107 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.107+0000 7f69b3864700 10 librados: watch_flush exit 2023-09-30T02:06:21.107 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.107+0000 7f69b3864700 20 client.4256.objecter shutdown clearing up homeless session... 2023-09-30T02:06:21.107 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.107+0000 7f69b3864700 10 client.4256.objecter successfully canceled tick 2023-09-30T02:06:21.107 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.107+0000 7f69b3864700 1 -- 172.21.15.59:0/1056085006 >> v2:172.21.15.110:6800/115870 conn(0x7f6990068960 msgr2=0x7f699006ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:21.107 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.107+0000 7f69b3864700 1 --2- 172.21.15.59:0/1056085006 >> v2:172.21.15.110:6800/115870 conn(0x7f6990068960 0x7f699006ae10 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f69a8011fd0 tx=0x7f69a8017000 comp rx=0 tx=0).stop 2023-09-30T02:06:21.107 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.107+0000 7f69b3864700 10 monclient: shutdown 2023-09-30T02:06:21.107 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.107+0000 7f69b3864700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:21.107 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.107+0000 7f69b3864700 1 -- 172.21.15.59:0/1056085006 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f69ac142290 msgr2=0x7f69ac13c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:21.107 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.107+0000 7f69b3864700 1 --2- 172.21.15.59:0/1056085006 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f69ac142290 0x7f69ac13c890 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f69a400ecc0 tx=0x7f69a400b040 comp rx=0 tx=0).stop 2023-09-30T02:06:21.107 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.107+0000 7f69b3864700 1 -- 172.21.15.59:0/1056085006 shutdown_connections 2023-09-30T02:06:21.107 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.107+0000 7f69b3864700 1 --2- 172.21.15.59:0/1056085006 >> v2:172.21.15.110:6800/115870 conn(0x7f6990068960 0x7f699006ae10 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:21.108 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.107+0000 7f69b3864700 1 --2- 172.21.15.59:0/1056085006 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f69ac0a5b70 0x7f69ac151550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:21.108 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.107+0000 7f69b3864700 1 --2- 172.21.15.59:0/1056085006 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f69ac142290 0x7f69ac13c890 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:21.108 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.107+0000 7f69b3864700 1 --2- 172.21.15.59:0/1056085006 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f69ac0a5f40 0x7f69ac13c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:21.108 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.107+0000 7f69b3864700 1 -- 172.21.15.59:0/1056085006 >> 172.21.15.59:0/1056085006 conn(0x7f69ac0631f0 msgr2=0x7f69ac0b3220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:21.108 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.107+0000 7f69b3864700 1 -- 172.21.15.59:0/1056085006 shutdown_connections 2023-09-30T02:06:21.108 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.107+0000 7f69b3864700 1 -- 172.21.15.59:0/1056085006 wait complete. 2023-09-30T02:06:21.108 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.107+0000 7f69b3864700 1 librados: shutdown 2023-09-30T02:06:21.120 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-09-30T02:06:21.121 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-09-30T02:06:21.121 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-09-30T02:06:21.122 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-09-30T02:06:21.122 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-09-30T02:06:21.122 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-09-30T02:06:21.123 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-09-30T02:06:21.123 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-09-30T02:06:21.373 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.374+0000 7f9d199df700 10 monclient: get_monmap_and_config 2023-09-30T02:06:21.374 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.374+0000 7f9d199df700 10 monclient: build_initial_monmap 2023-09-30T02:06:21.374 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.374+0000 7f9d199df700 10 monclient: monmap: 2023-09-30T02:06:21.374 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:21.374 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:21.374 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:06:21.374837+0000 2023-09-30T02:06:21.374 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:06:21.374837+0000 2023-09-30T02:06:21.374 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:21.374 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:21.374 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:06:21.374 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:06:21.375 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:06:21.375 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:06:21.375 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:06:21.375 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:21.376 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.377+0000 7f9d199df700 1 Processor -- start 2023-09-30T02:06:21.376 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.377+0000 7f9d199df700 1 -- start start 2023-09-30T02:06:21.377 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.375+0000 7f291972e700 10 monclient: get_monmap_and_config 2023-09-30T02:06:21.377 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.375+0000 7f291972e700 10 monclient: build_initial_monmap 2023-09-30T02:06:21.377 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.375+0000 7f291972e700 10 monclient: monmap: 2023-09-30T02:06:21.377 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:21.377 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:21.377 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:06:21.375620+0000 2023-09-30T02:06:21.377 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:06:21.375620+0000 2023-09-30T02:06:21.377 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:21.377 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:21.378 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:06:21.378 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:06:21.378 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:06:21.378 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:06:21.378 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:06:21.378 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:21.378 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.377+0000 7f9d199df700 10 monclient: init 2023-09-30T02:06:21.379 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.378+0000 7f9d199df700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:21.379 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.378+0000 7f9d199df700 10 monclient: _add_conns ranks=[1,3,2,0,4] 2023-09-30T02:06:21.379 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.378+0000 7f9d199df700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9d140b1480 addr v1:172.21.15.110:6789/0 2023-09-30T02:06:21.379 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.378+0000 7f9d199df700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f9d140b1850 0x7f9d14145bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:21.379 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.378+0000 7f9d199df700 10 monclient(hunting): picked mon.noname-a con 0x7f9d140b1850 addr v2:172.21.15.59:3300/0 2023-09-30T02:06:21.379 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.378+0000 7f9d199df700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9d14140010 0x7f9d14146110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:21.379 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.378+0000 7f9d199df700 10 monclient(hunting): picked mon.noname-b con 0x7f9d14140010 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:21.379 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.378+0000 7f9d199df700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d140afda0 con 0x7f9d140b1480 2023-09-30T02:06:21.379 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.378+0000 7f9d199df700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:21.379 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.378+0000 7f9d199df700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d140b0230 con 0x7f9d140b1850 2023-09-30T02:06:21.379 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.378+0000 7f9d199df700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:21.380 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.378+0000 7f9d199df700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d1406b0a0 con 0x7f9d14140010 2023-09-30T02:06:21.380 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.378+0000 7f9d199df700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:21.380 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.378+0000 7f9d199df700 10 monclient(hunting): authenticate will time out at 1826.680675s 2023-09-30T02:06:21.380 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.379+0000 7f291972e700 1 Processor -- start 2023-09-30T02:06:21.381 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.380+0000 7f9d12ffd700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f9d140b1480 0x7f9d1413f900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.59:54792/0 (socket says 172.21.15.59:54792) 2023-09-30T02:06:21.381 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.380+0000 7f9d12ffd700 1 -- 172.21.15.59:0/266507798 learned_addr learned my addr 172.21.15.59:0/266507798 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:06:21.381 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.380+0000 7f9d137fe700 1 --2- 172.21.15.59:0/266507798 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9d14140010 0x7f9d14146110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:21.381 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.380+0000 7f9d137fe700 10 monclient(hunting): get_auth_request con 0x7f9d14140010 auth_method 0 2023-09-30T02:06:21.381 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.380+0000 7f9d137fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:21.382 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.380+0000 7f9d137fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:21.382 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.380+0000 7f9d137fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:21.382 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.379+0000 7f291972e700 1 -- start start 2023-09-30T02:06:21.382 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.379+0000 7f291972e700 10 monclient: init 2023-09-30T02:06:21.382 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.380+0000 7f291972e700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:21.382 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.380+0000 7f291972e700 10 monclient: _add_conns ranks=[0,4,2,1,3] 2023-09-30T02:06:21.382 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.380+0000 7f291972e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f29140b13f0 addr v1:172.21.15.59:6789/0 2023-09-30T02:06:21.383 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.380+0000 7f291972e700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f2914069fd0 0x7f291413dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:21.383 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.380+0000 7f291972e700 10 monclient(hunting): picked mon.noname-c con 0x7f2914069fd0 addr v2:172.21.15.110:3300/0 2023-09-30T02:06:21.383 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.380+0000 7f291972e700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f291413e140 0x7f291413a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:21.383 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.380+0000 7f291972e700 10 monclient(hunting): picked mon.noname-b con 0x7f291413e140 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:21.383 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.380+0000 7f291972e700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f291406db80 con 0x7f29140b13f0 2023-09-30T02:06:21.383 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.380+0000 7f291972e700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:21.383 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.380+0000 7f291972e700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f291406a430 con 0x7f2914069fd0 2023-09-30T02:06:21.383 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.380+0000 7f291972e700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:21.383 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.380+0000 7f291972e700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f291406c560 con 0x7f291413e140 2023-09-30T02:06:21.383 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.380+0000 7f291972e700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:21.383 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.380+0000 7f291972e700 10 monclient(hunting): authenticate will time out at 1826.682820s 2023-09-30T02:06:21.384 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.381+0000 7f29137fe700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f291413e140 0x7f291413a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:21.384 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.382+0000 7f29137fe700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f291413e140 0x7f291413a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:47476/0 (socket says 172.21.15.59:47476) 2023-09-30T02:06:21.384 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.382+0000 7f29137fe700 1 -- 172.21.15.59:0/2417451478 learned_addr learned my addr 172.21.15.59:0/2417451478 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:06:21.384 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.382+0000 7f29137fe700 10 monclient(hunting): get_auth_request con 0x7f291413e140 auth_method 0 2023-09-30T02:06:21.384 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.382+0000 7f29137fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:21.384 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.382+0000 7f29137fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:21.384 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.382+0000 7f29137fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:21.384 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.375+0000 7fbe64a5d700 10 monclient: get_monmap_and_config 2023-09-30T02:06:21.384 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.375+0000 7fbe64a5d700 10 monclient: build_initial_monmap 2023-09-30T02:06:21.385 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.375+0000 7fbe64a5d700 10 monclient: monmap: 2023-09-30T02:06:21.385 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:21.385 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:21.385 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:06:21.375977+0000 2023-09-30T02:06:21.385 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:06:21.375977+0000 2023-09-30T02:06:21.385 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:21.385 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:21.385 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:06:21.385 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:06:21.385 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:06:21.385 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:06:21.386 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:06:21.386 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:21.386 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.380+0000 7fbe64a5d700 1 Processor -- start 2023-09-30T02:06:21.386 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.380+0000 7fbe64a5d700 1 -- start start 2023-09-30T02:06:21.386 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.380+0000 7fbe64a5d700 10 monclient: init 2023-09-30T02:06:21.386 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.381+0000 7fbe64a5d700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:21.386 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.381+0000 7fbe64a5d700 10 monclient: _add_conns ranks=[3,2,4,1,0] 2023-09-30T02:06:21.386 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.382+0000 7fbe64a5d700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fbe60069b10 0x7fbe6006b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:21.386 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.382+0000 7fbe64a5d700 10 monclient(hunting): picked mon.noname-a con 0x7fbe60069b10 addr v2:172.21.15.59:3300/0 2023-09-30T02:06:21.386 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.382+0000 7fbe64a5d700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fbe60087810 0x7fbe6006bde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:21.386 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.382+0000 7fbe64a5d700 10 monclient(hunting): picked mon.noname-b con 0x7fbe60087810 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:21.386 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.382+0000 7fbe64a5d700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fbe600ab2d0 0x7fbe600ab770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:21.387 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.382+0000 7fbe64a5d700 10 monclient(hunting): picked mon.noname-c con 0x7fbe600ab2d0 addr v2:172.21.15.110:3300/0 2023-09-30T02:06:21.387 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.382+0000 7fbe64a5d700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:21.387 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.382+0000 7fbe64a5d700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe60069ee0 con 0x7fbe60069b10 2023-09-30T02:06:21.387 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.382+0000 7fbe64a5d700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:21.387 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.382+0000 7fbe64a5d700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe600ace90 con 0x7fbe600ab2d0 2023-09-30T02:06:21.387 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.382+0000 7fbe64a5d700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:21.387 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.382+0000 7fbe64a5d700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe6006cd50 con 0x7fbe60087810 2023-09-30T02:06:21.387 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.382+0000 7fbe64a5d700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:21.387 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.382+0000 7fbe64a5d700 10 monclient(hunting): authenticate will time out at 1826.684103s 2023-09-30T02:06:21.388 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.383+0000 7f29137fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:21.388 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.383+0000 7f29137fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:21.388 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.383+0000 7f29137fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:21.388 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.386+0000 7f29127fc700 1 --2- 172.21.15.59:0/2417451478 >> v2:172.21.15.110:3300/0 conn(0x7f2914069fd0 0x7f291413dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:21.388 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.386+0000 7f29127fc700 10 monclient(hunting): get_auth_request con 0x7f2914069fd0 auth_method 0 2023-09-30T02:06:21.388 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.386+0000 7f29127fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:21.388 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.386+0000 7f29127fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:21.388 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.386+0000 7f29127fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:21.389 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.385+0000 7f9d11ffb700 1 -- 172.21.15.59:0/266507798 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1564461709 0 0) 0x7f9d140afda0 con 0x7f9d140b1480 2023-09-30T02:06:21.389 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.385+0000 7f9d11ffb700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:21.389 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.385+0000 7f9d11ffb700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:21.389 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.385+0000 7f9d11ffb700 10 monclient(hunting): my global_id is 4297 2023-09-30T02:06:21.389 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.385+0000 7f9d11ffb700 1 -- 172.21.15.59:0/266507798 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9cfc003570 con 0x7f9d140b1480 2023-09-30T02:06:21.389 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.386+0000 7ff3483d3700 10 monclient: get_monmap_and_config 2023-09-30T02:06:21.389 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.386+0000 7ff3483d3700 10 monclient: build_initial_monmap 2023-09-30T02:06:21.390 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.386+0000 7ff3483d3700 10 monclient: monmap: 2023-09-30T02:06:21.390 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:21.390 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:21.390 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:06:21.386923+0000 2023-09-30T02:06:21.390 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:06:21.386923+0000 2023-09-30T02:06:21.390 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:21.390 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:21.390 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:06:21.390 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:06:21.390 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:06:21.390 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:06:21.391 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:06:21.391 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:21.391 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.387+0000 7f29127fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:21.391 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.387+0000 7f29127fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:21.391 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.387+0000 7f29127fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:21.392 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.387+0000 7f29137fe700 10 monclient(hunting): handle_auth_done global_id 4262 payload 274 2023-09-30T02:06:21.392 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.387+0000 7f29137fe700 1 -- 172.21.15.59:0/2417451478 >> v2:172.21.15.110:3300/0 conn(0x7f2914069fd0 msgr2=0x7f291413dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:06:21.392 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.387+0000 7f29137fe700 1 --2- 172.21.15.59:0/2417451478 >> v2:172.21.15.110:3300/0 conn(0x7f2914069fd0 0x7f291413dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:21.392 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.387+0000 7f29137fe700 1 -- 172.21.15.59:0/2417451478 >> v1:172.21.15.59:6789/0 conn(0x7f29140b13f0 legacy=0x7f291406bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:06:21.392 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.387+0000 7f29137fe700 10 monclient: _finish_hunting 0 2023-09-30T02:06:21.392 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.387+0000 7f29137fe700 1 monclient: found mon.noname-b 2023-09-30T02:06:21.392 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.387+0000 7f29137fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:21.392 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.387+0000 7f29137fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.59:3301/0 2023-09-30T02:06:21.392 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.387+0000 7f29137fe700 1 -- 172.21.15.59:0/2417451478 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f291406d480 con 0x7f291413e140 2023-09-30T02:06:21.392 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.390+0000 7f29137fe700 1 --2- 172.21.15.59:0/2417451478 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f291413e140 0x7f291413a750 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f290c009f20 tx=0x7f290c009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f82ec547ac9697 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:21.393 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.391+0000 7ff3483d3700 1 Processor -- start 2023-09-30T02:06:21.393 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.392+0000 7ff3483d3700 1 -- start start 2023-09-30T02:06:21.393 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.392+0000 7ff3483d3700 10 monclient: init 2023-09-30T02:06:21.393 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.391+0000 7f32f314d700 10 monclient: get_monmap_and_config 2023-09-30T02:06:21.393 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.391+0000 7f32f314d700 10 monclient: build_initial_monmap 2023-09-30T02:06:21.393 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.391+0000 7f32f314d700 10 monclient: monmap: 2023-09-30T02:06:21.393 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:21.393 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:21.394 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:06:21.392153+0000 2023-09-30T02:06:21.394 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:06:21.392153+0000 2023-09-30T02:06:21.394 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:21.394 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:21.394 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:06:21.394 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:06:21.394 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:06:21.394 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:06:21.394 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:06:21.394 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:21.395 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.393+0000 7ff3483d3700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:21.395 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.393+0000 7ff3483d3700 10 monclient: _add_conns ranks=[1,2,3,0,4] 2023-09-30T02:06:21.395 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.393+0000 7ff3483d3700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff340069e90 addr v1:172.21.15.110:6789/0 2023-09-30T02:06:21.395 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.393+0000 7ff3483d3700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff34006ba90 0x7ff34006e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:21.395 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.393+0000 7ff3483d3700 10 monclient(hunting): picked mon.noname-b con 0x7ff34006ba90 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:21.395 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.393+0000 7ff3483d3700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7ff3400ab2d0 0x7ff3400ab710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:21.395 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.393+0000 7ff3483d3700 10 monclient(hunting): picked mon.noname-a con 0x7ff3400ab2d0 addr v2:172.21.15.59:3300/0 2023-09-30T02:06:21.396 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.393+0000 7ff3483d3700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff34006a260 con 0x7ff340069e90 2023-09-30T02:06:21.396 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.393+0000 7ff3483d3700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:21.396 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.393+0000 7ff3483d3700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff34006c080 con 0x7ff3400ab2d0 2023-09-30T02:06:21.396 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.393+0000 7ff3483d3700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:21.396 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.393+0000 7ff3483d3700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3400ace80 con 0x7ff34006ba90 2023-09-30T02:06:21.396 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.393+0000 7ff3483d3700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:21.396 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.393+0000 7ff3483d3700 10 monclient(hunting): authenticate will time out at 1826.695272s 2023-09-30T02:06:21.397 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.392+0000 7fbe5ed9d700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fbe600ab2d0 0x7fbe600ab770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:21.397 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.392+0000 7fbe5ed9d700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fbe600ab2d0 0x7fbe600ab770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:38870/0 (socket says 172.21.15.59:38870) 2023-09-30T02:06:21.397 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.392+0000 7fbe5ed9d700 1 -- 172.21.15.59:0/2929340633 learned_addr learned my addr 172.21.15.59:0/2929340633 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:06:21.397 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.392+0000 7fbe5ed9d700 10 monclient(hunting): get_auth_request con 0x7fbe600ab2d0 auth_method 0 2023-09-30T02:06:21.397 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.392+0000 7fbe5ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:21.397 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.392+0000 7fbe5ed9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:21.397 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.392+0000 7fbe5ed9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:21.397 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.394+0000 7fbe5dd9b700 1 --2- 172.21.15.59:0/2929340633 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fbe60087810 0x7fbe6006bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:21.397 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.394+0000 7fbe5dd9b700 10 monclient(hunting): get_auth_request con 0x7fbe60087810 auth_method 0 2023-09-30T02:06:21.397 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.394+0000 7fbe5dd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:21.397 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.394+0000 7fbe5dd9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:21.398 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.394+0000 7fbe5dd9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:21.398 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.395+0000 7f32f314d700 1 Processor -- start 2023-09-30T02:06:21.398 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.395+0000 7f32f314d700 1 -- start start 2023-09-30T02:06:21.398 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.395+0000 7f32f314d700 10 monclient: init 2023-09-30T02:06:21.398 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.396+0000 7f32f314d700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:21.398 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.396+0000 7f32f314d700 10 monclient: _add_conns ranks=[1,2,3,0,4] 2023-09-30T02:06:21.399 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.396+0000 7f32f314d700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f32ec0b13f0 addr v1:172.21.15.110:6789/0 2023-09-30T02:06:21.399 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.396+0000 7f32f314d700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f32ec069fd0 0x7f32ec13dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:21.399 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.396+0000 7f32f314d700 10 monclient(hunting): picked mon.noname-b con 0x7f32ec069fd0 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:21.399 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.396+0000 7f32f314d700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f32ec13e140 0x7f32ec13a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:21.399 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.396+0000 7f32f314d700 10 monclient(hunting): picked mon.noname-a con 0x7f32ec13e140 addr v2:172.21.15.59:3300/0 2023-09-30T02:06:21.399 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.396+0000 7f32f314d700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32ec06db80 con 0x7f32ec0b13f0 2023-09-30T02:06:21.399 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.396+0000 7f32f314d700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:21.399 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.396+0000 7f32f314d700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32ec06c560 con 0x7f32ec13e140 2023-09-30T02:06:21.399 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.396+0000 7f32f314d700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:21.399 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.396+0000 7f32f314d700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32ec06a400 con 0x7f32ec069fd0 2023-09-30T02:06:21.399 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.396+0000 7f32f314d700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:21.399 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.396+0000 7f32f314d700 10 monclient(hunting): authenticate will time out at 1826.698781s 2023-09-30T02:06:21.400 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.397+0000 7fbe5ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:21.400 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.397+0000 7fbe5ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:21.400 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.397+0000 7fbe5ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:21.400 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.397+0000 7ff34616f700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7ff340069e90 0x7ff34006d9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.59:54802/0 (socket says 172.21.15.59:54802) 2023-09-30T02:06:21.400 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.397+0000 7ff34616f700 1 -- 172.21.15.59:0/4021846443 learned_addr learned my addr 172.21.15.59:0/4021846443 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:06:21.401 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.397+0000 7f9d137fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:21.401 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.397+0000 7f9d137fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:21.401 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.397+0000 7f9d137fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:21.401 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.397+0000 7f9d11ffb700 1 -- 172.21.15.59:0/266507798 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 371926426 0 0) 0x7f9cfc003570 con 0x7f9d140b1480 2023-09-30T02:06:21.401 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.397+0000 7f9d11ffb700 1 -- 172.21.15.59:0/266507798 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9d14140010 msgr2=0x7f9d14146110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:06:21.401 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.397+0000 7f9d11ffb700 1 --2- 172.21.15.59:0/266507798 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9d14140010 0x7f9d14146110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:21.401 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.397+0000 7f9d11ffb700 1 -- 172.21.15.59:0/266507798 >> v2:172.21.15.59:3300/0 conn(0x7f9d140b1850 msgr2=0x7f9d14145bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:06:21.401 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.397+0000 7f9d11ffb700 1 --2- 172.21.15.59:0/266507798 >> v2:172.21.15.59:3300/0 conn(0x7f9d140b1850 0x7f9d14145bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:21.401 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.397+0000 7f9d11ffb700 10 monclient: _finish_hunting 0 2023-09-30T02:06:21.402 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.397+0000 7f9d11ffb700 1 monclient: found mon.noname-c-legacy 2023-09-30T02:06:21.402 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.397+0000 7f9d11ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:21.402 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.397+0000 7f9d11ffb700 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.110:6789/0 2023-09-30T02:06:21.402 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.397+0000 7f9d11ffb700 1 -- 172.21.15.59:0/266507798 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d14146650 con 0x7f9d140b1480 2023-09-30T02:06:21.402 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.397+0000 7f9d11ffb700 10 monclient: _finish_auth 0 2023-09-30T02:06:21.402 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.397+0000 7f9d11ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:21.402 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.397+0000 7f9d11ffb700 1 -- 172.21.15.59:0/266507798 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2174283827 0 0) 0x7f9d08001470 con 0x7f9d140b1480 2023-09-30T02:06:21.402 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.398+0000 7f9d11ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:21.402 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.398+0000 7f9d11ffb700 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2023-09-30T02:06:21.402 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.398+0000 7f9d11ffb700 10 monclient: dump: 2023-09-30T02:06:21.402 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:21.403 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:21.403 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:21.403 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:21.403 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:21.403 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:21.403 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:21.403 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:21.403 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:21.403 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:21.403 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.398+0000 7f9d11ffb700 1 monclient: mon.b has (v2) addrs [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] but i'm connected to v1:172.21.15.110:6789/0, reconnecting 2023-09-30T02:06:21.403 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.398+0000 7f9d11ffb700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:21.403 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.398+0000 7f9d11ffb700 1 -- 172.21.15.59:0/266507798 >> v1:172.21.15.110:6789/0 conn(0x7f9d140b1480 legacy=0x7f9d1413f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:21.404 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.398+0000 7f9d11ffb700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T02:06:21.404 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.398+0000 7f9d11ffb700 1 --2- 172.21.15.59:0/266507798 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9cfc003ae0 0x7f9cfc003ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:21.404 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.398+0000 7f9d11ffb700 10 monclient(hunting): picked mon.b con 0x7f9cfc003ae0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:06:21.404 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.398+0000 7f9d11ffb700 1 --2- 172.21.15.59:0/266507798 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9cfc004810 0x7f9cfc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:21.404 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.398+0000 7f9d11ffb700 10 monclient(hunting): picked mon.a con 0x7f9cfc004810 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:06:21.404 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.398+0000 7f9d11ffb700 1 --2- 172.21.15.59:0/266507798 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9cfc0052e0 0x7f9cfc00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:21.404 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.398+0000 7f9d11ffb700 10 monclient(hunting): picked mon.c con 0x7f9cfc0052e0 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:21.404 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.398+0000 7f9d11ffb700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:21.404 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.398+0000 7f9d11ffb700 1 -- 172.21.15.59:0/266507798 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d140b0230 con 0x7f9cfc004810 2023-09-30T02:06:21.404 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.398+0000 7f9d11ffb700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:21.404 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.398+0000 7f9d11ffb700 1 -- 172.21.15.59:0/266507798 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d1406b0a0 con 0x7f9cfc003ae0 2023-09-30T02:06:21.404 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.398+0000 7f9d11ffb700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:21.404 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.398+0000 7f9d11ffb700 1 -- 172.21.15.59:0/266507798 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cfc00c260 con 0x7f9cfc0052e0 2023-09-30T02:06:21.405 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.398+0000 7f9d11ffb700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:21.405 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.398+0000 7f9d11ffb700 10 monclient(hunting): _finish_auth 0 2023-09-30T02:06:21.405 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.400+0000 7fbe5ed9d700 10 monclient(hunting): handle_auth_done global_id 4303 payload 274 2023-09-30T02:06:21.405 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.400+0000 7fbe5ed9d700 1 -- 172.21.15.59:0/2929340633 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fbe60087810 msgr2=0x7fbe6006bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:06:21.405 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.400+0000 7fbe5ed9d700 1 --2- 172.21.15.59:0/2929340633 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fbe60087810 0x7fbe6006bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:21.405 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.400+0000 7fbe5ed9d700 1 -- 172.21.15.59:0/2929340633 >> v2:172.21.15.59:3300/0 conn(0x7fbe60069b10 msgr2=0x7fbe6006b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:06:21.405 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.400+0000 7fbe5ed9d700 1 --2- 172.21.15.59:0/2929340633 >> v2:172.21.15.59:3300/0 conn(0x7fbe60069b10 0x7fbe6006b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:21.405 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.400+0000 7fbe5ed9d700 10 monclient: _finish_hunting 0 2023-09-30T02:06:21.406 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.400+0000 7fbe5ed9d700 1 monclient: found mon.noname-c 2023-09-30T02:06:21.406 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.400+0000 7fbe5ed9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:21.406 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.400+0000 7fbe5ed9d700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3300/0 2023-09-30T02:06:21.406 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.400+0000 7fbe5ed9d700 1 -- 172.21.15.59:0/2929340633 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe6006d030 con 0x7fbe600ab2d0 2023-09-30T02:06:21.406 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.401+0000 7fa04e664700 10 monclient: get_monmap_and_config 2023-09-30T02:06:21.406 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.401+0000 7fa04e664700 10 monclient: build_initial_monmap 2023-09-30T02:06:21.406 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.401+0000 7fa04e664700 10 monclient: monmap: 2023-09-30T02:06:21.406 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:21.406 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:21.407 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:06:21.402358+0000 2023-09-30T02:06:21.407 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:06:21.402358+0000 2023-09-30T02:06:21.407 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:21.407 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:21.407 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:06:21.407 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:06:21.407 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:06:21.407 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:06:21.407 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:06:21.407 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:21.408 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.402+0000 7fbe5ed9d700 1 --2- 172.21.15.59:0/2929340633 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbe600ab2d0 0x7fbe600ab770 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fbe6006c350 tx=0x7fbe500083f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=26067d034fd9562b server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:21.408 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.403+0000 7ff346970700 1 --2- 172.21.15.59:0/4021846443 >> v2:172.21.15.59:3300/0 conn(0x7ff3400ab2d0 0x7ff3400ab710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:21.408 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.403+0000 7fa04e664700 1 Processor -- start 2023-09-30T02:06:21.408 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.403+0000 7fa04e664700 1 -- start start 2023-09-30T02:06:21.408 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.403+0000 7fa04e664700 10 monclient: init 2023-09-30T02:06:21.409 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.403+0000 7ff346970700 10 monclient(hunting): get_auth_request con 0x7ff3400ab2d0 auth_method 0 2023-09-30T02:06:21.409 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.403+0000 7ff346970700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:21.409 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.403+0000 7ff346970700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:21.409 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.403+0000 7ff346970700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:21.409 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.404+0000 7fa04e664700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:21.410 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.404+0000 7fa04e664700 10 monclient: _add_conns ranks=[4,2,1,0,3] 2023-09-30T02:06:21.410 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.404+0000 7fa04e664700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fa0480b12c0 0x7fa0480b1690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:21.410 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.404+0000 7fa04e664700 10 monclient(hunting): picked mon.noname-c con 0x7fa0480b12c0 addr v2:172.21.15.110:3300/0 2023-09-30T02:06:21.410 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.404+0000 7fa04e664700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa04813e3b0 0x7fa04813e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:21.410 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.404+0000 7fa04e664700 10 monclient(hunting): picked mon.noname-b con 0x7fa04813e3b0 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:21.410 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.404+0000 7fa04e664700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa04813ed20 addr v1:172.21.15.110:6789/0 2023-09-30T02:06:21.410 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.404+0000 7fa04e664700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0480a5920 con 0x7fa04813ed20 2023-09-30T02:06:21.410 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.404+0000 7fa04e664700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:21.410 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.404+0000 7fa04e664700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0480a5c00 con 0x7fa0480b12c0 2023-09-30T02:06:21.410 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.404+0000 7fa04e664700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:21.410 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.404+0000 7fa04e664700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0480b1bd0 con 0x7fa04813e3b0 2023-09-30T02:06:21.411 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.404+0000 7fa04e664700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:21.411 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.404+0000 7fa04e664700 10 monclient(hunting): authenticate will time out at 1826.706535s 2023-09-30T02:06:21.411 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.407+0000 7ff34596e700 1 --2- 172.21.15.59:0/4021846443 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff34006ba90 0x7ff34006e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:21.411 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.405+0000 7f9d127fc700 1 --2- 172.21.15.59:0/266507798 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9cfc004810 0x7f9cfc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:21.411 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.406+0000 7f9d12ffd700 1 --2- 172.21.15.59:0/266507798 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9cfc003ae0 0x7f9cfc003ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:21.411 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.406+0000 7f9d12ffd700 10 monclient(hunting): get_auth_request con 0x7f9cfc003ae0 auth_method 0 2023-09-30T02:06:21.411 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.406+0000 7f9d12ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:21.412 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.406+0000 7f9d12ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:21.412 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.406+0000 7f9d12ffd700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T02:06:21.412 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.408+0000 7ff34516d700 1 -- 172.21.15.59:0/4021846443 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2459598967 0 0) 0x7ff34006a260 con 0x7ff340069e90 2023-09-30T02:06:21.412 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.408+0000 7ff34516d700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:21.412 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.408+0000 7ff34516d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:21.412 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.408+0000 7ff34516d700 10 monclient(hunting): my global_id is 4306 2023-09-30T02:06:21.412 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.408+0000 7ff34516d700 1 -- 172.21.15.59:0/4021846443 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff32c003570 con 0x7ff340069e90 2023-09-30T02:06:21.413 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.408+0000 7fbe5d59a700 1 -- 172.21.15.59:0/2929340633 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe50010070 con 0x7fbe600ab2d0 2023-09-30T02:06:21.413 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.409+0000 7fbe5d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:21.413 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.409+0000 7fbe5d59a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:06:21.413 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.409+0000 7fbe5d59a700 10 monclient: dump: 2023-09-30T02:06:21.413 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:21.413 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:21.413 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:21.413 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:21.413 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:21.413 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:21.414 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:21.414 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:21.414 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:21.414 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:21.414 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.409+0000 7fbe5d59a700 10 monclient: _finish_auth 0 2023-09-30T02:06:21.414 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.409+0000 7fbe5d59a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:21.415 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.409+0000 7f9d127fc700 10 monclient(hunting): get_auth_request con 0x7f9cfc004810 auth_method 0 2023-09-30T02:06:21.415 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.409+0000 7f9d127fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:21.415 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.409+0000 7f9d127fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:21.415 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.409+0000 7f9d127fc700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T02:06:21.415 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.409+0000 7fbe64a5d700 5 monclient: authenticate success, global_id 4303 2023-09-30T02:06:21.415 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.409+0000 7fbe64a5d700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:06:21.416 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.410+0000 7f9d12ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:21.416 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.410+0000 7f9d12ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:21.416 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.410+0000 7f9d12ffd700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T02:06:21.416 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.410+0000 7f32f0ee9700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f32ec0b13f0 0x7f32ec06bc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.59:54806/0 (socket says 172.21.15.59:54806) 2023-09-30T02:06:21.416 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.410+0000 7f32f0ee9700 1 -- 172.21.15.59:0/818876457 learned_addr learned my addr 172.21.15.59:0/818876457 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:06:21.416 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.409+0000 7ff34596e700 10 monclient(hunting): get_auth_request con 0x7ff34006ba90 auth_method 0 2023-09-30T02:06:21.416 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.409+0000 7ff34596e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:21.417 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.409+0000 7ff34596e700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:21.417 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.409+0000 7ff34596e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:21.417 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.410+0000 7fbe5d59a700 1 -- 172.21.15.59:0/2929340633 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe5000da40 con 0x7fbe600ab2d0 2023-09-30T02:06:21.417 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.410+0000 7fbe5d59a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:21.417 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.410+0000 7fbe5d59a700 1 -- 172.21.15.59:0/2929340633 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe5000de00 con 0x7fbe600ab2d0 2023-09-30T02:06:21.417 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.410+0000 7fbe5d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:21.417 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.410+0000 7fbe5d59a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:06:21.417 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.410+0000 7fbe5d59a700 10 monclient: dump: 2023-09-30T02:06:21.418 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:21.418 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:21.418 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:21.418 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:21.418 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:21.418 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:21.418 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:21.418 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:21.418 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:21.418 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:21.419 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.412+0000 7f9d137fe700 1 --2- 172.21.15.59:0/266507798 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9cfc0052e0 0x7f9cfc00ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:21.419 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.412+0000 7fa04cc01700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fa04813ed20 0x7fa048141790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.59:54818/0 (socket says 172.21.15.59:54818) 2023-09-30T02:06:21.419 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.412+0000 7fa04cc01700 1 -- 172.21.15.59:0/441925120 learned_addr learned my addr 172.21.15.59:0/441925120 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:06:21.419 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.410+0000 7fbe64a5d700 10 monclient: get_monmap_and_config success 2023-09-30T02:06:21.420 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.411+0000 7fbe64a5d700 4 set_mon_vals no callback set 2023-09-30T02:06:21.420 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.411+0000 7fbe64a5d700 10 monclient: shutdown 2023-09-30T02:06:21.420 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.412+0000 7fbe64a5d700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:21.420 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.412+0000 7fbe64a5d700 1 -- 172.21.15.59:0/2929340633 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbe600ab2d0 msgr2=0x7fbe600ab770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:21.420 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.412+0000 7fbe64a5d700 1 --2- 172.21.15.59:0/2929340633 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbe600ab2d0 0x7fbe600ab770 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fbe6006c350 tx=0x7fbe500083f0 comp rx=0 tx=0).stop 2023-09-30T02:06:21.420 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.412+0000 7fbe64a5d700 1 -- 172.21.15.59:0/2929340633 shutdown_connections 2023-09-30T02:06:21.420 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.412+0000 7fbe64a5d700 1 --2- 172.21.15.59:0/2929340633 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbe600ab2d0 0x7fbe600ab770 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:21.420 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.412+0000 7fbe64a5d700 1 --2- 172.21.15.59:0/2929340633 >> v2:172.21.15.59:3300/0 conn(0x7fbe60069b10 0x7fbe6006b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:21.420 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.412+0000 7fbe64a5d700 1 --2- 172.21.15.59:0/2929340633 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fbe60087810 0x7fbe6006bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:21.420 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.412+0000 7fbe64a5d700 1 -- 172.21.15.59:0/2929340633 >> 172.21.15.59:0/2929340633 conn(0x7fbe600b08e0 msgr2=0x7fbe600b0ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:21.420 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.412+0000 7fbe64a5d700 1 -- 172.21.15.59:0/2929340633 shutdown_connections 2023-09-30T02:06:21.421 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.413+0000 7ff346970700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:21.421 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.413+0000 7ff346970700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:21.421 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.413+0000 7ff346970700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:21.421 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.413+0000 7ff34596e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:21.421 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.413+0000 7ff34596e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:21.421 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.413+0000 7ff34596e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:21.421 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.412+0000 7fbe64a5d700 1 -- 172.21.15.59:0/2929340633 wait complete. 2023-09-30T02:06:21.422 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.413+0000 7fbe64a5d700 10 monclient: build_initial_monmap 2023-09-30T02:06:21.422 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.413+0000 7fbe64a5d700 10 monclient: monmap: 2023-09-30T02:06:21.422 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:21.422 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:06:21.422 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:06:21.422 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:06:21.422 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:21.422 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:06:21.422 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:06:21.422 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:06:21.423 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:06:21.423 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:21.423 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.414+0000 7fbe64a5d700 1 librados: starting msgr at 2023-09-30T02:06:21.423 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.414+0000 7fbe64a5d700 1 librados: starting objecter 2023-09-30T02:06:21.423 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.414+0000 7fbe64a5d700 1 Processor -- start 2023-09-30T02:06:21.423 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.413+0000 7f2911ffb700 1 -- 172.21.15.59:0/2417451478 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f290c00e070 con 0x7f291413e140 2023-09-30T02:06:21.423 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.413+0000 7f2911ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:21.423 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.413+0000 7f2911ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:06:21.423 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.413+0000 7f2911ffb700 10 monclient: dump: 2023-09-30T02:06:21.424 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:21.424 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:21.424 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:21.424 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:21.424 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:21.424 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:21.424 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:21.424 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:21.424 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:21.424 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:21.424 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.413+0000 7f2911ffb700 10 monclient: _finish_auth 0 2023-09-30T02:06:21.424 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.413+0000 7f2911ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:21.425 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.413+0000 7f2911ffb700 1 -- 172.21.15.59:0/2417451478 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f290c00b990 con 0x7f291413e140 2023-09-30T02:06:21.425 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.413+0000 7f2911ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:21.425 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.413+0000 7f2911ffb700 1 -- 172.21.15.59:0/2417451478 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f290c0092e0 con 0x7f291413e140 2023-09-30T02:06:21.425 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.413+0000 7f2911ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:21.425 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.413+0000 7f2911ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:06:21.425 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.413+0000 7f2911ffb700 10 monclient: dump: 2023-09-30T02:06:21.425 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:21.425 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:21.425 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:21.425 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:21.425 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:21.425 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:21.425 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:21.426 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:21.426 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:21.426 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:21.426 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.414+0000 7fbe64a5d700 1 -- start start 2023-09-30T02:06:21.426 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.414+0000 7fbe64a5d700 1 librados: setting wanted keys 2023-09-30T02:06:21.426 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.414+0000 7fbe64a5d700 1 librados: calling monclient init 2023-09-30T02:06:21.426 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.414+0000 7fbe64a5d700 10 monclient: init 2023-09-30T02:06:21.426 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.415+0000 7fbe64a5d700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:21.426 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.415+0000 7fbe64a5d700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T02:06:21.427 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.415+0000 7fbe64a5d700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fbe60069b10 0x7fbe601513b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:21.427 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.415+0000 7fbe64a5d700 10 monclient(hunting): picked mon.noname-c con 0x7fbe60069b10 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:21.427 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.415+0000 7fbe64a5d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbe60087810 0x7fbe600a90a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:21.427 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.415+0000 7fbe64a5d700 10 monclient(hunting): picked mon.noname-b con 0x7fbe60087810 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:06:21.427 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.415+0000 7fbe64a5d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbe600ab2d0 0x7fbe600a95e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:21.427 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.415+0000 7fbe64a5d700 10 monclient(hunting): picked mon.noname-a con 0x7fbe600ab2d0 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:06:21.427 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.415+0000 7fbe64a5d700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:21.427 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.415+0000 7fbe64a5d700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe600a8c10 con 0x7fbe600ab2d0 2023-09-30T02:06:21.427 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.415+0000 7fbe64a5d700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:21.427 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.415+0000 7fbe64a5d700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe60150b80 con 0x7fbe60087810 2023-09-30T02:06:21.427 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.415+0000 7fbe64a5d700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:21.427 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.415+0000 7fbe64a5d700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe60151980 con 0x7fbe60069b10 2023-09-30T02:06:21.427 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.415+0000 7fbe64a5d700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:21.428 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.415+0000 7fbe64a5d700 10 monclient(hunting): authenticate will time out at 1826.717546s 2023-09-30T02:06:21.428 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.413+0000 7f291972e700 5 monclient: authenticate success, global_id 4262 2023-09-30T02:06:21.428 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.413+0000 7f291972e700 10 monclient: get_monmap_and_config success 2023-09-30T02:06:21.428 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.413+0000 7f291972e700 4 set_mon_vals no callback set 2023-09-30T02:06:21.428 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.414+0000 7f291972e700 10 monclient: shutdown 2023-09-30T02:06:21.428 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.414+0000 7f291972e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:21.429 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.414+0000 7f291972e700 1 -- 172.21.15.59:0/2417451478 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f291413e140 msgr2=0x7f291413a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:21.429 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.414+0000 7f291972e700 1 --2- 172.21.15.59:0/2417451478 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f291413e140 0x7f291413a750 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f290c009f20 tx=0x7f290c009840 comp rx=0 tx=0).stop 2023-09-30T02:06:21.429 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.414+0000 7f291972e700 1 -- 172.21.15.59:0/2417451478 shutdown_connections 2023-09-30T02:06:21.429 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.414+0000 7f291972e700 1 --2- 172.21.15.59:0/2417451478 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f291413e140 0x7f291413a750 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:21.429 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.414+0000 7f291972e700 1 --2- 172.21.15.59:0/2417451478 >> v2:172.21.15.110:3300/0 conn(0x7f2914069fd0 0x7f291413dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:21.429 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.414+0000 7f291972e700 1 -- 172.21.15.59:0/2417451478 >> 172.21.15.59:0/2417451478 conn(0x7f29140b08e0 msgr2=0x7f29140b0ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:21.429 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.414+0000 7f291972e700 1 -- 172.21.15.59:0/2417451478 shutdown_connections 2023-09-30T02:06:21.429 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.414+0000 7f291972e700 1 -- 172.21.15.59:0/2417451478 wait complete. 2023-09-30T02:06:21.429 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.415+0000 7f291972e700 10 monclient: build_initial_monmap 2023-09-30T02:06:21.429 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.415+0000 7f291972e700 10 monclient: monmap: 2023-09-30T02:06:21.429 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:21.429 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:06:21.429 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:06:21.430 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:06:21.430 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:21.430 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:06:21.430 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:06:21.430 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:06:21.430 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:06:21.430 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:21.430 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.415+0000 7f291972e700 1 librados: starting msgr at 2023-09-30T02:06:21.430 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.415+0000 7f291972e700 1 librados: starting objecter 2023-09-30T02:06:21.430 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.416+0000 7f291972e700 1 Processor -- start 2023-09-30T02:06:21.430 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.416+0000 7f291972e700 1 -- start start 2023-09-30T02:06:21.431 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.416+0000 7f291972e700 1 librados: setting wanted keys 2023-09-30T02:06:21.431 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.416+0000 7f291972e700 1 librados: calling monclient init 2023-09-30T02:06:21.431 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.416+0000 7f291972e700 10 monclient: init 2023-09-30T02:06:21.431 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.416+0000 7f291972e700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:21.431 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.416+0000 7f291972e700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T02:06:21.431 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.416+0000 7f291972e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2914069fd0 0x7f29140a7860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:21.431 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.416+0000 7f291972e700 10 monclient(hunting): picked mon.noname-a con 0x7f2914069fd0 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:06:21.431 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.416+0000 7f291972e700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f29140b13f0 0x7f29140a54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:21.431 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.416+0000 7f291972e700 10 monclient(hunting): picked mon.noname-c con 0x7f29140b13f0 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:21.431 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.416+0000 7f291972e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f291413e140 0x7f29140aa090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:21.431 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.416+0000 7f291972e700 10 monclient(hunting): picked mon.noname-b con 0x7f291413e140 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:06:21.432 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.416+0000 7f291972e700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:21.432 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.416+0000 7f291972e700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29140a7fd0 con 0x7f2914069fd0 2023-09-30T02:06:21.432 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.416+0000 7f291972e700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:21.432 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.416+0000 7f291972e700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29140aa5d0 con 0x7f291413e140 2023-09-30T02:06:21.432 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.416+0000 7f291972e700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:21.432 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.416+0000 7f291972e700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29140a9c70 con 0x7f29140b13f0 2023-09-30T02:06:21.432 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.416+0000 7f291972e700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:21.432 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.416+0000 7f291972e700 10 monclient(hunting): authenticate will time out at 1826.718772s 2023-09-30T02:06:21.433 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.416+0000 7ff346970700 10 monclient(hunting): handle_auth_done global_id 4281 payload 274 2023-09-30T02:06:21.433 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.416+0000 7ff346970700 1 -- 172.21.15.59:0/4021846443 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff34006ba90 msgr2=0x7ff34006e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:06:21.433 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.416+0000 7ff346970700 1 --2- 172.21.15.59:0/4021846443 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff34006ba90 0x7ff34006e0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:21.433 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.416+0000 7ff346970700 1 -- 172.21.15.59:0/4021846443 >> v1:172.21.15.110:6789/0 conn(0x7ff340069e90 legacy=0x7ff34006d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:21.433 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.417+0000 7ff346970700 10 monclient: _finish_hunting 0 2023-09-30T02:06:21.433 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.417+0000 7ff346970700 1 monclient: found mon.noname-a 2023-09-30T02:06:21.433 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.417+0000 7ff346970700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:21.433 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.417+0000 7ff346970700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.59:3300/0 2023-09-30T02:06:21.433 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.417+0000 7ff346970700 1 -- 172.21.15.59:0/4021846443 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff34006e610 con 0x7ff3400ab2d0 2023-09-30T02:06:21.434 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.414+0000 7f9d12ffd700 10 monclient(hunting): handle_auth_done global_id 4297 payload 293 2023-09-30T02:06:21.434 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.414+0000 7f9d12ffd700 1 -- 172.21.15.59:0/266507798 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9cfc0052e0 msgr2=0x7f9cfc00ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:06:21.434 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.414+0000 7f9d12ffd700 1 --2- 172.21.15.59:0/266507798 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9cfc0052e0 0x7f9cfc00ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:21.434 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.414+0000 7f9d12ffd700 1 -- 172.21.15.59:0/266507798 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9cfc004810 msgr2=0x7f9cfc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:06:21.434 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.414+0000 7f9d12ffd700 1 --2- 172.21.15.59:0/266507798 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9cfc004810 0x7f9cfc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:21.434 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.414+0000 7f9d12ffd700 10 monclient: _finish_hunting 0 2023-09-30T02:06:21.435 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.414+0000 7f9d12ffd700 1 monclient: found mon.b 2023-09-30T02:06:21.435 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.414+0000 7f9d12ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:21.435 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.414+0000 7f9d12ffd700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:21.435 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.414+0000 7f9d12ffd700 1 -- 172.21.15.59:0/266507798 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d14146650 con 0x7f9cfc003ae0 2023-09-30T02:06:21.435 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.414+0000 7f9d12ffd700 10 monclient: _finish_auth 0 2023-09-30T02:06:21.435 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.414+0000 7f9d12ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:21.435 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.417+0000 7f9d199df700 5 monclient: authenticate success, global_id 4297 2023-09-30T02:06:21.435 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.417+0000 7f9d199df700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:06:21.435 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.417+0000 7f32ebfff700 1 --2- 172.21.15.59:0/818876457 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f32ec069fd0 0x7f32ec13dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:21.436 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.413+0000 7f0f9055c700 10 monclient: get_monmap_and_config 2023-09-30T02:06:21.436 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.413+0000 7f0f9055c700 10 monclient: build_initial_monmap 2023-09-30T02:06:21.436 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.413+0000 7f0f9055c700 10 monclient: monmap: 2023-09-30T02:06:21.436 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:21.436 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:21.436 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:06:21.413624+0000 2023-09-30T02:06:21.436 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:06:21.413624+0000 2023-09-30T02:06:21.437 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:21.437 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:21.437 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:06:21.437 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:06:21.437 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:06:21.437 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:06:21.437 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:06:21.437 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:21.437 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.418+0000 7f0f9055c700 1 Processor -- start 2023-09-30T02:06:21.437 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.418+0000 7f0f9055c700 1 -- start start 2023-09-30T02:06:21.437 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.418+0000 7f0f9055c700 10 monclient: init 2023-09-30T02:06:21.437 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.419+0000 7f0f9055c700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:21.437 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.419+0000 7f0f9055c700 10 monclient: _add_conns ranks=[1,4,3,0,2] 2023-09-30T02:06:21.438 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.419+0000 7f0f9055c700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0f880ab4e0 addr v1:172.21.15.110:6789/0 2023-09-30T02:06:21.438 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.419+0000 7f0f9055c700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f0f880ab8b0 0x7f0f881419d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:21.438 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.419+0000 7f0f9055c700 10 monclient(hunting): picked mon.noname-c con 0x7f0f880ab8b0 addr v2:172.21.15.110:3300/0 2023-09-30T02:06:21.438 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.419+0000 7f0f9055c700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f0f8806a170 0x7f0f88141f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:21.438 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.419+0000 7f0f9055c700 10 monclient(hunting): picked mon.noname-a con 0x7f0f8806a170 addr v2:172.21.15.59:3300/0 2023-09-30T02:06:21.438 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.419+0000 7f0f9055c700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f88142450 con 0x7f0f880ab4e0 2023-09-30T02:06:21.438 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.419+0000 7f0f9055c700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:21.438 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.419+0000 7f0f9055c700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f8806b8f0 con 0x7f0f8806a170 2023-09-30T02:06:21.438 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.419+0000 7f0f9055c700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:21.438 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.419+0000 7f0f9055c700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f880ac760 con 0x7f0f880ab8b0 2023-09-30T02:06:21.438 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.419+0000 7f0f9055c700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:21.438 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.419+0000 7f0f9055c700 10 monclient(hunting): authenticate will time out at 1826.721497s 2023-09-30T02:06:21.439 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.419+0000 7f32ebfff700 10 monclient(hunting): get_auth_request con 0x7f32ec069fd0 auth_method 0 2023-09-30T02:06:21.439 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.419+0000 7f32ebfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:21.439 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.419+0000 7f32ebfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:21.439 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.419+0000 7f32ebfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:21.439 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.420+0000 7fa03f7fe700 1 --2- 172.21.15.59:0/441925120 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa04813e3b0 0x7fa04813e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:21.440 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.420+0000 7fa03f7fe700 10 monclient(hunting): get_auth_request con 0x7fa04813e3b0 auth_method 0 2023-09-30T02:06:21.440 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.420+0000 7fa03f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:21.440 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.420+0000 7fa03f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:21.440 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.420+0000 7fa03f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:21.440 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.420+0000 7f32f16ea700 1 --2- 172.21.15.59:0/818876457 >> v2:172.21.15.59:3300/0 conn(0x7f32ec13e140 0x7f32ec13a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:21.440 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.420+0000 7f32f16ea700 10 monclient(hunting): get_auth_request con 0x7f32ec13e140 auth_method 0 2023-09-30T02:06:21.440 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.420+0000 7f32f16ea700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:21.440 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.420+0000 7f32f16ea700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:21.441 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.420+0000 7f32f16ea700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:21.441 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.420+0000 7fa03f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:21.441 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.420+0000 7fa03f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:21.441 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.420+0000 7fa03f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:21.441 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.421+0000 7f32f16ea700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:21.442 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.421+0000 7f32f16ea700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:21.442 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.421+0000 7f32f16ea700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:21.442 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.421+0000 7f9d12ffd700 1 --2- 172.21.15.59:0/266507798 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9cfc003ae0 0x7f9cfc003ef0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9d0800c320 tx=0x7f9d0800c630 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ea88d5bed0f20746 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:21.443 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.422+0000 7fa03f7fe700 10 monclient(hunting): handle_auth_done global_id 4274 payload 274 2023-09-30T02:06:21.443 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.422+0000 7fa03f7fe700 1 -- 172.21.15.59:0/441925120 >> v2:172.21.15.110:3300/0 conn(0x7fa0480b12c0 msgr2=0x7fa0480b1690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:06:21.443 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.422+0000 7fa03f7fe700 1 --2- 172.21.15.59:0/441925120 >> v2:172.21.15.110:3300/0 conn(0x7fa0480b12c0 0x7fa0480b1690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:21.443 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.422+0000 7fa03f7fe700 1 -- 172.21.15.59:0/441925120 >> v1:172.21.15.110:6789/0 conn(0x7fa04813ed20 legacy=0x7fa048141790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:21.443 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.422+0000 7fa03f7fe700 10 monclient: _finish_hunting 0 2023-09-30T02:06:21.443 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.422+0000 7fa03f7fe700 1 monclient: found mon.noname-b 2023-09-30T02:06:21.443 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.422+0000 7fa03f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:21.443 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.422+0000 7fa03f7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.59:3301/0 2023-09-30T02:06:21.443 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.422+0000 7fa03f7fe700 1 -- 172.21.15.59:0/441925120 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0480a7210 con 0x7fa04813e3b0 2023-09-30T02:06:21.444 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.423+0000 7f32f16ea700 10 monclient(hunting): handle_auth_done global_id 4284 payload 274 2023-09-30T02:06:21.444 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.423+0000 7f32f16ea700 1 -- 172.21.15.59:0/818876457 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f32ec069fd0 msgr2=0x7f32ec13dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:06:21.444 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.423+0000 7f32f16ea700 1 --2- 172.21.15.59:0/818876457 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f32ec069fd0 0x7f32ec13dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:21.444 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.423+0000 7f32f16ea700 1 -- 172.21.15.59:0/818876457 >> v1:172.21.15.110:6789/0 conn(0x7f32ec0b13f0 legacy=0x7f32ec06bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:21.444 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.423+0000 7f32f16ea700 10 monclient: _finish_hunting 0 2023-09-30T02:06:21.444 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.423+0000 7f32f16ea700 1 monclient: found mon.noname-a 2023-09-30T02:06:21.444 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.423+0000 7f32f16ea700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:21.445 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.423+0000 7f32f16ea700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.59:3300/0 2023-09-30T02:06:21.445 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.423+0000 7f32f16ea700 1 -- 172.21.15.59:0/818876457 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32ec06d480 con 0x7f32ec13e140 2023-09-30T02:06:21.445 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.427+0000 7ff346970700 1 --2- 172.21.15.59:0/4021846443 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3400ab2d0 0x7ff3400ab710 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff338009e60 tx=0x7ff338009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14c8c0e03cbaf95e server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:21.445 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.429+0000 7fa03f7fe700 1 --2- 172.21.15.59:0/441925120 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa04813e3b0 0x7fa04813e7e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa0300098e0 tx=0x7fa0300091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e2bc9048085a4786 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:21.445 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.433+0000 7f32f16ea700 1 --2- 172.21.15.59:0/818876457 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32ec13e140 0x7f32ec13a750 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f32e00098e0 tx=0x7f32e00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e21fe6f7b08f811 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:21.446 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.434+0000 7f29137fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f291413e140 0x7f29140aa090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:21.446 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.434+0000 7f9d11ffb700 1 -- 172.21.15.59:0/266507798 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d0800cc80 con 0x7f9cfc003ae0 2023-09-30T02:06:21.446 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.434+0000 7f9d11ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:21.446 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.434+0000 7f9d11ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:06:21.447 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.434+0000 7f9d11ffb700 10 monclient: dump: 2023-09-30T02:06:21.447 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:21.447 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:21.447 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:21.447 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:21.447 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:21.447 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:21.447 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:21.447 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:21.447 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:21.447 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:21.447 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.434+0000 7f9d11ffb700 1 -- 172.21.15.59:0/266507798 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d0800cde0 con 0x7f9cfc003ae0 2023-09-30T02:06:21.448 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.434+0000 7f9d199df700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:06:21.448 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.434+0000 7f9d11ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:21.448 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.434+0000 7f9d11ffb700 1 -- 172.21.15.59:0/266507798 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d08005b70 con 0x7f9cfc003ae0 2023-09-30T02:06:21.448 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.434+0000 7f9d11ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:21.448 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.434+0000 7f9d11ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:06:21.448 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.434+0000 7f9d11ffb700 10 monclient: dump: 2023-09-30T02:06:21.448 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:21.448 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:21.448 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:21.448 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:21.448 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:21.448 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:21.449 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:21.449 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:21.449 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:21.449 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:21.449 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.435+0000 7f29137fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f291413e140 0x7f29140aa090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:38894/0 (socket says 172.21.15.59:38894) 2023-09-30T02:06:21.449 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.435+0000 7f29137fe700 1 -- 172.21.15.59:0/1269097232 learned_addr learned my addr 172.21.15.59:0/1269097232 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:06:21.450 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.435+0000 7f29137fe700 10 monclient(hunting): get_auth_request con 0x7f291413e140 auth_method 0 2023-09-30T02:06:21.450 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.435+0000 7f29137fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:21.450 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.435+0000 7f29137fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:21.450 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.435+0000 7f29137fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:21.450 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.434+0000 7f9d199df700 10 monclient: get_monmap_and_config success 2023-09-30T02:06:21.450 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.434+0000 7f9d199df700 4 set_mon_vals no callback set 2023-09-30T02:06:21.450 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.435+0000 7f9d199df700 10 monclient: shutdown 2023-09-30T02:06:21.451 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.435+0000 7f9d199df700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:21.451 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.435+0000 7f9d199df700 1 -- 172.21.15.59:0/266507798 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9cfc003ae0 msgr2=0x7f9cfc003ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:21.451 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.435+0000 7f9d199df700 1 --2- 172.21.15.59:0/266507798 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9cfc003ae0 0x7f9cfc003ef0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9d0800c320 tx=0x7f9d0800c630 comp rx=0 tx=0).stop 2023-09-30T02:06:21.451 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.435+0000 7f9d137fe700 1 -- 172.21.15.59:0/266507798 reap_dead start 2023-09-30T02:06:21.451 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.435+0000 7f9d199df700 1 -- 172.21.15.59:0/266507798 shutdown_connections 2023-09-30T02:06:21.451 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.435+0000 7f9d199df700 1 -- 172.21.15.59:0/266507798 >> 172.21.15.59:0/266507798 conn(0x7f9d140631f0 msgr2=0x7f9d140635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:21.451 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.435+0000 7f0f8daf7700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f0f880ab8b0 0x7f0f881419d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:21.451 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.435+0000 7f0f8daf7700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f0f880ab8b0 0x7f0f881419d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:38904/0 (socket says 172.21.15.59:38904) 2023-09-30T02:06:21.452 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.435+0000 7f0f8daf7700 1 -- 172.21.15.59:0/3080185083 learned_addr learned my addr 172.21.15.59:0/3080185083 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:06:21.452 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.435+0000 7f0f8daf7700 10 monclient(hunting): get_auth_request con 0x7f0f880ab8b0 auth_method 0 2023-09-30T02:06:21.452 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.435+0000 7f0f8daf7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:21.452 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.435+0000 7f0f8daf7700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:21.452 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.435+0000 7f0f8daf7700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:21.452 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.436+0000 7f29127fc700 1 --2- 172.21.15.59:0/1269097232 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f29140b13f0 0x7f29140a54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:21.453 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.437+0000 7f9d199df700 1 -- 172.21.15.59:0/266507798 shutdown_connections 2023-09-30T02:06:21.453 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.437+0000 7f9d199df700 1 -- 172.21.15.59:0/266507798 wait complete. 2023-09-30T02:06:21.453 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.437+0000 7f0f8daf7700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:21.453 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.437+0000 7f0f8daf7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:21.453 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.437+0000 7f0f8daf7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:21.454 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.438+0000 7f9d199df700 10 monclient: build_initial_monmap 2023-09-30T02:06:21.454 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.438+0000 7f9d199df700 10 monclient: monmap: 2023-09-30T02:06:21.454 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:21.454 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:06:21.454 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:06:21.454 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:06:21.454 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:21.454 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:06:21.454 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:06:21.454 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:06:21.455 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:06:21.455 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:21.455 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.438+0000 7f9d199df700 1 librados: starting msgr at 2023-09-30T02:06:21.455 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.438+0000 7f9d199df700 1 librados: starting objecter 2023-09-30T02:06:21.455 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.438+0000 7f9d199df700 1 Processor -- start 2023-09-30T02:06:21.455 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.438+0000 7f9d199df700 1 -- start start 2023-09-30T02:06:21.455 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.438+0000 7f9d199df700 1 librados: setting wanted keys 2023-09-30T02:06:21.455 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.438+0000 7f9d199df700 1 librados: calling monclient init 2023-09-30T02:06:21.455 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.438+0000 7f9d199df700 10 monclient: init 2023-09-30T02:06:21.455 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.439+0000 7f9d199df700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:21.455 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.439+0000 7f9d199df700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T02:06:21.455 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.439+0000 7f9d199df700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9cfc003ae0 0x7f9d1411c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:21.456 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.439+0000 7f9d199df700 10 monclient(hunting): picked mon.noname-a con 0x7f9cfc003ae0 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:06:21.456 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.439+0000 7f9d199df700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9d14143010 0x7f9d14152490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:21.456 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.439+0000 7f9d199df700 10 monclient(hunting): picked mon.noname-b con 0x7f9d14143010 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:06:21.456 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.439+0000 7f9d199df700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9d141529d0 0x7f9d1413ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:21.456 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.439+0000 7f9d199df700 10 monclient(hunting): picked mon.noname-c con 0x7f9d141529d0 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:21.456 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.439+0000 7f9d199df700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:21.456 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.439+0000 7f9d199df700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d1411d190 con 0x7f9cfc003ae0 2023-09-30T02:06:21.456 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.439+0000 7f9d199df700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:21.456 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.439+0000 7f9d199df700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d1411e160 con 0x7f9d14143010 2023-09-30T02:06:21.456 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.439+0000 7f9d199df700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:21.456 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.439+0000 7f9d199df700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d14151f70 con 0x7f9d141529d0 2023-09-30T02:06:21.456 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.439+0000 7f9d199df700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:21.457 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.439+0000 7f9d199df700 10 monclient(hunting): authenticate will time out at 1826.741318s 2023-09-30T02:06:21.457 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.439+0000 7f32eb7fe700 1 -- 172.21.15.59:0/818876457 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e0009e80 con 0x7f32ec13e140 2023-09-30T02:06:21.457 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.439+0000 7f32eb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:21.458 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.439+0000 7f32eb7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T02:06:21.458 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.439+0000 7f32eb7fe700 10 monclient: dump: 2023-09-30T02:06:21.458 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:21.458 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:21.458 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:21.458 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:21.458 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:21.458 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:21.458 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:21.458 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:21.458 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:21.458 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:21.459 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.439+0000 7f32eb7fe700 10 monclient: _finish_auth 0 2023-09-30T02:06:21.459 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.439+0000 7f32eb7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:21.459 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.439+0000 7f32eb7fe700 1 -- 172.21.15.59:0/818876457 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32e000b860 con 0x7f32ec13e140 2023-09-30T02:06:21.459 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.439+0000 7f32f314d700 5 monclient: authenticate success, global_id 4284 2023-09-30T02:06:21.459 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.439+0000 7f32f314d700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:06:21.459 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.439+0000 7f32eb7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:21.459 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.439+0000 7f32f314d700 10 monclient: get_monmap_and_config success 2023-09-30T02:06:21.459 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.439+0000 7f32eb7fe700 1 -- 172.21.15.59:0/818876457 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e000bbe0 con 0x7f32ec13e140 2023-09-30T02:06:21.459 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.439+0000 7f32f314d700 4 set_mon_vals no callback set 2023-09-30T02:06:21.459 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.439+0000 7f32eb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:21.459 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.439+0000 7f32eb7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T02:06:21.459 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.439+0000 7f32eb7fe700 10 monclient: dump: 2023-09-30T02:06:21.460 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:21.460 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:21.460 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:21.460 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:21.460 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:21.460 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:21.460 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:21.460 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:21.460 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:21.460 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:21.461 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.440+0000 7fa0477fe700 1 -- 172.21.15.59:0/441925120 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa030009e80 con 0x7fa04813e3b0 2023-09-30T02:06:21.461 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.440+0000 7fa0477fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:21.461 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.440+0000 7fa0477fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:06:21.461 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.440+0000 7fa0477fe700 10 monclient: dump: 2023-09-30T02:06:21.461 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:21.461 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:21.461 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:21.461 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:21.461 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:21.461 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:21.461 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:21.461 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:21.462 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:21.462 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:21.462 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.440+0000 7fa0477fe700 10 monclient: _finish_auth 0 2023-09-30T02:06:21.462 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.440+0000 7fa0477fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:21.462 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.440+0000 7fa0477fe700 1 -- 172.21.15.59:0/441925120 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa03000b860 con 0x7fa04813e3b0 2023-09-30T02:06:21.462 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.440+0000 7fa0477fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:21.462 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.440+0000 7f32f314d700 10 monclient: shutdown 2023-09-30T02:06:21.462 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.440+0000 7f32f314d700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:21.463 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.440+0000 7f32f314d700 1 -- 172.21.15.59:0/818876457 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32ec13e140 msgr2=0x7f32ec13a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:21.463 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.440+0000 7f32f314d700 1 --2- 172.21.15.59:0/818876457 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32ec13e140 0x7f32ec13a750 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f32e00098e0 tx=0x7f32e00091a0 comp rx=0 tx=0).stop 2023-09-30T02:06:21.463 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.440+0000 7fa0477fe700 1 -- 172.21.15.59:0/441925120 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa03000bbe0 con 0x7fa04813e3b0 2023-09-30T02:06:21.463 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.440+0000 7fa0477fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:21.463 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.440+0000 7fa0477fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:06:21.463 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.440+0000 7fa0477fe700 10 monclient: dump: 2023-09-30T02:06:21.463 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:21.463 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:21.464 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:21.464 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:21.464 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:21.464 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:21.464 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:21.464 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:21.464 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:21.464 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:21.464 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.440+0000 7fa04e664700 5 monclient: authenticate success, global_id 4274 2023-09-30T02:06:21.464 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.440+0000 7fa04e664700 10 monclient: get_monmap_and_config success 2023-09-30T02:06:21.464 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.440+0000 7fa04e664700 4 set_mon_vals no callback set 2023-09-30T02:06:21.465 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.440+0000 7f9d137fe700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9d141529d0 0x7f9d1413ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:21.465 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.441+0000 7f9d137fe700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9d141529d0 0x7f9d1413ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:47550/0 (socket says 172.21.15.59:47550) 2023-09-30T02:06:21.465 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.441+0000 7f9d137fe700 1 -- 172.21.15.59:0/3859569758 learned_addr learned my addr 172.21.15.59:0/3859569758 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:06:21.465 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.441+0000 7f9d137fe700 10 monclient(hunting): get_auth_request con 0x7f9d141529d0 auth_method 0 2023-09-30T02:06:21.465 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.441+0000 7f9d137fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:21.465 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.441+0000 7f9d137fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:21.465 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.441+0000 7f9d137fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:21.466 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.441+0000 7f32f314d700 1 -- 172.21.15.59:0/818876457 shutdown_connections 2023-09-30T02:06:21.466 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.441+0000 7f32f314d700 1 --2- 172.21.15.59:0/818876457 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32ec13e140 0x7f32ec13a750 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:21.466 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.441+0000 7f32f314d700 1 --2- 172.21.15.59:0/818876457 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f32ec069fd0 0x7f32ec13dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:21.466 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.441+0000 7f32f314d700 1 -- 172.21.15.59:0/818876457 >> 172.21.15.59:0/818876457 conn(0x7f32ec0b08e0 msgr2=0x7f32ec0b0ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:21.466 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.441+0000 7f32f314d700 1 -- 172.21.15.59:0/818876457 shutdown_connections 2023-09-30T02:06:21.466 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.441+0000 7f32f314d700 1 -- 172.21.15.59:0/818876457 wait complete. 2023-09-30T02:06:21.467 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.441+0000 7fa04e664700 10 monclient: shutdown 2023-09-30T02:06:21.467 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.441+0000 7fa04e664700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:21.467 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.441+0000 7fa04e664700 1 -- 172.21.15.59:0/441925120 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa04813e3b0 msgr2=0x7fa04813e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:21.467 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.441+0000 7fa04e664700 1 --2- 172.21.15.59:0/441925120 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa04813e3b0 0x7fa04813e7e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa0300098e0 tx=0x7fa0300091a0 comp rx=0 tx=0).stop 2023-09-30T02:06:21.467 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.440+0000 7f29137fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:21.467 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.440+0000 7f29137fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:21.467 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.440+0000 7f29137fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:21.468 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.441+0000 7f9d137fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:21.468 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.441+0000 7f9d137fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:21.468 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.441+0000 7f9d137fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:21.468 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.441+0000 7fa04e664700 1 -- 172.21.15.59:0/441925120 shutdown_connections 2023-09-30T02:06:21.469 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.441+0000 7fa04e664700 1 --2- 172.21.15.59:0/441925120 >> v2:172.21.15.110:3300/0 conn(0x7fa0480b12c0 0x7fa0480b1690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:21.469 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.441+0000 7fa04e664700 1 --2- 172.21.15.59:0/441925120 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa04813e3b0 0x7fa04813e7e0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:21.469 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.441+0000 7fa04e664700 1 -- 172.21.15.59:0/441925120 >> 172.21.15.59:0/441925120 conn(0x7fa0480631f0 msgr2=0x7fa0480635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:21.469 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.441+0000 7fa04e664700 1 -- 172.21.15.59:0/441925120 shutdown_connections 2023-09-30T02:06:21.469 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.441+0000 7fa04e664700 1 -- 172.21.15.59:0/441925120 wait complete. 2023-09-30T02:06:21.469 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.442+0000 7fa04e664700 10 monclient: build_initial_monmap 2023-09-30T02:06:21.469 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.442+0000 7fa04e664700 10 monclient: monmap: 2023-09-30T02:06:21.469 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:21.469 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:06:21.469 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:06:21.469 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:06:21.469 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:21.470 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:06:21.470 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:06:21.470 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:06:21.470 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:06:21.470 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:21.470 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.442+0000 7f32f314d700 10 monclient: build_initial_monmap 2023-09-30T02:06:21.470 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.442+0000 7f32f314d700 10 monclient: monmap: 2023-09-30T02:06:21.471 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:21.471 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:06:21.471 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:06:21.471 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:06:21.471 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:21.471 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:06:21.471 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:06:21.471 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:06:21.471 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:06:21.471 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:21.471 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.442+0000 7f32f314d700 1 librados: starting msgr at 2023-09-30T02:06:21.471 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.442+0000 7f32f314d700 1 librados: starting objecter 2023-09-30T02:06:21.472 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.442+0000 7f32f314d700 1 Processor -- start 2023-09-30T02:06:21.472 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.442+0000 7fa04e664700 1 librados: starting msgr at 2023-09-30T02:06:21.472 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.442+0000 7fa04e664700 1 librados: starting objecter 2023-09-30T02:06:21.472 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.443+0000 7fa04e664700 1 Processor -- start 2023-09-30T02:06:21.472 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.435+0000 7fbe5dd9b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbe60087810 0x7fbe600a90a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:21.472 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.435+0000 7fbe5dd9b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbe60087810 0x7fbe600a90a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:38890/0 (socket says 172.21.15.59:38890) 2023-09-30T02:06:21.473 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.435+0000 7fbe5dd9b700 1 -- 172.21.15.59:0/656608041 learned_addr learned my addr 172.21.15.59:0/656608041 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:06:21.473 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.435+0000 7fbe5dd9b700 10 monclient(hunting): get_auth_request con 0x7fbe60087810 auth_method 0 2023-09-30T02:06:21.473 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.435+0000 7fbe5dd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:21.473 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.435+0000 7fbe5dd9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:21.473 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.435+0000 7fbe5dd9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:21.473 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.441+0000 7fbe5dd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:21.473 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.441+0000 7fbe5dd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:21.473 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.441+0000 7fbe5dd9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:21.473 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.443+0000 7fbe5e59c700 1 --2- 172.21.15.59:0/656608041 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fbe60069b10 0x7fbe601513b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:21.473 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.443+0000 7fbe5e59c700 10 monclient(hunting): get_auth_request con 0x7fbe60069b10 auth_method 0 2023-09-30T02:06:21.474 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.443+0000 7fbe5e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:21.474 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.443+0000 7fbe5e59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:21.474 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.443+0000 7fbe5e59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:21.474 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.443+0000 7f32f314d700 1 -- start start 2023-09-30T02:06:21.474 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.443+0000 7f32f314d700 1 librados: setting wanted keys 2023-09-30T02:06:21.474 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.443+0000 7f32f314d700 1 librados: calling monclient init 2023-09-30T02:06:21.474 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.443+0000 7f32f314d700 10 monclient: init 2023-09-30T02:06:21.474 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.444+0000 7f32f314d700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:21.474 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.444+0000 7f32f314d700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T02:06:21.475 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.444+0000 7f32f314d700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f32ec069fd0 0x7f32ec147500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:21.475 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.444+0000 7f32f314d700 10 monclient(hunting): picked mon.noname-c con 0x7f32ec069fd0 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:21.475 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.444+0000 7f32f314d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32ec0b13f0 0x7f32ec11a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:21.475 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.444+0000 7f32f314d700 10 monclient(hunting): picked mon.noname-a con 0x7f32ec0b13f0 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:06:21.475 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.444+0000 7f32f314d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f32ec13e140 0x7f32ec11ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:21.475 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.444+0000 7f32f314d700 10 monclient(hunting): picked mon.noname-b con 0x7f32ec13e140 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:06:21.475 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.444+0000 7f32f314d700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:21.475 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.444+0000 7f32f314d700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32ec11a5f0 con 0x7f32ec0b13f0 2023-09-30T02:06:21.475 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.444+0000 7f32f314d700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:21.475 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.444+0000 7f32f314d700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32ec11a860 con 0x7f32ec13e140 2023-09-30T02:06:21.475 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.444+0000 7f32f314d700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:21.475 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.444+0000 7f32f314d700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32ec11c150 con 0x7f32ec069fd0 2023-09-30T02:06:21.476 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.444+0000 7f32f314d700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:21.476 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.444+0000 7f32f314d700 10 monclient(hunting): authenticate will time out at 1826.746236s 2023-09-30T02:06:21.476 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.443+0000 7fa04e664700 1 -- start start 2023-09-30T02:06:21.476 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.443+0000 7fa04e664700 1 librados: setting wanted keys 2023-09-30T02:06:21.476 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.443+0000 7fa04e664700 1 librados: calling monclient init 2023-09-30T02:06:21.477 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.443+0000 7fa04e664700 10 monclient: init 2023-09-30T02:06:21.477 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.444+0000 7fa04e664700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:21.477 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.444+0000 7fa04e664700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T02:06:21.477 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.444+0000 7fa04e664700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0480b12c0 0x7fa04806b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:21.477 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.444+0000 7fa04e664700 10 monclient(hunting): picked mon.noname-a con 0x7fa0480b12c0 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:06:21.477 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.444+0000 7fa04e664700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa04813e3b0 0x7fa04813a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:21.477 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.444+0000 7fa04e664700 10 monclient(hunting): picked mon.noname-b con 0x7fa04813e3b0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:06:21.477 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.444+0000 7fa04e664700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa04813ed20 0x7fa04813a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:21.477 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.444+0000 7fa04e664700 10 monclient(hunting): picked mon.noname-c con 0x7fa04813ed20 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:21.477 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.444+0000 7fa04e664700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:21.477 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.444+0000 7fa04e664700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa04806cf30 con 0x7fa0480b12c0 2023-09-30T02:06:21.478 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.444+0000 7fa04e664700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:21.478 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.444+0000 7fa04e664700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa048139c90 con 0x7fa04813e3b0 2023-09-30T02:06:21.478 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.444+0000 7fa04e664700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:21.478 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.444+0000 7fa04e664700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa04806ab40 con 0x7fa04813ed20 2023-09-30T02:06:21.478 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.444+0000 7fa04e664700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:21.478 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.444+0000 7fa04e664700 10 monclient(hunting): authenticate will time out at 1826.746530s 2023-09-30T02:06:21.479 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.445+0000 7f2912ffd700 1 --2- 172.21.15.59:0/1269097232 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2914069fd0 0x7f29140a7860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:21.479 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.445+0000 7f29137fe700 10 monclient(hunting): handle_auth_done global_id 4318 payload 824 2023-09-30T02:06:21.479 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.445+0000 7f29137fe700 1 -- 172.21.15.59:0/1269097232 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f29140b13f0 msgr2=0x7f29140a54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:21.479 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.445+0000 7f29137fe700 1 --2- 172.21.15.59:0/1269097232 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f29140b13f0 0x7f29140a54e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:21.479 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.445+0000 7f29137fe700 1 -- 172.21.15.59:0/1269097232 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2914069fd0 msgr2=0x7f29140a7860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:21.479 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.445+0000 7f29137fe700 1 --2- 172.21.15.59:0/1269097232 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2914069fd0 0x7f29140a7860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:21.479 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.445+0000 7f29137fe700 10 monclient: _finish_hunting 0 2023-09-30T02:06:21.479 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.445+0000 7f29137fe700 1 monclient: found mon.noname-b 2023-09-30T02:06:21.479 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.446+0000 7f29137fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:21.480 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.446+0000 7f29137fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:06:21.480 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.446+0000 7f29137fe700 1 -- 172.21.15.59:0/1269097232 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f290c009ae0 con 0x7f291413e140 2023-09-30T02:06:21.480 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.446+0000 7f0f8daf7700 10 monclient(hunting): handle_auth_done global_id 4315 payload 274 2023-09-30T02:06:21.480 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.446+0000 7f0f8daf7700 1 -- 172.21.15.59:0/3080185083 >> v2:172.21.15.59:3300/0 conn(0x7f0f8806a170 msgr2=0x7f0f88141f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:06:21.480 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.446+0000 7f0f8daf7700 1 --2- 172.21.15.59:0/3080185083 >> v2:172.21.15.59:3300/0 conn(0x7f0f8806a170 0x7f0f88141f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:21.480 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.446+0000 7f0f8daf7700 1 -- 172.21.15.59:0/3080185083 >> v1:172.21.15.110:6789/0 conn(0x7f0f880ab4e0 legacy=0x7f0f8813b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:06:21.480 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.446+0000 7f0f8daf7700 10 monclient: _finish_hunting 0 2023-09-30T02:06:21.481 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.446+0000 7f0f8daf7700 1 monclient: found mon.noname-c 2023-09-30T02:06:21.481 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.446+0000 7f0f8daf7700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:21.481 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.446+0000 7f0f8daf7700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3300/0 2023-09-30T02:06:21.481 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.446+0000 7f0f8daf7700 1 -- 172.21.15.59:0/3080185083 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f8806baf0 con 0x7f0f880ab8b0 2023-09-30T02:06:21.481 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.445+0000 7fbe5ed9d700 1 --2- 172.21.15.59:0/656608041 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbe600ab2d0 0x7fbe600a95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:21.481 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.448+0000 7fbe5ed9d700 10 monclient(hunting): get_auth_request con 0x7fbe600ab2d0 auth_method 0 2023-09-30T02:06:21.482 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.448+0000 7fbe5ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:21.482 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.448+0000 7fbe5ed9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:21.482 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.448+0000 7fbe5ed9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:21.482 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.449+0000 7fbe5dd9b700 10 monclient(hunting): handle_auth_done global_id 4321 payload 824 2023-09-30T02:06:21.482 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.449+0000 7ff34516d700 1 -- 172.21.15.59:0/4021846443 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff33800e070 con 0x7ff3400ab2d0 2023-09-30T02:06:21.483 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.449+0000 7ff34516d700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:21.483 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.449+0000 7ff34516d700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T02:06:21.483 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.449+0000 7ff34516d700 10 monclient: dump: 2023-09-30T02:06:21.483 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:21.483 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:21.483 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:21.483 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:21.483 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:21.483 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:21.483 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:21.483 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:21.484 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:21.484 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:21.484 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.450+0000 7ff34516d700 10 monclient: _finish_auth 0 2023-09-30T02:06:21.484 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.450+0000 7ff34516d700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:21.484 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.449+0000 7fbe5dd9b700 1 -- 172.21.15.59:0/656608041 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fbe60069b10 msgr2=0x7fbe601513b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:21.484 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.449+0000 7fbe5dd9b700 1 --2- 172.21.15.59:0/656608041 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fbe60069b10 0x7fbe601513b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:21.484 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.449+0000 7fbe5dd9b700 1 -- 172.21.15.59:0/656608041 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbe600ab2d0 msgr2=0x7fbe600a95e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:21.484 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.449+0000 7fbe5dd9b700 1 --2- 172.21.15.59:0/656608041 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbe600ab2d0 0x7fbe600a95e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:21.485 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.450+0000 7ff34516d700 1 -- 172.21.15.59:0/4021846443 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff33800bab0 con 0x7ff3400ab2d0 2023-09-30T02:06:21.485 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.450+0000 7ff34516d700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:21.485 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.450+0000 7ff34516d700 1 -- 172.21.15.59:0/4021846443 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff33800be50 con 0x7ff3400ab2d0 2023-09-30T02:06:21.485 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.450+0000 7ff34516d700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:21.485 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.450+0000 7ff34516d700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T02:06:21.485 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.450+0000 7ff34516d700 10 monclient: dump: 2023-09-30T02:06:21.485 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:21.485 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:21.485 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:21.485 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:21.485 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:21.486 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:21.486 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:21.486 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:21.486 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:21.486 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:21.486 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.449+0000 7fbe5dd9b700 10 monclient: _finish_hunting 0 2023-09-30T02:06:21.486 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.449+0000 7fbe5dd9b700 1 monclient: found mon.noname-b 2023-09-30T02:06:21.486 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.449+0000 7fbe5dd9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:21.486 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.449+0000 7fbe5dd9b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:06:21.487 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.449+0000 7fbe5dd9b700 1 -- 172.21.15.59:0/656608041 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe50008690 con 0x7fbe60087810 2023-09-30T02:06:21.487 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.450+0000 7ff3483d3700 5 monclient: authenticate success, global_id 4281 2023-09-30T02:06:21.487 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.450+0000 7ff3483d3700 10 monclient: get_monmap_and_config success 2023-09-30T02:06:21.487 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.450+0000 7ff3483d3700 4 set_mon_vals no callback set 2023-09-30T02:06:21.487 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.450+0000 7ff3483d3700 10 monclient: shutdown 2023-09-30T02:06:21.488 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.450+0000 7ff3483d3700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:21.488 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.450+0000 7ff3483d3700 1 -- 172.21.15.59:0/4021846443 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3400ab2d0 msgr2=0x7ff3400ab710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:21.488 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.450+0000 7ff3483d3700 1 --2- 172.21.15.59:0/4021846443 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3400ab2d0 0x7ff3400ab710 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff338009e60 tx=0x7ff338009780 comp rx=0 tx=0).stop 2023-09-30T02:06:21.488 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.451+0000 7ff3483d3700 1 -- 172.21.15.59:0/4021846443 shutdown_connections 2023-09-30T02:06:21.488 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.451+0000 7ff3483d3700 1 --2- 172.21.15.59:0/4021846443 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3400ab2d0 0x7ff3400ab710 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:21.488 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.451+0000 7ff3483d3700 1 --2- 172.21.15.59:0/4021846443 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff34006ba90 0x7ff34006e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:21.488 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.451+0000 7ff3483d3700 1 -- 172.21.15.59:0/4021846443 >> 172.21.15.59:0/4021846443 conn(0x7ff3400b08e0 msgr2=0x7ff3400b0ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:21.488 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.451+0000 7ff3483d3700 1 -- 172.21.15.59:0/4021846443 shutdown_connections 2023-09-30T02:06:21.488 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.451+0000 7ff3483d3700 1 -- 172.21.15.59:0/4021846443 wait complete. 2023-09-30T02:06:21.489 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.451+0000 7f9d137fe700 10 monclient(hunting): handle_auth_done global_id 4280 payload 824 2023-09-30T02:06:21.489 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.451+0000 7f9d137fe700 1 -- 172.21.15.59:0/3859569758 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9d14143010 msgr2=0x7f9d14152490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:21.489 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.451+0000 7f9d137fe700 1 --2- 172.21.15.59:0/3859569758 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9d14143010 0x7f9d14152490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:21.489 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.451+0000 7f9d137fe700 1 -- 172.21.15.59:0/3859569758 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9cfc003ae0 msgr2=0x7f9d1411c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:21.489 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.451+0000 7f9d137fe700 1 --2- 172.21.15.59:0/3859569758 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9cfc003ae0 0x7f9d1411c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:21.489 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.451+0000 7f9d137fe700 10 monclient: _finish_hunting 0 2023-09-30T02:06:21.489 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.451+0000 7f9d137fe700 1 monclient: found mon.noname-c 2023-09-30T02:06:21.489 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.451+0000 7f9d137fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:21.489 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.451+0000 7f9d137fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.59:3301/0 2023-09-30T02:06:21.489 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.451+0000 7f9d137fe700 1 -- 172.21.15.59:0/3859569758 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d14143500 con 0x7f9d141529d0 2023-09-30T02:06:21.490 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.452+0000 7ff3483d3700 10 monclient: build_initial_monmap 2023-09-30T02:06:21.490 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.452+0000 7ff3483d3700 10 monclient: monmap: 2023-09-30T02:06:21.490 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:21.490 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:06:21.490 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:06:21.490 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:06:21.490 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:21.490 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:06:21.491 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:06:21.491 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:06:21.491 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:06:21.491 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:21.491 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.452+0000 7ff3483d3700 1 librados: starting msgr at 2023-09-30T02:06:21.491 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.452+0000 7ff3483d3700 1 librados: starting objecter 2023-09-30T02:06:21.491 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.452+0000 7ff3483d3700 1 Processor -- start 2023-09-30T02:06:21.491 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.452+0000 7f32ebfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32ec0b13f0 0x7f32ec11a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:21.492 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.452+0000 7ff3483d3700 1 -- start start 2023-09-30T02:06:21.492 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.452+0000 7ff3483d3700 1 librados: setting wanted keys 2023-09-30T02:06:21.492 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.452+0000 7ff3483d3700 1 librados: calling monclient init 2023-09-30T02:06:21.492 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.452+0000 7ff3483d3700 10 monclient: init 2023-09-30T02:06:21.492 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.453+0000 7ff3483d3700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:21.492 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.453+0000 7ff3483d3700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T02:06:21.492 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.453+0000 7ff3483d3700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff340069e90 0x7ff3400a9ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:21.492 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.453+0000 7ff3483d3700 10 monclient(hunting): picked mon.noname-a con 0x7ff340069e90 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:06:21.493 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.453+0000 7ff3483d3700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff34006ba90 0x7ff3400a9370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:21.493 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.453+0000 7ff3483d3700 10 monclient(hunting): picked mon.noname-c con 0x7ff34006ba90 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:21.493 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.453+0000 7ff3483d3700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff3400ab2d0 0x7ff340155410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:21.493 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.453+0000 7ff3483d3700 10 monclient(hunting): picked mon.noname-b con 0x7ff3400ab2d0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:06:21.493 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.453+0000 7ff3483d3700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:21.493 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.453+0000 7ff3483d3700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3400a84e0 con 0x7ff340069e90 2023-09-30T02:06:21.493 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.453+0000 7ff3483d3700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:21.493 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.453+0000 7ff3483d3700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3400a8650 con 0x7ff3400ab2d0 2023-09-30T02:06:21.493 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.453+0000 7ff3483d3700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:21.493 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.453+0000 7ff3483d3700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff340120d40 con 0x7ff34006ba90 2023-09-30T02:06:21.493 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.453+0000 7ff3483d3700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:21.494 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.453+0000 7ff3483d3700 10 monclient(hunting): authenticate will time out at 1826.755692s 2023-09-30T02:06:21.494 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.454+0000 7fa047fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0480b12c0 0x7fa04806b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:21.494 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.455+0000 7fa047fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0480b12c0 0x7fa04806b7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51868/0 (socket says 172.21.15.59:51868) 2023-09-30T02:06:21.494 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.455+0000 7fa047fff700 1 -- 172.21.15.59:0/2373407985 learned_addr learned my addr 172.21.15.59:0/2373407985 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:06:21.494 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.455+0000 7fa047fff700 10 monclient(hunting): get_auth_request con 0x7fa0480b12c0 auth_method 0 2023-09-30T02:06:21.494 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.455+0000 7fa047fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:21.494 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.455+0000 7fa047fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:21.495 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.455+0000 7fa047fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:21.495 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.455+0000 7f32ebfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32ec0b13f0 0x7f32ec11a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51860/0 (socket says 172.21.15.59:51860) 2023-09-30T02:06:21.495 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.455+0000 7f32ebfff700 1 -- 172.21.15.59:0/802744325 learned_addr learned my addr 172.21.15.59:0/802744325 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:06:21.495 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.455+0000 7f32ebfff700 10 monclient(hunting): get_auth_request con 0x7f32ec0b13f0 auth_method 0 2023-09-30T02:06:21.495 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.455+0000 7f32ebfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:21.495 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.455+0000 7f32ebfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:21.495 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.455+0000 7f32ebfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:21.496 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.456+0000 7f0f8daf7700 1 --2- 172.21.15.59:0/3080185083 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0f880ab8b0 0x7f0f881419d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0f88063b90 tx=0x7f0f78009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9dac50aac1193fb5 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:21.496 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.456+0000 7fa047fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:21.496 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.456+0000 7fa047fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:21.496 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.456+0000 7fa047fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:21.496 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.456+0000 7f29137fe700 1 --2- 172.21.15.59:0/1269097232 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f291413e140 0x7f29140aa090 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f290c00b610 tx=0x7f290c017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:21.497 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.456+0000 7f28faffd700 10 client.?.objecter ms_handle_connect 0x7f291413e140 2023-09-30T02:06:21.497 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.456+0000 7f28faffd700 10 client.?.objecter resend_mon_ops 2023-09-30T02:06:21.497 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.458+0000 7f32f0ee9700 1 --2- 172.21.15.59:0/802744325 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f32ec069fd0 0x7f32ec147500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:21.497 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.458+0000 7f9d137fe700 1 --2- 172.21.15.59:0/3859569758 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9d141529d0 0x7f9d1413ec50 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9d141403e0 tx=0x7f9d04008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:21.497 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.458+0000 7f9cfaffd700 10 client.?.objecter ms_handle_connect 0x7f9d141529d0 2023-09-30T02:06:21.497 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.458+0000 7f9cfaffd700 10 client.?.objecter resend_mon_ops 2023-09-30T02:06:21.498 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.460+0000 7ff34596e700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff34006ba90 0x7ff3400a9370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:21.498 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.463+0000 7ff34596e700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff34006ba90 0x7ff3400a9370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:47578/0 (socket says 172.21.15.59:47578) 2023-09-30T02:06:21.498 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.463+0000 7ff34596e700 1 -- 172.21.15.59:0/723961325 learned_addr learned my addr 172.21.15.59:0/723961325 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:06:21.498 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.463+0000 7ff34596e700 10 monclient(hunting): get_auth_request con 0x7ff34006ba90 auth_method 0 2023-09-30T02:06:21.498 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.463+0000 7ff34596e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:21.498 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.463+0000 7ff34596e700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:21.498 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.463+0000 7ff34596e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:21.499 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.459+0000 7f32f16ea700 1 --2- 172.21.15.59:0/802744325 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f32ec13e140 0x7f32ec11ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:21.499 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.463+0000 7f32f0ee9700 10 monclient(hunting): get_auth_request con 0x7f32ec069fd0 auth_method 0 2023-09-30T02:06:21.499 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.463+0000 7f32f0ee9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:21.499 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.463+0000 7f32f0ee9700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:21.499 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.463+0000 7f32f0ee9700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:21.499 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.465+0000 7f32f16ea700 10 monclient(hunting): get_auth_request con 0x7f32ec13e140 auth_method 0 2023-09-30T02:06:21.499 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.465+0000 7f32f16ea700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:21.499 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.465+0000 7f32f16ea700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:21.499 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.465+0000 7f32f16ea700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:21.500 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.465+0000 7fa047fff700 10 monclient(hunting): handle_auth_done global_id 4290 payload 824 2023-09-30T02:06:21.500 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.465+0000 7fa047fff700 1 -- 172.21.15.59:0/2373407985 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa04813ed20 msgr2=0x7fa04813a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:21.500 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.465+0000 7fa047fff700 1 --2- 172.21.15.59:0/2373407985 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa04813ed20 0x7fa04813a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:21.500 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.465+0000 7fa047fff700 1 -- 172.21.15.59:0/2373407985 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa04813e3b0 msgr2=0x7fa04813a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:21.500 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.465+0000 7fa047fff700 1 --2- 172.21.15.59:0/2373407985 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa04813e3b0 0x7fa04813a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:21.500 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.465+0000 7fa047fff700 10 monclient: _finish_hunting 0 2023-09-30T02:06:21.500 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.465+0000 7fa047fff700 1 monclient: found mon.noname-a 2023-09-30T02:06:21.500 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.465+0000 7fa047fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:21.500 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.465+0000 7fa047fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.59:3300/0 2023-09-30T02:06:21.500 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.465+0000 7fa047fff700 1 -- 172.21.15.59:0/2373407985 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa030009420 con 0x7fa0480b12c0 2023-09-30T02:06:21.501 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.465+0000 7f43faed4700 10 monclient: get_monmap_and_config 2023-09-30T02:06:21.501 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.465+0000 7f43faed4700 10 monclient: build_initial_monmap 2023-09-30T02:06:21.501 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.465+0000 7f43faed4700 10 monclient: monmap: 2023-09-30T02:06:21.501 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:21.501 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:21.501 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:06:21.466323+0000 2023-09-30T02:06:21.501 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:06:21.466323+0000 2023-09-30T02:06:21.501 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:21.501 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:21.502 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:06:21.502 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:06:21.502 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:06:21.502 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:06:21.502 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:06:21.502 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:21.502 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.465+0000 7f28faffd700 1 -- 172.21.15.59:0/1269097232 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f290c012c80 con 0x7f291413e140 2023-09-30T02:06:21.502 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.465+0000 7f28faffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:21.503 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.465+0000 7f28faffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:06:21.503 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.465+0000 7f28faffd700 10 monclient: dump: 2023-09-30T02:06:21.503 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:21.503 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:21.503 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:21.503 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:21.503 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:21.503 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:21.503 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:21.503 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:21.503 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:21.504 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:21.504 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.465+0000 7f28faffd700 10 monclient: _finish_auth 0 2023-09-30T02:06:21.504 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.465+0000 7f28faffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:21.504 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.465+0000 7f28faffd700 1 -- 172.21.15.59:0/1269097232 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f290c012e00 con 0x7f291413e140 2023-09-30T02:06:21.504 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.465+0000 7f28faffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:21.504 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.465+0000 7f28faffd700 1 -- 172.21.15.59:0/1269097232 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f290c01d670 con 0x7f291413e140 2023-09-30T02:06:21.504 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.465+0000 7f28faffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:21.504 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.465+0000 7f28faffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:06:21.504 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.465+0000 7f28faffd700 10 monclient: dump: 2023-09-30T02:06:21.504 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:21.504 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:21.505 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:21.505 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:21.505 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:21.505 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:21.505 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:21.505 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:21.505 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:21.505 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:21.505 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.465+0000 7f29117fa700 4 set_mon_vals no callback set 2023-09-30T02:06:21.505 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.465+0000 7f32ebfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:21.506 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.465+0000 7f32ebfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:21.506 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.465+0000 7f32ebfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:21.506 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.466+0000 7f291972e700 5 monclient: authenticate success, global_id 4318 2023-09-30T02:06:21.506 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.466+0000 7f291972e700 10 monclient: _renew_subs 2023-09-30T02:06:21.506 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.466+0000 7f291972e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:21.507 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.466+0000 7f291972e700 1 -- 172.21.15.59:0/1269097232 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2914134bd0 con 0x7f291413e140 2023-09-30T02:06:21.507 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.466+0000 7f291972e700 10 client.4318.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:06:21.507 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.466+0000 7f291972e700 10 monclient: _renew_subs 2023-09-30T02:06:21.507 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.466+0000 7f291972e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:21.507 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.466+0000 7f291972e700 1 -- 172.21.15.59:0/1269097232 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29140a8620 con 0x7f291413e140 2023-09-30T02:06:21.507 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.466+0000 7f291972e700 1 librados: init done 2023-09-30T02:06:21.507 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.467+0000 7f291972e700 10 client.4318.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:21.507 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.467+0000 7f291972e700 15 client.4318.objecter _session_command_op_assign -1 1 2023-09-30T02:06:21.507 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.467+0000 7f291972e700 20 client.4318.objecter _get_session osd=-1 returning homeless 2023-09-30T02:06:21.507 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.467+0000 7f291972e700 10 client.4318.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:06:21.507 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.467+0000 7f291972e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:21.507 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.467+0000 7f291972e700 1 -- 172.21.15.59:0/1269097232 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2914072b30 con 0x7f291413e140 2023-09-30T02:06:21.508 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.467+0000 7ff34596e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:21.508 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.467+0000 7ff34596e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:21.508 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.467+0000 7ff34596e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:21.508 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.468+0000 7f43faed4700 1 Processor -- start 2023-09-30T02:06:21.508 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.468+0000 7f32f0ee9700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:21.509 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.468+0000 7f32f0ee9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:21.509 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.468+0000 7f32f0ee9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:21.509 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.468+0000 7f43faed4700 1 -- start start 2023-09-30T02:06:21.509 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.468+0000 7f43faed4700 10 monclient: init 2023-09-30T02:06:21.509 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.468+0000 7f0f8d2f6700 1 -- 172.21.15.59:0/3080185083 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f7800b820 con 0x7f0f880ab8b0 2023-09-30T02:06:21.510 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.469+0000 7f0f8d2f6700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:21.510 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.469+0000 7f0f8d2f6700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:06:21.510 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.469+0000 7f0f8d2f6700 10 monclient: dump: 2023-09-30T02:06:21.510 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:21.510 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:21.510 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:21.510 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:21.510 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:21.510 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:21.510 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:21.511 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:21.511 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:21.511 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:21.511 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.469+0000 7f0f8d2f6700 10 monclient: _finish_auth 0 2023-09-30T02:06:21.511 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.469+0000 7f0f8d2f6700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:21.511 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.469+0000 7f0f8d2f6700 1 -- 172.21.15.59:0/3080185083 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f7800b980 con 0x7f0f880ab8b0 2023-09-30T02:06:21.511 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.469+0000 7f0f8d2f6700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:21.511 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.469+0000 7f0f8d2f6700 1 -- 172.21.15.59:0/3080185083 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f7800bd20 con 0x7f0f880ab8b0 2023-09-30T02:06:21.511 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.469+0000 7f0f8d2f6700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:21.511 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.469+0000 7f0f8d2f6700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:06:21.511 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.469+0000 7f0f8d2f6700 10 monclient: dump: 2023-09-30T02:06:21.512 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:21.512 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:21.512 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:21.512 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:21.512 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:21.512 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:21.512 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:21.512 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:21.512 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:21.512 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:21.513 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.469+0000 7f32f16ea700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:21.513 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.469+0000 7f32f16ea700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:21.513 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.469+0000 7f32f16ea700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:21.513 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.469+0000 7f43faed4700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:21.513 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.469+0000 7f43faed4700 10 monclient: _add_conns ranks=[0,3,2,4,1] 2023-09-30T02:06:21.513 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.469+0000 7f43faed4700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f43f40a5b70 addr v1:172.21.15.59:6789/0 2023-09-30T02:06:21.513 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.469+0000 7f43faed4700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f43f4142290 0x7f43f40b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:21.513 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.469+0000 7f43faed4700 10 monclient(hunting): picked mon.noname-a con 0x7f43f4142290 addr v2:172.21.15.59:3300/0 2023-09-30T02:06:21.514 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.469+0000 7ff346970700 1 --2- 172.21.15.59:0/723961325 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff3400ab2d0 0x7ff340155410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:21.514 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.469+0000 7f43faed4700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f43f40a5f40 0x7f43f413af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:21.514 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.469+0000 7f43faed4700 10 monclient(hunting): picked mon.noname-b con 0x7f43f40a5f40 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:21.515 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.469+0000 7f43faed4700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43f40a6380 con 0x7f43f40a5b70 2023-09-30T02:06:21.515 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.469+0000 7f43faed4700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:21.515 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.469+0000 7f43faed4700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43f40a75a0 con 0x7f43f4142290 2023-09-30T02:06:21.515 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.469+0000 7f43faed4700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:21.515 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.469+0000 7f43faed4700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43f40af3f0 con 0x7f43f40a5f40 2023-09-30T02:06:21.515 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.469+0000 7f43faed4700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:21.515 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.469+0000 7f43faed4700 10 monclient(hunting): authenticate will time out at 1826.771959s 2023-09-30T02:06:21.516 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.469+0000 7f0f9055c700 5 monclient: authenticate success, global_id 4315 2023-09-30T02:06:21.516 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.469+0000 7f0f9055c700 10 monclient: get_monmap_and_config success 2023-09-30T02:06:21.516 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.469+0000 7f0f9055c700 4 set_mon_vals no callback set 2023-09-30T02:06:21.517 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.469+0000 7f0f9055c700 10 monclient: shutdown 2023-09-30T02:06:21.517 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.469+0000 7f0f9055c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:21.517 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.469+0000 7f0f9055c700 1 -- 172.21.15.59:0/3080185083 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0f880ab8b0 msgr2=0x7f0f881419d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:21.517 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.470+0000 7f0f9055c700 1 --2- 172.21.15.59:0/3080185083 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0f880ab8b0 0x7f0f881419d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0f88063b90 tx=0x7f0f78009250 comp rx=0 tx=0).stop 2023-09-30T02:06:21.517 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.470+0000 7f0f9055c700 1 -- 172.21.15.59:0/3080185083 shutdown_connections 2023-09-30T02:06:21.517 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.470+0000 7f0f9055c700 1 --2- 172.21.15.59:0/3080185083 >> v2:172.21.15.59:3300/0 conn(0x7f0f8806a170 0x7f0f88141f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:21.517 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.470+0000 7f0f9055c700 1 --2- 172.21.15.59:0/3080185083 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0f880ab8b0 0x7f0f881419d0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:21.517 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.470+0000 7f0f9055c700 1 -- 172.21.15.59:0/3080185083 >> 172.21.15.59:0/3080185083 conn(0x7f0f880b08e0 msgr2=0x7f0f880b0ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:21.517 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.470+0000 7f0f9055c700 1 -- 172.21.15.59:0/3080185083 shutdown_connections 2023-09-30T02:06:21.517 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.470+0000 7f0f9055c700 1 -- 172.21.15.59:0/3080185083 wait complete. 2023-09-30T02:06:21.517 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.471+0000 7f0f9055c700 10 monclient: build_initial_monmap 2023-09-30T02:06:21.517 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.471+0000 7f0f9055c700 10 monclient: monmap: 2023-09-30T02:06:21.518 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:21.518 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:06:21.518 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:06:21.518 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:06:21.518 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:21.518 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:06:21.518 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:06:21.518 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:06:21.518 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:06:21.518 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:21.518 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.471+0000 7f0f9055c700 1 librados: starting msgr at 2023-09-30T02:06:21.518 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.471+0000 7f0f9055c700 1 librados: starting objecter 2023-09-30T02:06:21.519 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.471+0000 7f0f9055c700 1 Processor -- start 2023-09-30T02:06:21.519 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.472+0000 7f0f9055c700 1 -- start start 2023-09-30T02:06:21.519 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.472+0000 7f0f9055c700 1 librados: setting wanted keys 2023-09-30T02:06:21.519 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.472+0000 7f0f9055c700 1 librados: calling monclient init 2023-09-30T02:06:21.519 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.472+0000 7f0f9055c700 10 monclient: init 2023-09-30T02:06:21.519 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.472+0000 7ff346970700 10 monclient(hunting): get_auth_request con 0x7ff3400ab2d0 auth_method 0 2023-09-30T02:06:21.519 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.472+0000 7ff346970700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:21.519 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.472+0000 7ff346970700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:21.519 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.472+0000 7ff346970700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:21.520 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.472+0000 7f0f9055c700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:21.520 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.472+0000 7f0f9055c700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T02:06:21.520 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.472+0000 7f0f9055c700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0f8806a170 0x7f0f88151fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:21.520 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.472+0000 7f0f9055c700 10 monclient(hunting): picked mon.noname-c con 0x7f0f8806a170 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:21.520 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.472+0000 7f0f9055c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0f880ab4e0 0x7f0f8811ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:21.520 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.472+0000 7f0f9055c700 10 monclient(hunting): picked mon.noname-b con 0x7f0f880ab4e0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:06:21.521 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.472+0000 7f0f9055c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f880ab8b0 0x7f0f8811b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:21.521 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.472+0000 7f0f9055c700 10 monclient(hunting): picked mon.noname-a con 0x7f0f880ab8b0 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:06:21.521 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.472+0000 7f0f9055c700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:21.521 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.472+0000 7f0f9055c700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f8811b3f0 con 0x7f0f880ab8b0 2023-09-30T02:06:21.521 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.472+0000 7f0f9055c700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:21.521 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.472+0000 7f0f9055c700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f8811b0d0 con 0x7f0f880ab4e0 2023-09-30T02:06:21.521 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.472+0000 7f0f9055c700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:21.521 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.472+0000 7f0f9055c700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f88152a40 con 0x7f0f8806a170 2023-09-30T02:06:21.521 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.472+0000 7f0f9055c700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:21.521 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.472+0000 7f0f9055c700 10 monclient(hunting): authenticate will time out at 1826.774687s 2023-09-30T02:06:21.522 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.473+0000 7fbe5dd9b700 1 --2- 172.21.15.59:0/656608041 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbe60087810 0x7fbe600a90a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fbe5800e190 tx=0x7fbe58009920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:21.522 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.473+0000 7fbe4e7fc700 10 client.?.objecter ms_handle_connect 0x7fbe60087810 2023-09-30T02:06:21.522 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.473+0000 7fbe4e7fc700 10 client.?.objecter resend_mon_ops 2023-09-30T02:06:21.522 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.473+0000 7ff346970700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:21.522 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.473+0000 7ff346970700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:21.522 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.473+0000 7ff346970700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:21.523 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.473+0000 7f32ebfff700 10 monclient(hunting): handle_auth_done global_id 4293 payload 824 2023-09-30T02:06:21.523 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.473+0000 7f32ebfff700 1 -- 172.21.15.59:0/802744325 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f32ec069fd0 msgr2=0x7f32ec147500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:21.523 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.473+0000 7f32ebfff700 1 --2- 172.21.15.59:0/802744325 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f32ec069fd0 0x7f32ec147500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:21.523 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.473+0000 7f32ebfff700 1 -- 172.21.15.59:0/802744325 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f32ec13e140 msgr2=0x7f32ec11ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:21.523 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.473+0000 7f32ebfff700 1 --2- 172.21.15.59:0/802744325 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f32ec13e140 0x7f32ec11ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:21.523 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.473+0000 7f32ebfff700 10 monclient: _finish_hunting 0 2023-09-30T02:06:21.523 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.473+0000 7f32ebfff700 1 monclient: found mon.noname-a 2023-09-30T02:06:21.523 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.473+0000 7f32ebfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:21.523 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.473+0000 7f32ebfff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.59:3300/0 2023-09-30T02:06:21.523 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.473+0000 7f32ebfff700 1 -- 172.21.15.59:0/802744325 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32e0009420 con 0x7f32ec0b13f0 2023-09-30T02:06:21.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.473+0000 7ff34596e700 10 monclient(hunting): handle_auth_done global_id 4286 payload 824 2023-09-30T02:06:21.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.473+0000 7ff34596e700 1 -- 172.21.15.59:0/723961325 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff3400ab2d0 msgr2=0x7ff340155410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:21.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.473+0000 7ff34596e700 1 --2- 172.21.15.59:0/723961325 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff3400ab2d0 0x7ff340155410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:21.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.473+0000 7ff34596e700 1 -- 172.21.15.59:0/723961325 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff340069e90 msgr2=0x7ff3400a9ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:21.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.473+0000 7ff34596e700 1 --2- 172.21.15.59:0/723961325 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff340069e90 0x7ff3400a9ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:21.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.473+0000 7ff34596e700 10 monclient: _finish_hunting 0 2023-09-30T02:06:21.526 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.473+0000 7ff34596e700 1 monclient: found mon.noname-c 2023-09-30T02:06:21.526 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.473+0000 7ff34596e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:21.526 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.473+0000 7ff34596e700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.59:3301/0 2023-09-30T02:06:21.526 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.473+0000 7ff34596e700 1 -- 172.21.15.59:0/723961325 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff338009a20 con 0x7ff34006ba90 2023-09-30T02:06:21.526 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.476+0000 7fa047fff700 1 --2- 172.21.15.59:0/2373407985 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0480b12c0 0x7fa04806b7e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa03800bc30 tx=0x7fa038009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:21.526 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.476+0000 7fa044ff9700 10 client.?.objecter ms_handle_connect 0x7fa0480b12c0 2023-09-30T02:06:21.526 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.476+0000 7fa044ff9700 10 client.?.objecter resend_mon_ops 2023-09-30T02:06:21.527 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.476+0000 7f9cfaffd700 1 -- 172.21.15.59:0/3859569758 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d04014600 con 0x7f9d141529d0 2023-09-30T02:06:21.527 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.476+0000 7f9cfaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:21.527 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.476+0000 7f9cfaffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:06:21.527 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.476+0000 7f9cfaffd700 10 monclient: dump: 2023-09-30T02:06:21.527 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:21.527 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:21.527 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:21.527 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:21.528 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:21.528 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:21.528 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:21.528 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:21.528 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:21.528 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:21.528 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.476+0000 7f9cfaffd700 10 monclient: _finish_auth 0 2023-09-30T02:06:21.528 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.476+0000 7f9cfaffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:21.528 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.477+0000 7f9cfaffd700 1 -- 172.21.15.59:0/3859569758 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d04014760 con 0x7f9d141529d0 2023-09-30T02:06:21.528 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.477+0000 7f9cfaffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:21.528 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.477+0000 7f9cfaffd700 1 -- 172.21.15.59:0/3859569758 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d04014b00 con 0x7f9d141529d0 2023-09-30T02:06:21.529 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.477+0000 7f9cfaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:21.529 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.477+0000 7f9cfaffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:06:21.529 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.477+0000 7f9cfaffd700 10 monclient: dump: 2023-09-30T02:06:21.529 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:21.529 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:21.529 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:21.529 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:21.529 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:21.529 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:21.529 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:21.529 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:21.529 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:21.529 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:21.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.477+0000 7f9d199df700 5 monclient: authenticate success, global_id 4280 2023-09-30T02:06:21.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.477+0000 7f9d199df700 10 monclient: _renew_subs 2023-09-30T02:06:21.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.477+0000 7f9d199df700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:06:21.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.477+0000 7f9d199df700 1 -- 172.21.15.59:0/3859569758 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d14141f20 con 0x7f9d141529d0 2023-09-30T02:06:21.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.477+0000 7f9d199df700 10 client.4280.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:06:21.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.477+0000 7f9d199df700 10 monclient: _renew_subs 2023-09-30T02:06:21.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.477+0000 7f9d199df700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:06:21.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.477+0000 7f9d199df700 1 -- 172.21.15.59:0/3859569758 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d1413f190 con 0x7f9d141529d0 2023-09-30T02:06:21.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.477+0000 7f9d199df700 1 librados: init done 2023-09-30T02:06:21.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.477+0000 7f9d117fa700 4 set_mon_vals no callback set 2023-09-30T02:06:21.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.477+0000 7f9d199df700 10 client.4280.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:21.531 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.478+0000 7f9d199df700 15 client.4280.objecter _session_command_op_assign -1 1 2023-09-30T02:06:21.531 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.478+0000 7f9d199df700 20 client.4280.objecter _get_session osd=-1 returning homeless 2023-09-30T02:06:21.531 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.478+0000 7f9d199df700 10 client.4280.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:06:21.531 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.478+0000 7f9d199df700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:06:21.531 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.478+0000 7f9d199df700 1 -- 172.21.15.59:0/3859569758 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9d1406eaa0 con 0x7f9d141529d0 2023-09-30T02:06:21.531 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.477+0000 7f32ebfff700 1 --2- 172.21.15.59:0/802744325 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32ec0b13f0 0x7f32ec11a0b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f32dc00d890 tx=0x7f32dc008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:21.531 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.478+0000 7f32e8ff9700 10 client.?.objecter ms_handle_connect 0x7f32ec0b13f0 2023-09-30T02:06:21.531 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.478+0000 7f32e8ff9700 10 client.?.objecter resend_mon_ops 2023-09-30T02:06:21.532 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.478+0000 7f0f8daf7700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0f880ab4e0 0x7f0f8811ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:21.532 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.479+0000 7f0f8daf7700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0f880ab4e0 0x7f0f8811ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:38970/0 (socket says 172.21.15.59:38970) 2023-09-30T02:06:21.532 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.479+0000 7f0f8daf7700 1 -- 172.21.15.59:0/3803865518 learned_addr learned my addr 172.21.15.59:0/3803865518 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:06:21.532 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.479+0000 7f0f8daf7700 10 monclient(hunting): get_auth_request con 0x7f0f880ab4e0 auth_method 0 2023-09-30T02:06:21.532 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.479+0000 7f0f8daf7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:21.532 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.479+0000 7f0f8daf7700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:21.532 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.479+0000 7f0f8daf7700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:21.533 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.481+0000 7f0f8daf7700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:21.533 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.481+0000 7f0f8daf7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:21.533 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.481+0000 7f0f8daf7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:21.534 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.483+0000 7fa044ff9700 1 -- 172.21.15.59:0/2373407985 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0380092e0 con 0x7fa0480b12c0 2023-09-30T02:06:21.534 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.483+0000 7fa044ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:21.534 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.483+0000 7fa044ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T02:06:21.534 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.483+0000 7fa044ff9700 10 monclient: dump: 2023-09-30T02:06:21.534 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:21.534 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:21.534 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:21.534 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:21.534 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:21.535 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:21.535 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:21.535 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:21.535 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:21.535 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:21.535 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.483+0000 7fa044ff9700 10 monclient: _finish_auth 0 2023-09-30T02:06:21.535 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.483+0000 7fa044ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:21.535 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.483+0000 7fa044ff9700 1 -- 172.21.15.59:0/2373407985 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa03800c7a0 con 0x7fa0480b12c0 2023-09-30T02:06:21.535 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.483+0000 7fa044ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:21.535 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.483+0000 7fa044ff9700 1 -- 172.21.15.59:0/2373407985 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa03800cb80 con 0x7fa0480b12c0 2023-09-30T02:06:21.536 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.483+0000 7fa044ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:21.536 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.483+0000 7fa044ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T02:06:21.536 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.483+0000 7fa044ff9700 10 monclient: dump: 2023-09-30T02:06:21.536 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:21.536 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:21.536 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:21.536 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:21.536 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:21.536 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:21.536 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:21.536 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:21.537 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:21.537 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:21.537 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.483+0000 7fa046ffd700 4 set_mon_vals no callback set 2023-09-30T02:06:21.537 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.483+0000 7fa04e664700 5 monclient: authenticate success, global_id 4290 2023-09-30T02:06:21.537 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.483+0000 7fa04e664700 10 monclient: _renew_subs 2023-09-30T02:06:21.537 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.483+0000 7fa04e664700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:06:21.537 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.483+0000 7fa04e664700 1 -- 172.21.15.59:0/2373407985 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa04813ac30 con 0x7fa0480b12c0 2023-09-30T02:06:21.537 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.483+0000 7fa04e664700 10 client.4290.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:06:21.537 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.483+0000 7fa04e664700 10 monclient: _renew_subs 2023-09-30T02:06:21.537 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.483+0000 7fa04e664700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:06:21.537 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.483+0000 7fa04e664700 1 -- 172.21.15.59:0/2373407985 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0481368d0 con 0x7fa0480b12c0 2023-09-30T02:06:21.538 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.483+0000 7fa04e664700 1 librados: init done 2023-09-30T02:06:21.538 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.484+0000 7fa04e664700 10 client.4290.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:21.538 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.484+0000 7fa04e664700 15 client.4290.objecter _session_command_op_assign -1 1 2023-09-30T02:06:21.538 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.484+0000 7fa04e664700 20 client.4290.objecter _get_session osd=-1 returning homeless 2023-09-30T02:06:21.538 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.484+0000 7fa04e664700 10 client.4290.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:06:21.538 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.484+0000 7fa04e664700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:06:21.538 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.484+0000 7fa04e664700 1 -- 172.21.15.59:0/2373407985 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa048139e00 con 0x7fa0480b12c0 2023-09-30T02:06:21.539 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.496+0000 7f0f8daf7700 10 monclient(hunting): handle_auth_done global_id 4330 payload 824 2023-09-30T02:06:21.539 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.496+0000 7f0f8daf7700 1 -- 172.21.15.59:0/3803865518 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0f8806a170 msgr2=0x7f0f88151fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:21.539 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.496+0000 7f0f8daf7700 1 --2- 172.21.15.59:0/3803865518 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0f8806a170 0x7f0f88151fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:21.539 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.496+0000 7f0f8daf7700 1 -- 172.21.15.59:0/3803865518 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f880ab8b0 msgr2=0x7f0f8811b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:21.539 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.496+0000 7f0f8daf7700 1 --2- 172.21.15.59:0/3803865518 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0f880ab8b0 0x7f0f8811b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:21.539 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.496+0000 7f0f8daf7700 10 monclient: _finish_hunting 0 2023-09-30T02:06:21.539 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.496+0000 7f0f8daf7700 1 monclient: found mon.noname-b 2023-09-30T02:06:21.539 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.496+0000 7f0f8daf7700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:21.539 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.496+0000 7f0f8daf7700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:06:21.539 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.496+0000 7f0f8daf7700 1 -- 172.21.15.59:0/3803865518 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f780094f0 con 0x7f0f880ab4e0 2023-09-30T02:06:21.540 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.501+0000 7fbe4e7fc700 1 -- 172.21.15.59:0/656608041 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe5800edd0 con 0x7fbe60087810 2023-09-30T02:06:21.540 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.501+0000 7fbe4e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:21.541 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.501+0000 7fbe4e7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:06:21.541 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.501+0000 7fbe4e7fc700 10 monclient: dump: 2023-09-30T02:06:21.541 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:21.541 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:21.541 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:21.541 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:21.541 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:21.541 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:21.541 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:21.541 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:21.542 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:21.542 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:21.542 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.501+0000 7fbe4e7fc700 10 monclient: _finish_auth 0 2023-09-30T02:06:21.542 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.501+0000 7fbe4e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:21.542 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.501+0000 7fbe4e7fc700 1 -- 172.21.15.59:0/656608041 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe58015480 con 0x7fbe60087810 2023-09-30T02:06:21.542 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.501+0000 7fbe4e7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:21.542 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.501+0000 7fbe4e7fc700 1 -- 172.21.15.59:0/656608041 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe58015860 con 0x7fbe60087810 2023-09-30T02:06:21.542 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.501+0000 7fbe64a5d700 5 monclient: authenticate success, global_id 4321 2023-09-30T02:06:21.542 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.501+0000 7fbe64a5d700 10 monclient: _renew_subs 2023-09-30T02:06:21.542 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.501+0000 7fbe64a5d700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:21.542 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.501+0000 7fbe64a5d700 1 -- 172.21.15.59:0/656608041 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe6014f9c0 con 0x7fbe60087810 2023-09-30T02:06:21.542 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.501+0000 7fbe5cd99700 4 set_mon_vals no callback set 2023-09-30T02:06:21.543 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.501+0000 7fbe4e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:21.543 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.501+0000 7fbe4e7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:06:21.543 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.501+0000 7fbe4e7fc700 10 monclient: dump: 2023-09-30T02:06:21.543 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:21.543 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:21.543 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:21.543 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:21.543 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:21.543 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:21.543 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:21.543 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:21.543 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:21.544 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:21.544 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.501+0000 7fbe64a5d700 10 client.4321.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:06:21.544 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.501+0000 7fbe64a5d700 10 monclient: _renew_subs 2023-09-30T02:06:21.544 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.501+0000 7fbe64a5d700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:21.544 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.501+0000 7fbe64a5d700 1 -- 172.21.15.59:0/656608041 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe600a9b20 con 0x7fbe60087810 2023-09-30T02:06:21.544 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.501+0000 7fbe64a5d700 1 librados: init done 2023-09-30T02:06:21.544 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.502+0000 7f0f8daf7700 1 --2- 172.21.15.59:0/3803865518 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0f880ab4e0 0x7f0f8811ab90 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0f78003080 tx=0x7f0f78016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:21.544 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.502+0000 7f0f7e7fc700 10 client.?.objecter ms_handle_connect 0x7f0f880ab4e0 2023-09-30T02:06:21.544 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.502+0000 7f0f7e7fc700 10 client.?.objecter resend_mon_ops 2023-09-30T02:06:21.545 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.502+0000 7fbe64a5d700 10 client.4321.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:21.545 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.502+0000 7fbe64a5d700 15 client.4321.objecter _session_command_op_assign -1 1 2023-09-30T02:06:21.545 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.502+0000 7fbe64a5d700 20 client.4321.objecter _get_session osd=-1 returning homeless 2023-09-30T02:06:21.545 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.502+0000 7fbe64a5d700 10 client.4321.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:06:21.545 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.502+0000 7fbe64a5d700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:21.545 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.502+0000 7fbe64a5d700 1 -- 172.21.15.59:0/656608041 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fbe600a9d70 con 0x7fbe60087810 2023-09-30T02:06:21.546 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.526+0000 7f0f7e7fc700 1 -- 172.21.15.59:0/3803865518 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f7800bd10 con 0x7f0f880ab4e0 2023-09-30T02:06:21.546 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.526+0000 7f0f7e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:21.546 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.526+0000 7f0f7e7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:06:21.546 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.526+0000 7f0f7e7fc700 10 monclient: dump: 2023-09-30T02:06:21.546 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:21.546 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:21.546 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:21.547 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:21.547 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:21.547 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:21.547 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:21.547 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:21.547 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:21.547 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:21.547 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.526+0000 7f0f7e7fc700 10 monclient: _finish_auth 0 2023-09-30T02:06:21.547 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.526+0000 7f0f7e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:21.547 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.526+0000 7f0f9055c700 5 monclient: authenticate success, global_id 4330 2023-09-30T02:06:21.547 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.526+0000 7f0f9055c700 10 monclient: _renew_subs 2023-09-30T02:06:21.547 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.526+0000 7f0f9055c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:21.548 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.526+0000 7f0f9055c700 1 -- 172.21.15.59:0/3803865518 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f88152c00 con 0x7f0f880ab4e0 2023-09-30T02:06:21.548 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.526+0000 7f0f9055c700 10 client.4330.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:06:21.548 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.526+0000 7f0f9055c700 10 monclient: _renew_subs 2023-09-30T02:06:21.548 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.526+0000 7f0f9055c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:21.548 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.526+0000 7f0f9055c700 1 -- 172.21.15.59:0/3803865518 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f881186f0 con 0x7f0f880ab4e0 2023-09-30T02:06:21.548 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.526+0000 7f0f9055c700 1 librados: init done 2023-09-30T02:06:21.548 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.527+0000 7f0f9055c700 10 client.4330.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:21.548 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.527+0000 7f0f9055c700 15 client.4330.objecter _session_command_op_assign -1 1 2023-09-30T02:06:21.548 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.527+0000 7f0f9055c700 20 client.4330.objecter _get_session osd=-1 returning homeless 2023-09-30T02:06:21.548 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.527+0000 7f0f9055c700 10 client.4330.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:06:21.548 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.527+0000 7f0f9055c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:21.549 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.527+0000 7f0f9055c700 1 -- 172.21.15.59:0/3803865518 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0f8811b530 con 0x7f0f880ab4e0 2023-09-30T02:06:21.550 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.529+0000 7fa044ff9700 1 -- 172.21.15.59:0/2373407985 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fa038012070 con 0x7fa0480b12c0 2023-09-30T02:06:21.550 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.530+0000 7fa044ff9700 1 --2- 172.21.15.59:0/2373407985 >> v2:172.21.15.110:6800/115870 conn(0x7fa028068890 0x7fa02806ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:21.550 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.530+0000 7fa044ff9700 1 -- 172.21.15.59:0/2373407985 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fa038083da0 con 0x7fa0480b12c0 2023-09-30T02:06:21.550 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.530+0000 7fa044ff9700 10 client.4290.objecter ms_dispatch 0x7fa048001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:06:21.550 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.530+0000 7fa044ff9700 3 client.4290.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:06:21.550 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.530+0000 7fa044ff9700 3 client.4290.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:06:21.550 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.530+0000 7fa03f7fe700 1 --2- 172.21.15.59:0/2373407985 >> v2:172.21.15.110:6800/115870 conn(0x7fa028068890 0x7fa02806ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:21.550 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.530+0000 7fa03f7fe700 10 monclient: get_auth_request con 0x7fa028068890 auth_method 0 2023-09-30T02:06:21.550 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.530+0000 7fa044ff9700 10 client.4290.objecter checking command 1 2023-09-30T02:06:21.551 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.530+0000 7fa044ff9700 1 --2- 172.21.15.59:0/2373407985 >> v2:172.21.15.59:6812/116667 conn(0x7fa02806e800 0x7fa028070c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:21.551 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.530+0000 7fa044ff9700 20 client.4290.objecter _get_session s=0x7fa02806e150 osd=3 3 2023-09-30T02:06:21.551 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.530+0000 7fa044ff9700 20 client.4290.objecter put_session s=0x7fa02806e150 osd=3 3 2023-09-30T02:06:21.551 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.530+0000 7fa044ff9700 15 client.4290.objecter _session_command_op_remove -1 1 2023-09-30T02:06:21.551 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.530+0000 7fa044ff9700 20 client.4290.objecter _get_session s=0x7fa02806e150 osd=3 3 2023-09-30T02:06:21.551 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.530+0000 7fa044ff9700 20 client.4290.objecter get_session s=0x7fa02806e150 osd=3 3 2023-09-30T02:06:21.551 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.530+0000 7fa044ff9700 15 client.4290.objecter _session_command_op_assign 3 1 2023-09-30T02:06:21.551 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.530+0000 7fa044ff9700 20 client.4290.objecter put_session s=0x7fa02806e150 osd=3 4 2023-09-30T02:06:21.551 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.530+0000 7fa044ff9700 10 client.4290.objecter _send_command 1 2023-09-30T02:06:21.551 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.530+0000 7fa044ff9700 1 -- 172.21.15.59:0/2373407985 --> v2:172.21.15.59:6812/116667 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa028071280 con 0x7fa02806e800 2023-09-30T02:06:21.551 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.530+0000 7fa044ff9700 20 client.4290.objecter dump_active .. 0 homeless 2023-09-30T02:06:21.552 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.530+0000 7fa044ff9700 1 -- 172.21.15.59:0/2373407985 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa038084170 con 0x7fa0480b12c0 2023-09-30T02:06:21.552 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.530+0000 7fa044ff9700 10 monclient: handle_get_version_reply finishing 1 version 25 2023-09-30T02:06:21.552 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.531+0000 7fa044ff9700 10 client.4290.objecter ms_handle_connect 0x7fa028068890 2023-09-30T02:06:21.552 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.531+0000 7fa03f7fe700 1 --2- 172.21.15.59:0/2373407985 >> v2:172.21.15.110:6800/115870 conn(0x7fa028068890 0x7fa02806ad40 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa03000c010 tx=0x7fa030017000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:21.552 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.531+0000 7fa04cc01700 1 --2- 172.21.15.59:0/2373407985 >> v2:172.21.15.59:6812/116667 conn(0x7fa02806e800 0x7fa028070c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:21.552 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.531+0000 7f43f3fff700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f43f4142290 0x7f43f40b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:21.552 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.531+0000 7fa04cc01700 10 monclient: get_auth_request con 0x7fa02806e800 auth_method 0 2023-09-30T02:06:21.553 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.532+0000 7f43f3fff700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f43f4142290 0x7f43f40b1500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51884/0 (socket says 172.21.15.59:51884) 2023-09-30T02:06:21.553 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.532+0000 7f43f3fff700 1 -- 172.21.15.59:0/75758567 learned_addr learned my addr 172.21.15.59:0/75758567 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:06:21.553 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.532+0000 7f43f3fff700 10 monclient(hunting): get_auth_request con 0x7f43f4142290 auth_method 0 2023-09-30T02:06:21.553 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.532+0000 7f43f3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:21.553 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.532+0000 7f43f3fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:21.553 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.532+0000 7f43f3fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:21.554 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.536+0000 7f9cfaffd700 1 -- 172.21.15.59:0/3859569758 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f9d040123a0 con 0x7f9d141529d0 2023-09-30T02:06:21.555 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.536+0000 7f9cfaffd700 1 --2- 172.21.15.59:0/3859569758 >> v2:172.21.15.110:6800/115870 conn(0x7f9d00068960 0x7f9d0006ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:21.555 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.536+0000 7f9cfaffd700 1 -- 172.21.15.59:0/3859569758 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f9d04083ea0 con 0x7f9d141529d0 2023-09-30T02:06:21.555 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.536+0000 7f9cfaffd700 10 client.4280.objecter ms_dispatch 0x7f9d14001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:06:21.555 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.536+0000 7f9cfaffd700 3 client.4280.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:06:21.555 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.536+0000 7f9cfaffd700 3 client.4280.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:06:21.555 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.536+0000 7f9d12ffd700 1 --2- 172.21.15.59:0/3859569758 >> v2:172.21.15.110:6800/115870 conn(0x7f9d00068960 0x7f9d0006ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:21.555 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.537+0000 7f9cfaffd700 10 client.4280.objecter checking command 1 2023-09-30T02:06:21.555 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.537+0000 7f9d12ffd700 10 monclient: get_auth_request con 0x7f9d00068960 auth_method 0 2023-09-30T02:06:21.555 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.537+0000 7f9cfaffd700 1 --2- 172.21.15.59:0/3859569758 >> v2:172.21.15.59:6804/116666 conn(0x7f9d0006e8d0 0x7f9d00070ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:21.556 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.537+0000 7f9cfaffd700 20 client.4280.objecter _get_session s=0x7f9d0006e220 osd=2 3 2023-09-30T02:06:21.556 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.537+0000 7f9cfaffd700 20 client.4280.objecter put_session s=0x7f9d0006e220 osd=2 3 2023-09-30T02:06:21.556 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.537+0000 7f9cfaffd700 15 client.4280.objecter _session_command_op_remove -1 1 2023-09-30T02:06:21.556 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.537+0000 7f9cfaffd700 20 client.4280.objecter _get_session s=0x7f9d0006e220 osd=2 3 2023-09-30T02:06:21.556 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.537+0000 7f9cfaffd700 20 client.4280.objecter get_session s=0x7f9d0006e220 osd=2 3 2023-09-30T02:06:21.556 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.537+0000 7f9cfaffd700 15 client.4280.objecter _session_command_op_assign 2 1 2023-09-30T02:06:21.556 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.537+0000 7f9cfaffd700 20 client.4280.objecter put_session s=0x7f9d0006e220 osd=2 4 2023-09-30T02:06:21.556 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.537+0000 7f9cfaffd700 10 client.4280.objecter _send_command 1 2023-09-30T02:06:21.556 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.537+0000 7f9cfaffd700 1 -- 172.21.15.59:0/3859569758 --> v2:172.21.15.59:6804/116666 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9d00071350 con 0x7f9d0006e8d0 2023-09-30T02:06:21.556 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.537+0000 7f9cfaffd700 20 client.4280.objecter dump_active .. 0 homeless 2023-09-30T02:06:21.556 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.537+0000 7f9cfaffd700 1 -- 172.21.15.59:0/3859569758 <== mon.2 v2:172.21.15.59:3301/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9d04084190 con 0x7f9d141529d0 2023-09-30T02:06:21.557 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.537+0000 7f9cfaffd700 10 monclient: handle_get_version_reply finishing 1 version 25 2023-09-30T02:06:21.557 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.537+0000 7f9cfaffd700 10 client.4280.objecter ms_handle_connect 0x7f9d00068960 2023-09-30T02:06:21.557 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.537+0000 7f9d12ffd700 1 --2- 172.21.15.59:0/3859569758 >> v2:172.21.15.110:6800/115870 conn(0x7f9d00068960 0x7f9d0006ae10 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9d08000c00 tx=0x7f9d08015000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:21.557 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.537+0000 7f9d127fc700 1 --2- 172.21.15.59:0/3859569758 >> v2:172.21.15.59:6804/116666 conn(0x7f9d0006e8d0 0x7f9d00070ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:21.557 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.537+0000 7f43f3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:21.557 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.537+0000 7f43f3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:21.557 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.537+0000 7f43f3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:21.558 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.538+0000 7f9d127fc700 10 monclient: get_auth_request con 0x7f9d0006e8d0 auth_method 0 2023-09-30T02:06:21.558 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.538+0000 7fa044ff9700 10 client.4290.objecter ms_handle_connect 0x7fa02806e800 2023-09-30T02:06:21.558 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.538+0000 7fa04cc01700 1 --2- 172.21.15.59:0/2373407985 >> v2:172.21.15.59:6812/116667 conn(0x7fa02806e800 0x7fa028070c10 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:21.559 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.540+0000 7f43f3fff700 10 monclient(hunting): handle_auth_done global_id 4296 payload 274 2023-09-30T02:06:21.559 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.540+0000 7f43f3fff700 1 -- 172.21.15.59:0/75758567 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f43f40a5f40 msgr2=0x7f43f413af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:06:21.559 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.540+0000 7f43f3fff700 1 --2- 172.21.15.59:0/75758567 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f43f40a5f40 0x7f43f413af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:21.559 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.540+0000 7f43f3fff700 1 -- 172.21.15.59:0/75758567 >> v1:172.21.15.59:6789/0 conn(0x7f43f40a5b70 legacy=0x7f43f4141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:06:21.559 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.540+0000 7f43f3fff700 10 monclient: _finish_hunting 0 2023-09-30T02:06:21.559 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.540+0000 7f43f3fff700 1 monclient: found mon.noname-a 2023-09-30T02:06:21.559 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.540+0000 7f43f3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:21.559 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.540+0000 7f43f3fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.59:3300/0 2023-09-30T02:06:21.559 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.540+0000 7f43f3fff700 1 -- 172.21.15.59:0/75758567 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43f413b470 con 0x7f43f4142290 2023-09-30T02:06:21.560 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.541+0000 7ff34596e700 1 --2- 172.21.15.59:0/723961325 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff34006ba90 0x7ff3400a9370 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff33000ecb0 tx=0x7ff3300091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:21.560 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.541+0000 7ff3367fc700 10 client.?.objecter ms_handle_connect 0x7ff34006ba90 2023-09-30T02:06:21.560 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.541+0000 7ff3367fc700 10 client.?.objecter resend_mon_ops 2023-09-30T02:06:21.560 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.544+0000 7f9cfaffd700 10 client.4280.objecter ms_handle_connect 0x7f9d0006e8d0 2023-09-30T02:06:21.560 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.544+0000 7f9d127fc700 1 --2- 172.21.15.59:0/3859569758 >> v2:172.21.15.59:6804/116666 conn(0x7f9d0006e8d0 0x7f9d00070ce0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:21.561 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.546+0000 7f0f7e7fc700 1 -- 172.21.15.59:0/3803865518 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f8811b530 con 0x7f0f880ab4e0 2023-09-30T02:06:21.561 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.546+0000 7f0f7e7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:21.561 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.546+0000 7f0f7e7fc700 1 -- 172.21.15.59:0/3803865518 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f7801a860 con 0x7f0f880ab4e0 2023-09-30T02:06:21.561 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.546+0000 7f0f7e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:21.561 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.546+0000 7f0f7e7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:06:21.561 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.546+0000 7f0f7e7fc700 10 monclient: dump: 2023-09-30T02:06:21.561 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:21.561 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:21.561 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:21.561 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:21.562 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:21.562 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:21.562 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:21.562 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:21.562 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:21.562 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:21.562 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.546+0000 7f0f8caf5700 4 set_mon_vals no callback set 2023-09-30T02:06:21.563 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.546+0000 7fbe4e7fc700 1 -- 172.21.15.59:0/656608041 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fbe58015c90 con 0x7fbe60087810 2023-09-30T02:06:21.563 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.546+0000 7fbe4e7fc700 1 --2- 172.21.15.59:0/656608041 >> v2:172.21.15.110:6800/115870 conn(0x7fbe40068960 0x7fbe4006ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:21.563 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.547+0000 7fbe4e7fc700 1 -- 172.21.15.59:0/656608041 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fbe58084e80 con 0x7fbe60087810 2023-09-30T02:06:21.563 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.547+0000 7fbe4e7fc700 10 client.4321.objecter ms_dispatch 0x7fbe60001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:06:21.564 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.547+0000 7fbe4e7fc700 3 client.4321.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:06:21.564 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.547+0000 7fbe4e7fc700 3 client.4321.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:06:21.564 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.547+0000 7fbe5e59c700 1 --2- 172.21.15.59:0/656608041 >> v2:172.21.15.110:6800/115870 conn(0x7fbe40068960 0x7fbe4006ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:21.564 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.547+0000 7fbe4e7fc700 10 client.4321.objecter checking command 1 2023-09-30T02:06:21.564 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.547+0000 7fbe5e59c700 10 monclient: get_auth_request con 0x7fbe40068960 auth_method 0 2023-09-30T02:06:21.564 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.547+0000 7fbe4e7fc700 1 --2- 172.21.15.59:0/656608041 >> v2:172.21.15.59:6808/116665 conn(0x7fbe4006e8d0 0x7fbe40070ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:21.564 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.547+0000 7fbe4e7fc700 20 client.4321.objecter _get_session s=0x7fbe4006e220 osd=1 3 2023-09-30T02:06:21.564 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.547+0000 7fbe4e7fc700 20 client.4321.objecter put_session s=0x7fbe4006e220 osd=1 3 2023-09-30T02:06:21.564 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.547+0000 7fbe4e7fc700 15 client.4321.objecter _session_command_op_remove -1 1 2023-09-30T02:06:21.564 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.547+0000 7fbe4e7fc700 20 client.4321.objecter _get_session s=0x7fbe4006e220 osd=1 3 2023-09-30T02:06:21.564 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.547+0000 7fbe4e7fc700 20 client.4321.objecter get_session s=0x7fbe4006e220 osd=1 3 2023-09-30T02:06:21.564 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.547+0000 7fbe4e7fc700 15 client.4321.objecter _session_command_op_assign 1 1 2023-09-30T02:06:21.565 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.547+0000 7fbe4e7fc700 20 client.4321.objecter put_session s=0x7fbe4006e220 osd=1 4 2023-09-30T02:06:21.565 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.547+0000 7fbe4e7fc700 10 client.4321.objecter _send_command 1 2023-09-30T02:06:21.565 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.547+0000 7fbe4e7fc700 1 -- 172.21.15.59:0/656608041 --> v2:172.21.15.59:6808/116665 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbe40071350 con 0x7fbe4006e8d0 2023-09-30T02:06:21.565 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.547+0000 7fbe4e7fc700 20 client.4321.objecter dump_active .. 0 homeless 2023-09-30T02:06:21.565 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.547+0000 7fbe4e7fc700 1 -- 172.21.15.59:0/656608041 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbe58085250 con 0x7fbe60087810 2023-09-30T02:06:21.565 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.547+0000 7fbe4e7fc700 10 monclient: handle_get_version_reply finishing 1 version 25 2023-09-30T02:06:21.565 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.547+0000 7fbe4e7fc700 10 client.4321.objecter ms_handle_connect 0x7fbe40068960 2023-09-30T02:06:21.565 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.547+0000 7fbe5e59c700 1 --2- 172.21.15.59:0/656608041 >> v2:172.21.15.110:6800/115870 conn(0x7fbe40068960 0x7fbe4006ae10 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fbe54006fd0 tx=0x7fbe54008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:21.565 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.552+0000 7fbe5ed9d700 1 --2- 172.21.15.59:0/656608041 >> v2:172.21.15.59:6808/116665 conn(0x7fbe4006e8d0 0x7fbe40070ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:21.566 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.552+0000 7f28faffd700 1 -- 172.21.15.59:0/1269097232 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f290c01da20 con 0x7f291413e140 2023-09-30T02:06:21.566 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.552+0000 7f28faffd700 1 --2- 172.21.15.59:0/1269097232 >> v2:172.21.15.110:6800/115870 conn(0x7f2900068960 0x7f290006ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:21.566 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.553+0000 7f28faffd700 1 -- 172.21.15.59:0/1269097232 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f290c0860e0 con 0x7f291413e140 2023-09-30T02:06:21.566 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.553+0000 7f28faffd700 10 client.4318.objecter ms_dispatch 0x7f2914001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:06:21.566 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.553+0000 7f28faffd700 3 client.4318.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:06:21.566 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.553+0000 7f28faffd700 3 client.4318.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:06:21.566 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.553+0000 7f28faffd700 10 client.4318.objecter checking command 1 2023-09-30T02:06:21.567 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.553+0000 7f28faffd700 1 --2- 172.21.15.59:0/1269097232 >> v2:172.21.15.110:6805/116561 conn(0x7f290006e8d0 0x7f2900070ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:21.567 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.553+0000 7f28faffd700 20 client.4318.objecter _get_session s=0x7f290006e220 osd=7 3 2023-09-30T02:06:21.567 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.553+0000 7f28faffd700 20 client.4318.objecter put_session s=0x7f290006e220 osd=7 3 2023-09-30T02:06:21.567 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.553+0000 7f2912ffd700 1 --2- 172.21.15.59:0/1269097232 >> v2:172.21.15.110:6800/115870 conn(0x7f2900068960 0x7f290006ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:21.567 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.553+0000 7f28faffd700 15 client.4318.objecter _session_command_op_remove -1 1 2023-09-30T02:06:21.567 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.553+0000 7f28faffd700 20 client.4318.objecter _get_session s=0x7f290006e220 osd=7 3 2023-09-30T02:06:21.567 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.553+0000 7f28faffd700 20 client.4318.objecter get_session s=0x7f290006e220 osd=7 3 2023-09-30T02:06:21.567 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.553+0000 7f28faffd700 15 client.4318.objecter _session_command_op_assign 7 1 2023-09-30T02:06:21.567 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.553+0000 7f28faffd700 20 client.4318.objecter put_session s=0x7f290006e220 osd=7 4 2023-09-30T02:06:21.567 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.553+0000 7f28faffd700 10 client.4318.objecter _send_command 1 2023-09-30T02:06:21.567 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.553+0000 7f28faffd700 1 -- 172.21.15.59:0/1269097232 --> v2:172.21.15.110:6805/116561 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2900071350 con 0x7f290006e8d0 2023-09-30T02:06:21.568 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.553+0000 7f28faffd700 20 client.4318.objecter dump_active .. 0 homeless 2023-09-30T02:06:21.568 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.553+0000 7f28faffd700 1 -- 172.21.15.59:0/1269097232 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f290c0864b0 con 0x7f291413e140 2023-09-30T02:06:21.568 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.553+0000 7f28faffd700 10 monclient: handle_get_version_reply finishing 1 version 25 2023-09-30T02:06:21.568 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.553+0000 7f2912ffd700 10 monclient: get_auth_request con 0x7f2900068960 auth_method 0 2023-09-30T02:06:21.568 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.553+0000 7f28faffd700 10 client.4318.objecter ms_handle_connect 0x7f2900068960 2023-09-30T02:06:21.568 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.553+0000 7f2912ffd700 1 --2- 172.21.15.59:0/1269097232 >> v2:172.21.15.110:6800/115870 conn(0x7f2900068960 0x7f290006ae10 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f290800a8c0 tx=0x7f2908008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:21.568 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.554+0000 7f32e8ff9700 1 -- 172.21.15.59:0/802744325 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32dc0125f0 con 0x7f32ec0b13f0 2023-09-30T02:06:21.569 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.554+0000 7f32e8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:21.569 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.554+0000 7f32e8ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T02:06:21.569 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.554+0000 7f32e8ff9700 10 monclient: dump: 2023-09-30T02:06:21.569 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:21.569 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:21.569 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:21.569 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:21.569 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:21.569 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:21.569 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:21.569 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:21.570 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:21.570 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:21.570 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.554+0000 7f32e8ff9700 10 monclient: _finish_auth 0 2023-09-30T02:06:21.570 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.554+0000 7f32e8ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:21.570 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.554+0000 7f32e8ff9700 1 -- 172.21.15.59:0/802744325 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32dc012750 con 0x7f32ec0b13f0 2023-09-30T02:06:21.570 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.554+0000 7f32f314d700 5 monclient: authenticate success, global_id 4293 2023-09-30T02:06:21.570 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.554+0000 7f32f314d700 10 monclient: _renew_subs 2023-09-30T02:06:21.570 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.554+0000 7f32f314d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:06:21.570 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.554+0000 7f32f314d700 1 -- 172.21.15.59:0/802744325 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32ec11c430 con 0x7f32ec0b13f0 2023-09-30T02:06:21.570 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.554+0000 7f32e8ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:21.570 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.554+0000 7f32e8ff9700 1 -- 172.21.15.59:0/802744325 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32dc012af0 con 0x7f32ec0b13f0 2023-09-30T02:06:21.571 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.554+0000 7f32e8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:21.571 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.554+0000 7f32e8ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T02:06:21.571 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.554+0000 7f32e8ff9700 10 monclient: dump: 2023-09-30T02:06:21.571 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:21.571 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:21.571 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:21.571 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:21.571 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:21.571 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:21.571 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:21.571 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:21.572 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:21.572 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:21.572 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.554+0000 7f32f314d700 10 client.4293.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:06:21.572 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.554+0000 7f32f314d700 10 monclient: _renew_subs 2023-09-30T02:06:21.572 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.554+0000 7f32f314d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:06:21.572 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.554+0000 7f32f314d700 1 -- 172.21.15.59:0/802744325 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32ec11b490 con 0x7f32ec0b13f0 2023-09-30T02:06:21.572 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.554+0000 7f32f314d700 1 librados: init done 2023-09-30T02:06:21.572 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.554+0000 7f32eaffd700 4 set_mon_vals no callback set 2023-09-30T02:06:21.572 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.555+0000 7f32f314d700 10 client.4293.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:21.572 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.555+0000 7f32f314d700 15 client.4293.objecter _session_command_op_assign -1 1 2023-09-30T02:06:21.572 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.555+0000 7f32f314d700 20 client.4293.objecter _get_session osd=-1 returning homeless 2023-09-30T02:06:21.573 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.555+0000 7f32f314d700 10 client.4293.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:06:21.573 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.555+0000 7f32f314d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:06:21.573 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.555+0000 7f32f314d700 1 -- 172.21.15.59:0/802744325 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f32ec072b30 con 0x7f32ec0b13f0 2023-09-30T02:06:21.573 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.551+0000 7f43f3fff700 1 --2- 172.21.15.59:0/75758567 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f43f4142290 0x7f43f40b1500 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f43e80099b0 tx=0x7f43e8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ff7fffe65b10693 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:21.573 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.556+0000 7f29127fc700 1 --2- 172.21.15.59:0/1269097232 >> v2:172.21.15.110:6805/116561 conn(0x7f290006e8d0 0x7f2900070ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:21.574 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.556+0000 7f29127fc700 10 monclient: get_auth_request con 0x7f290006e8d0 auth_method 0 2023-09-30T02:06:21.574 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.557+0000 7fbe5ed9d700 10 monclient: get_auth_request con 0x7fbe4006e8d0 auth_method 0 2023-09-30T02:06:21.574 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.560+0000 7f28faffd700 10 client.4318.objecter ms_handle_connect 0x7f290006e8d0 2023-09-30T02:06:21.574 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.560+0000 7f29127fc700 1 --2- 172.21.15.59:0/1269097232 >> v2:172.21.15.110:6805/116561 conn(0x7f290006e8d0 0x7f2900070ce0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:21.575 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.566+0000 7fbe4e7fc700 10 client.4321.objecter ms_handle_connect 0x7fbe4006e8d0 2023-09-30T02:06:21.575 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.566+0000 7fbe5ed9d700 1 --2- 172.21.15.59:0/656608041 >> v2:172.21.15.59:6808/116665 conn(0x7fbe4006e8d0 0x7fbe40070ce0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:21.576 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.569+0000 7ff3367fc700 1 -- 172.21.15.59:0/723961325 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff33000c920 con 0x7ff34006ba90 2023-09-30T02:06:21.576 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.569+0000 7ff3367fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:21.576 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.569+0000 7ff3367fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:06:21.576 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.569+0000 7ff3367fc700 10 monclient: dump: 2023-09-30T02:06:21.576 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:21.576 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:21.576 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:21.577 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:21.577 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:21.577 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:21.577 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:21.577 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:21.577 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:21.577 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:21.577 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.569+0000 7ff3367fc700 10 monclient: _finish_auth 0 2023-09-30T02:06:21.577 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.569+0000 7ff3367fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:21.577 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.569+0000 7ff3367fc700 1 -- 172.21.15.59:0/723961325 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff33000ca80 con 0x7ff34006ba90 2023-09-30T02:06:21.577 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.569+0000 7ff3483d3700 5 monclient: authenticate success, global_id 4286 2023-09-30T02:06:21.578 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.569+0000 7ff3483d3700 10 monclient: _renew_subs 2023-09-30T02:06:21.578 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.569+0000 7ff3483d3700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:06:21.578 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.569+0000 7ff3483d3700 1 -- 172.21.15.59:0/723961325 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff340155950 con 0x7ff34006ba90 2023-09-30T02:06:21.578 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.569+0000 7ff3483d3700 10 client.4286.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:06:21.578 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.569+0000 7ff3483d3700 10 monclient: _renew_subs 2023-09-30T02:06:21.578 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.569+0000 7ff3483d3700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:06:21.578 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.569+0000 7ff3483d3700 1 -- 172.21.15.59:0/723961325 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff340155f10 con 0x7ff34006ba90 2023-09-30T02:06:21.578 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.569+0000 7ff3483d3700 1 librados: init done 2023-09-30T02:06:21.578 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.569+0000 7ff3367fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:21.578 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.569+0000 7ff3367fc700 1 -- 172.21.15.59:0/723961325 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff33000ce40 con 0x7ff34006ba90 2023-09-30T02:06:21.578 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.569+0000 7ff3367fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:21.578 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.569+0000 7ff3367fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:06:21.579 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.569+0000 7ff3367fc700 10 monclient: dump: 2023-09-30T02:06:21.579 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:21.579 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:21.579 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:21.579 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:21.579 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:21.579 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:21.579 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:21.579 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:21.579 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:21.579 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:21.580 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.570+0000 7ff34496c700 4 set_mon_vals no callback set 2023-09-30T02:06:21.580 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.570+0000 7ff3483d3700 10 client.4286.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:21.580 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.570+0000 7ff3483d3700 15 client.4286.objecter _session_command_op_assign -1 1 2023-09-30T02:06:21.580 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.570+0000 7ff3483d3700 20 client.4286.objecter _get_session osd=-1 returning homeless 2023-09-30T02:06:21.580 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.570+0000 7ff3483d3700 10 client.4286.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:06:21.580 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.570+0000 7ff3483d3700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:06:21.580 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.570+0000 7ff3483d3700 1 -- 172.21.15.59:0/723961325 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff3280028c0 con 0x7ff34006ba90 2023-09-30T02:06:21.580 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.575+0000 7f43f37fe700 1 -- 172.21.15.59:0/75758567 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43e800b820 con 0x7f43f4142290 2023-09-30T02:06:21.581 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.575+0000 7f43f37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:21.581 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.575+0000 7f43f37fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T02:06:21.581 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.575+0000 7f43f37fe700 10 monclient: dump: 2023-09-30T02:06:21.581 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:21.581 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:21.581 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:21.581 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:21.581 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:21.581 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:21.581 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:21.581 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:21.582 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:21.582 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:21.582 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.575+0000 7f43f37fe700 10 monclient: _finish_auth 0 2023-09-30T02:06:21.582 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.575+0000 7f43f37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:21.582 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.576+0000 7f43faed4700 5 monclient: authenticate success, global_id 4296 2023-09-30T02:06:21.582 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.576+0000 7f43faed4700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:06:21.593 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.592+0000 7f32e8ff9700 1 -- 172.21.15.59:0/802744325 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f32dc01e460 con 0x7f32ec0b13f0 2023-09-30T02:06:21.593 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.593+0000 7f32e8ff9700 1 --2- 172.21.15.59:0/802744325 >> v2:172.21.15.110:6800/115870 conn(0x7f32d8068960 0x7f32d806ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:21.593 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.593+0000 7f32f0ee9700 1 --2- 172.21.15.59:0/802744325 >> v2:172.21.15.110:6800/115870 conn(0x7f32d8068960 0x7f32d806ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:21.593 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.593+0000 7f32f0ee9700 10 monclient: get_auth_request con 0x7f32d8068960 auth_method 0 2023-09-30T02:06:21.593 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.594+0000 7f32e8ff9700 10 client.4293.objecter ms_handle_connect 0x7f32d8068960 2023-09-30T02:06:21.594 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.594+0000 7f32f0ee9700 1 --2- 172.21.15.59:0/802744325 >> v2:172.21.15.110:6800/115870 conn(0x7f32d8068960 0x7f32d806ae10 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f32e40020c0 tx=0x7f32e4001a10 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:21.595 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.596+0000 7f0f7e7fc700 1 -- 172.21.15.59:0/3803865518 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f0f7801a9c0 con 0x7f0f880ab4e0 2023-09-30T02:06:21.596 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.597+0000 7f0f7e7fc700 1 --2- 172.21.15.59:0/3803865518 >> v2:172.21.15.110:6800/115870 conn(0x7f0f6c068890 0x7f0f6c06ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:21.596 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.597+0000 7f0f8e2f8700 1 --2- 172.21.15.59:0/3803865518 >> v2:172.21.15.110:6800/115870 conn(0x7f0f6c068890 0x7f0f6c06ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:21.597 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.597+0000 7f0f8e2f8700 10 monclient: get_auth_request con 0x7f0f6c068890 auth_method 0 2023-09-30T02:06:21.597 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.597+0000 7f0f7e7fc700 10 client.4330.objecter ms_handle_connect 0x7f0f6c068890 2023-09-30T02:06:21.597 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.597+0000 7f0f8e2f8700 1 --2- 172.21.15.59:0/3803865518 >> v2:172.21.15.110:6800/115870 conn(0x7f0f6c068890 0x7f0f6c06ad40 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0f8000ad10 tx=0x7f0f80009370 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:21.604 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.604+0000 7f0f7e7fc700 1 -- 172.21.15.59:0/3803865518 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f0f78021070 con 0x7f0f880ab4e0 2023-09-30T02:06:21.604 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.604+0000 7f0f7e7fc700 10 client.4330.objecter ms_dispatch 0x7f0f88001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:06:21.604 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.604+0000 7f0f7e7fc700 3 client.4330.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:06:21.604 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.604+0000 7f0f7e7fc700 3 client.4330.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:06:21.604 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.604+0000 7f0f7e7fc700 10 client.4330.objecter checking command 1 2023-09-30T02:06:21.604 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.604+0000 7f0f7e7fc700 1 --2- 172.21.15.59:0/3803865518 >> v2:172.21.15.110:6813/116563 conn(0x7f0f6c06e690 0x7f0f6c070aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:21.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.604+0000 7f0f7e7fc700 20 client.4330.objecter _get_session s=0x7f0f7800bd10 osd=6 3 2023-09-30T02:06:21.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.604+0000 7f0f7e7fc700 20 client.4330.objecter put_session s=0x7f0f7800bd10 osd=6 3 2023-09-30T02:06:21.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.604+0000 7f0f7e7fc700 15 client.4330.objecter _session_command_op_remove -1 1 2023-09-30T02:06:21.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.604+0000 7f0f7e7fc700 20 client.4330.objecter _get_session s=0x7f0f7800bd10 osd=6 3 2023-09-30T02:06:21.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.604+0000 7f0f7e7fc700 20 client.4330.objecter get_session s=0x7f0f7800bd10 osd=6 3 2023-09-30T02:06:21.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.604+0000 7f0f7e7fc700 15 client.4330.objecter _session_command_op_assign 6 1 2023-09-30T02:06:21.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.604+0000 7f0f7e7fc700 20 client.4330.objecter put_session s=0x7f0f7800bd10 osd=6 4 2023-09-30T02:06:21.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.604+0000 7f0f7e7fc700 10 client.4330.objecter _send_command 1 2023-09-30T02:06:21.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.604+0000 7f0f7e7fc700 1 -- 172.21.15.59:0/3803865518 --> v2:172.21.15.110:6813/116563 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0f6c071110 con 0x7f0f6c06e690 2023-09-30T02:06:21.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.604+0000 7f0f7e7fc700 20 client.4330.objecter dump_active .. 0 homeless 2023-09-30T02:06:21.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.604+0000 7f0f7e7fc700 1 -- 172.21.15.59:0/3803865518 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0f78053940 con 0x7f0f880ab4e0 2023-09-30T02:06:21.606 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.604+0000 7f0f7e7fc700 10 monclient: handle_get_version_reply finishing 1 version 25 2023-09-30T02:06:21.606 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.605+0000 7f0f8eaf9700 1 --2- 172.21.15.59:0/3803865518 >> v2:172.21.15.110:6813/116563 conn(0x7f0f6c06e690 0x7f0f6c070aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:21.606 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.605+0000 7f0f8eaf9700 10 monclient: get_auth_request con 0x7f0f6c06e690 auth_method 0 2023-09-30T02:06:21.607 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.605+0000 7f43f37fe700 1 -- 172.21.15.59:0/75758567 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43e800b980 con 0x7f43f4142290 2023-09-30T02:06:21.607 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.605+0000 7f43f37fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:21.607 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.605+0000 7f43f37fe700 1 -- 172.21.15.59:0/75758567 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43e800bd20 con 0x7f43f4142290 2023-09-30T02:06:21.607 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.605+0000 7f43f37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:21.607 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.605+0000 7f43f37fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T02:06:21.607 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.605+0000 7f43f37fe700 10 monclient: dump: 2023-09-30T02:06:21.607 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:21.607 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:21.608 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:21.608 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:21.608 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:21.608 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:21.608 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:21.608 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:21.608 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:21.608 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:21.608 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.606+0000 7f43faed4700 10 monclient: get_monmap_and_config success 2023-09-30T02:06:21.608 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.606+0000 7f43faed4700 4 set_mon_vals no callback set 2023-09-30T02:06:21.608 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.606+0000 7f43faed4700 10 monclient: shutdown 2023-09-30T02:06:21.608 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.606+0000 7f43faed4700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:21.609 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.606+0000 7f43faed4700 1 -- 172.21.15.59:0/75758567 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f43f4142290 msgr2=0x7f43f40b1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:21.609 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.606+0000 7f43faed4700 1 --2- 172.21.15.59:0/75758567 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f43f4142290 0x7f43f40b1500 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f43e80099b0 tx=0x7f43e8009250 comp rx=0 tx=0).stop 2023-09-30T02:06:21.609 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.608+0000 7f43faed4700 1 -- 172.21.15.59:0/75758567 shutdown_connections 2023-09-30T02:06:21.609 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.608+0000 7f43faed4700 1 --2- 172.21.15.59:0/75758567 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f43f40a5f40 0x7f43f413af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:21.609 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.608+0000 7f43faed4700 1 --2- 172.21.15.59:0/75758567 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f43f4142290 0x7f43f40b1500 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:21.609 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.608+0000 7f43faed4700 1 -- 172.21.15.59:0/75758567 >> 172.21.15.59:0/75758567 conn(0x7f43f40631f0 msgr2=0x7f43f40635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:21.610 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.609+0000 7f0f8eaf9700 1 --2- 172.21.15.59:0/3803865518 >> v2:172.21.15.110:6813/116563 conn(0x7f0f6c06e690 0x7f0f6c070aa0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:21.610 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.609+0000 7f0f7e7fc700 10 client.4330.objecter ms_handle_connect 0x7f0f6c06e690 2023-09-30T02:06:21.610 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.608+0000 7f32e8ff9700 1 -- 172.21.15.59:0/802744325 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f32dc056aa0 con 0x7f32ec0b13f0 2023-09-30T02:06:21.610 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.608+0000 7f32e8ff9700 10 client.4293.objecter ms_dispatch 0x7f32ec001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:06:21.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.608+0000 7f32e8ff9700 3 client.4293.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:06:21.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.608+0000 7f32e8ff9700 3 client.4293.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:06:21.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.608+0000 7f32e8ff9700 10 client.4293.objecter checking command 1 2023-09-30T02:06:21.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.608+0000 7f32e8ff9700 1 --2- 172.21.15.59:0/802744325 >> v2:172.21.15.110:6801/116560 conn(0x7f32d806e8d0 0x7f32d8070ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:21.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.608+0000 7f32e8ff9700 20 client.4293.objecter _get_session s=0x7f32d806e220 osd=4 3 2023-09-30T02:06:21.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.608+0000 7f32e8ff9700 20 client.4293.objecter put_session s=0x7f32d806e220 osd=4 3 2023-09-30T02:06:21.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.608+0000 7f32e8ff9700 15 client.4293.objecter _session_command_op_remove -1 1 2023-09-30T02:06:21.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.608+0000 7f32e8ff9700 20 client.4293.objecter _get_session s=0x7f32d806e220 osd=4 3 2023-09-30T02:06:21.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.608+0000 7f32e8ff9700 20 client.4293.objecter get_session s=0x7f32d806e220 osd=4 3 2023-09-30T02:06:21.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.608+0000 7f32e8ff9700 15 client.4293.objecter _session_command_op_assign 4 1 2023-09-30T02:06:21.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.608+0000 7f32e8ff9700 20 client.4293.objecter put_session s=0x7f32d806e220 osd=4 4 2023-09-30T02:06:21.612 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.608+0000 7f32e8ff9700 10 client.4293.objecter _send_command 1 2023-09-30T02:06:21.612 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.608+0000 7f32e8ff9700 1 -- 172.21.15.59:0/802744325 --> v2:172.21.15.110:6801/116560 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f32d8071350 con 0x7f32d806e8d0 2023-09-30T02:06:21.612 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.608+0000 7f32e8ff9700 20 client.4293.objecter dump_active .. 0 homeless 2023-09-30T02:06:21.612 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.608+0000 7f32e8ff9700 1 -- 172.21.15.59:0/802744325 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f32dc0526e0 con 0x7f32ec0b13f0 2023-09-30T02:06:21.612 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.608+0000 7f32e8ff9700 10 monclient: handle_get_version_reply finishing 1 version 25 2023-09-30T02:06:21.612 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.610+0000 7f32f16ea700 1 --2- 172.21.15.59:0/802744325 >> v2:172.21.15.110:6801/116560 conn(0x7f32d806e8d0 0x7f32d8070ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:21.612 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.610+0000 7fa044ff9700 1 -- 172.21.15.59:0/2373407985 <== osd.3 v2:172.21.15.59:6812/116667 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7fa028071280 con 0x7fa02806e800 2023-09-30T02:06:21.612 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.610+0000 7fa044ff9700 10 client.4290.objecter ms_dispatch 0x7fa048001550 command_reply(tid 1: 0 ) v1 2023-09-30T02:06:21.613 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.610+0000 7fa044ff9700 10 client.4290.objecter _finish_command 1 = system:0 2023-09-30T02:06:21.613 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.610+0000 7fa044ff9700 20 client.4290.objecter put_session s=0x7fa02806e150 osd=3 4 2023-09-30T02:06:21.613 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.610+0000 7fa044ff9700 15 client.4290.objecter _session_command_op_remove 3 1 2023-09-30T02:06:21.613 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.613+0000 7f32f16ea700 10 monclient: get_auth_request con 0x7f32d806e8d0 auth_method 0 2023-09-30T02:06:21.614 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.610+0000 7f43faed4700 1 -- 172.21.15.59:0/75758567 shutdown_connections 2023-09-30T02:06:21.614 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.610+0000 7f43faed4700 1 -- 172.21.15.59:0/75758567 wait complete. 2023-09-30T02:06:21.614 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.611+0000 7f43faed4700 10 monclient: build_initial_monmap 2023-09-30T02:06:21.614 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.611+0000 7f43faed4700 10 monclient: monmap: 2023-09-30T02:06:21.614 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:21.614 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:06:21.614 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:06:21.614 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:06:21.614 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:21.615 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:06:21.615 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:06:21.615 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:06:21.615 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:06:21.615 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:21.615 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.611+0000 7f43faed4700 1 librados: starting msgr at 2023-09-30T02:06:21.615 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.611+0000 7f43faed4700 1 librados: starting objecter 2023-09-30T02:06:21.615 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.611+0000 7f43faed4700 1 Processor -- start 2023-09-30T02:06:21.615 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.612+0000 7f43faed4700 1 -- start start 2023-09-30T02:06:21.615 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.612+0000 7f43faed4700 1 librados: setting wanted keys 2023-09-30T02:06:21.615 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.612+0000 7f43faed4700 1 librados: calling monclient init 2023-09-30T02:06:21.616 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.612+0000 7f43faed4700 10 monclient: init 2023-09-30T02:06:21.616 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.613+0000 7f43faed4700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:21.616 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.613+0000 7f43faed4700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T02:06:21.616 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.613+0000 7f43faed4700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f43f40a5b70 0x7f43f4135770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:21.616 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.613+0000 7f43faed4700 10 monclient(hunting): picked mon.noname-c con 0x7f43f40a5b70 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:21.616 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.613+0000 7f43faed4700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f43f40a5f40 0x7f43f406b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:21.616 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.613+0000 7f43faed4700 10 monclient(hunting): picked mon.noname-a con 0x7f43f40a5f40 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:06:21.616 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.613+0000 7f43faed4700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f43f4142290 0x7f43f416f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:21.616 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.613+0000 7f43faed4700 10 monclient(hunting): picked mon.noname-b con 0x7f43f4142290 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:06:21.616 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.613+0000 7f43faed4700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:21.616 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.613+0000 7f43faed4700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43f406acb0 con 0x7f43f40a5f40 2023-09-30T02:06:21.616 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.613+0000 7f43faed4700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:21.617 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.613+0000 7f43faed4700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43f406cc40 con 0x7f43f4142290 2023-09-30T02:06:21.617 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.613+0000 7f43faed4700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:21.617 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.613+0000 7f43faed4700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43f406a940 con 0x7f43f40a5b70 2023-09-30T02:06:21.617 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.613+0000 7f43faed4700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:21.617 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.613+0000 7f43faed4700 10 monclient(hunting): authenticate will time out at 1826.915150s 2023-09-30T02:06:21.617 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.614+0000 7f43f9471700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f43f4142290 0x7f43f416f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:21.617 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.614+0000 7f43f9471700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f43f4142290 0x7f43f416f180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:38982/0 (socket says 172.21.15.59:38982) 2023-09-30T02:06:21.617 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.614+0000 7f43f9471700 1 -- 172.21.15.59:0/1297509251 learned_addr learned my addr 172.21.15.59:0/1297509251 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:06:21.617 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.614+0000 7f43f9471700 10 monclient(hunting): get_auth_request con 0x7f43f4142290 auth_method 0 2023-09-30T02:06:21.617 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.614+0000 7f43f9471700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:21.617 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.614+0000 7f43f9471700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:21.617 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.614+0000 7f43f9471700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:21.618 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.615+0000 7f43f9471700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:21.618 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.615+0000 7f43f9471700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:21.618 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.615+0000 7f43f9471700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:21.619 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.615+0000 7ff3367fc700 1 -- 172.21.15.59:0/723961325 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7ff330013680 con 0x7ff34006ba90 2023-09-30T02:06:21.619 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.615+0000 7ff3367fc700 1 --2- 172.21.15.59:0/723961325 >> v2:172.21.15.110:6800/115870 conn(0x7ff324068910 0x7ff32406adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:21.619 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.615+0000 7ff3367fc700 1 -- 172.21.15.59:0/723961325 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7ff3300844f0 con 0x7ff34006ba90 2023-09-30T02:06:21.619 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.615+0000 7ff3367fc700 10 client.4286.objecter ms_dispatch 0x7ff340001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:06:21.619 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.615+0000 7ff3367fc700 3 client.4286.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:06:21.619 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.615+0000 7ff3367fc700 3 client.4286.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:06:21.619 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.615+0000 7ff34616f700 1 --2- 172.21.15.59:0/723961325 >> v2:172.21.15.110:6800/115870 conn(0x7ff324068910 0x7ff32406adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:21.619 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.615+0000 7ff34616f700 10 monclient: get_auth_request con 0x7ff324068910 auth_method 0 2023-09-30T02:06:21.619 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.616+0000 7ff3367fc700 10 client.4286.objecter checking command 1 2023-09-30T02:06:21.620 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.616+0000 7ff3367fc700 1 --2- 172.21.15.59:0/723961325 >> v2:172.21.15.59:6800/116664 conn(0x7ff32406e880 0x7ff324070c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:21.620 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.616+0000 7ff3367fc700 20 client.4286.objecter _get_session s=0x7ff32406e1d0 osd=0 3 2023-09-30T02:06:21.620 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.616+0000 7ff3367fc700 20 client.4286.objecter put_session s=0x7ff32406e1d0 osd=0 3 2023-09-30T02:06:21.620 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.616+0000 7ff3367fc700 15 client.4286.objecter _session_command_op_remove -1 1 2023-09-30T02:06:21.620 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.616+0000 7ff3367fc700 20 client.4286.objecter _get_session s=0x7ff32406e1d0 osd=0 3 2023-09-30T02:06:21.620 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.616+0000 7ff3367fc700 20 client.4286.objecter get_session s=0x7ff32406e1d0 osd=0 3 2023-09-30T02:06:21.620 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.616+0000 7ff3367fc700 15 client.4286.objecter _session_command_op_assign 0 1 2023-09-30T02:06:21.620 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.616+0000 7ff3367fc700 20 client.4286.objecter put_session s=0x7ff32406e1d0 osd=0 4 2023-09-30T02:06:21.620 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.616+0000 7ff3367fc700 10 client.4286.objecter _send_command 1 2023-09-30T02:06:21.620 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.616+0000 7ff3367fc700 1 -- 172.21.15.59:0/723961325 --> v2:172.21.15.59:6800/116664 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff324071300 con 0x7ff32406e880 2023-09-30T02:06:21.620 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.616+0000 7ff3367fc700 20 client.4286.objecter dump_active .. 0 homeless 2023-09-30T02:06:21.620 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.616+0000 7ff3367fc700 1 -- 172.21.15.59:0/723961325 <== mon.2 v2:172.21.15.59:3301/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff330084820 con 0x7ff34006ba90 2023-09-30T02:06:21.621 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.616+0000 7ff3367fc700 10 monclient: handle_get_version_reply finishing 1 version 25 2023-09-30T02:06:21.621 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.616+0000 7ff346970700 1 --2- 172.21.15.59:0/723961325 >> v2:172.21.15.59:6800/116664 conn(0x7ff32406e880 0x7ff324070c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:21.621 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.618+0000 7ff346970700 10 monclient: get_auth_request con 0x7ff32406e880 auth_method 0 2023-09-30T02:06:21.621 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.618+0000 7f43f8c70700 1 --2- 172.21.15.59:0/1297509251 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f43f40a5b70 0x7f43f4135770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:21.621 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.618+0000 7f43f8c70700 10 monclient(hunting): get_auth_request con 0x7f43f40a5b70 auth_method 0 2023-09-30T02:06:21.621 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.618+0000 7f43f8c70700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:21.621 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.618+0000 7f43f8c70700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:21.621 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.618+0000 7f43f8c70700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:21.622 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.617+0000 7f32f16ea700 1 --2- 172.21.15.59:0/802744325 >> v2:172.21.15.110:6801/116560 conn(0x7f32d806e8d0 0x7f32d8070ce0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:21.622 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.620+0000 7f43f8c70700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:21.622 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.620+0000 7f43f8c70700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:21.622 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.620+0000 7f43f8c70700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:21.622 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.620+0000 7f43f9471700 10 monclient(hunting): handle_auth_done global_id 4333 payload 824 2023-09-30T02:06:21.622 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.620+0000 7f43f9471700 1 -- 172.21.15.59:0/1297509251 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f43f40a5b70 msgr2=0x7f43f4135770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:21.622 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.620+0000 7f43f9471700 1 --2- 172.21.15.59:0/1297509251 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f43f40a5b70 0x7f43f4135770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:21.623 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.620+0000 7f43f9471700 1 -- 172.21.15.59:0/1297509251 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f43f40a5f40 msgr2=0x7f43f406b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:21.623 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.620+0000 7f43f9471700 1 --2- 172.21.15.59:0/1297509251 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f43f40a5f40 0x7f43f406b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:21.623 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.620+0000 7f43f9471700 10 monclient: _finish_hunting 0 2023-09-30T02:06:21.623 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.620+0000 7f43f9471700 1 monclient: found mon.noname-b 2023-09-30T02:06:21.623 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.620+0000 7f43f9471700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:21.623 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.620+0000 7f43f9471700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:06:21.623 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.620+0000 7f43f9471700 1 -- 172.21.15.59:0/1297509251 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43e80094f0 con 0x7f43f4142290 2023-09-30T02:06:21.623 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.623+0000 7f32e8ff9700 10 client.4293.objecter ms_handle_connect 0x7f32d806e8d0 2023-09-30T02:06:21.624 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.623+0000 7ff34616f700 1 --2- 172.21.15.59:0/723961325 >> v2:172.21.15.110:6800/115870 conn(0x7ff324068910 0x7ff32406adc0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff33c0058d0 tx=0x7ff33c0079d0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:21.624 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.624+0000 7f43f9471700 1 --2- 172.21.15.59:0/1297509251 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f43f4142290 0x7f43f416f180 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f43ec00bc30 tx=0x7f43ec009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:21.624 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.624+0000 7f43f0ff9700 10 client.?.objecter ms_handle_connect 0x7f43f4142290 2023-09-30T02:06:21.624 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.624+0000 7f43f0ff9700 10 client.?.objecter resend_mon_ops 2023-09-30T02:06:21.624 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.625+0000 7ff346970700 1 --2- 172.21.15.59:0/723961325 >> v2:172.21.15.59:6800/116664 conn(0x7ff32406e880 0x7ff324070c90 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:21.626 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.627+0000 7f9cfaffd700 1 -- 172.21.15.59:0/3859569758 <== osd.2 v2:172.21.15.59:6804/116666 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f9d00071350 con 0x7f9d0006e8d0 2023-09-30T02:06:21.626 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.627+0000 7f9cfaffd700 10 client.4280.objecter ms_dispatch 0x7f9d14001550 command_reply(tid 1: 0 ) v1 2023-09-30T02:06:21.627 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.627+0000 7f9cfaffd700 10 client.4280.objecter _finish_command 1 = system:0 2023-09-30T02:06:21.627 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.627+0000 7f9cfaffd700 20 client.4280.objecter put_session s=0x7f9d0006e220 osd=2 4 2023-09-30T02:06:21.627 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.627+0000 7f9cfaffd700 15 client.4280.objecter _session_command_op_remove 2 1 2023-09-30T02:06:21.628 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.627+0000 7f43f0ff9700 1 -- 172.21.15.59:0/1297509251 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43ec0092e0 con 0x7f43f4142290 2023-09-30T02:06:21.628 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.627+0000 7f43f0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:21.628 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.627+0000 7f43f0ff9700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:06:21.628 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.627+0000 7f43f0ff9700 10 monclient: dump: 2023-09-30T02:06:21.628 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:21.628 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:21.628 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:21.628 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:21.628 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:21.628 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:21.629 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:21.629 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:21.629 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:21.629 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:21.629 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.627+0000 7f43f0ff9700 10 monclient: _finish_auth 0 2023-09-30T02:06:21.629 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.627+0000 7f43f0ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:21.629 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.627+0000 7f43f0ff9700 1 -- 172.21.15.59:0/1297509251 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43ec00c6f0 con 0x7f43f4142290 2023-09-30T02:06:21.629 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.627+0000 7f43f0ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:21.629 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.627+0000 7f43f0ff9700 1 -- 172.21.15.59:0/1297509251 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43ec00cad0 con 0x7f43f4142290 2023-09-30T02:06:21.629 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.627+0000 7f43f0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:21.629 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.627+0000 7f43f0ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:06:21.629 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.627+0000 7f43f0ff9700 10 monclient: dump: 2023-09-30T02:06:21.630 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:21.630 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:21.630 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:21.630 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:21.630 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:21.630 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:21.630 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:21.630 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:21.630 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:21.630 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:21.630 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.627+0000 7f43f2ffd700 4 set_mon_vals no callback set 2023-09-30T02:06:21.630 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.628+0000 7f43faed4700 5 monclient: authenticate success, global_id 4333 2023-09-30T02:06:21.631 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.628+0000 7f43faed4700 10 monclient: _renew_subs 2023-09-30T02:06:21.631 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.628+0000 7f43faed4700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:21.631 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.628+0000 7f43faed4700 1 -- 172.21.15.59:0/1297509251 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43f413e320 con 0x7f43f4142290 2023-09-30T02:06:21.631 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.628+0000 7f43faed4700 10 client.4333.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:06:21.631 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.628+0000 7f43faed4700 10 monclient: _renew_subs 2023-09-30T02:06:21.631 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.628+0000 7f43faed4700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:21.631 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.628+0000 7f43faed4700 1 -- 172.21.15.59:0/1297509251 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43f413c4e0 con 0x7f43f4142290 2023-09-30T02:06:21.631 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.628+0000 7f43faed4700 1 librados: init done 2023-09-30T02:06:21.632 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.630+0000 7ff3367fc700 10 client.4286.objecter ms_handle_connect 0x7ff324068910 2023-09-30T02:06:21.632 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.630+0000 7ff3367fc700 10 client.4286.objecter ms_handle_connect 0x7ff32406e880 2023-09-30T02:06:21.632 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.631+0000 7f28faffd700 1 -- 172.21.15.59:0/1269097232 <== osd.7 v2:172.21.15.110:6805/116561 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f2900071350 con 0x7f290006e8d0 2023-09-30T02:06:21.632 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.631+0000 7f28faffd700 10 client.4318.objecter ms_dispatch 0x7f2914001550 command_reply(tid 1: 0 ) v1 2023-09-30T02:06:21.632 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.631+0000 7f28faffd700 10 client.4318.objecter _finish_command 1 = system:0 2023-09-30T02:06:21.632 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.631+0000 7f28faffd700 20 client.4318.objecter put_session s=0x7f290006e220 osd=7 4 2023-09-30T02:06:21.632 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.631+0000 7f28faffd700 15 client.4318.objecter _session_command_op_remove 7 1 2023-09-30T02:06:21.632 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.633+0000 7f43faed4700 10 client.4333.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:21.633 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.633+0000 7f43faed4700 15 client.4333.objecter _session_command_op_assign -1 1 2023-09-30T02:06:21.633 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.633+0000 7f43faed4700 20 client.4333.objecter _get_session osd=-1 returning homeless 2023-09-30T02:06:21.633 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.633+0000 7f43faed4700 10 client.4333.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:06:21.633 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.633+0000 7f43faed4700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:21.633 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.633+0000 7f43faed4700 1 -- 172.21.15.59:0/1297509251 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f43f406cad0 con 0x7f43f4142290 2023-09-30T02:06:21.641 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.641+0000 7fa04e664700 10 client.4290.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-09-30T02:06:21.641 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.641+0000 7fa04e664700 15 client.4290.objecter _session_command_op_assign -1 2 2023-09-30T02:06:21.641 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.641+0000 7fa04e664700 20 client.4290.objecter _get_session s=0x7fa02806e150 osd=3 3 2023-09-30T02:06:21.641 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.641+0000 7fa04e664700 20 client.4290.objecter put_session s=0x7fa02806e150 osd=3 3 2023-09-30T02:06:21.641 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.641+0000 7fa04e664700 20 client.4290.objecter _get_session s=0x7fa02806e150 osd=3 3 2023-09-30T02:06:21.641 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.641+0000 7fa04e664700 15 client.4290.objecter _session_command_op_remove -1 2 2023-09-30T02:06:21.641 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.641+0000 7fa04e664700 20 client.4290.objecter get_session s=0x7fa02806e150 osd=3 3 2023-09-30T02:06:21.641 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.641+0000 7fa04e664700 15 client.4290.objecter _session_command_op_assign 3 2 2023-09-30T02:06:21.642 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.641+0000 7fa04e664700 20 client.4290.objecter put_session s=0x7fa02806e150 osd=3 4 2023-09-30T02:06:21.642 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.641+0000 7fa04e664700 10 client.4290.objecter _send_command 2 2023-09-30T02:06:21.642 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.641+0000 7fa04e664700 1 -- 172.21.15.59:0/2373407985 --> v2:172.21.15.59:6812/116667 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa048139af0 con 0x7fa02806e800 2023-09-30T02:06:21.653 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.653+0000 7fbe4e7fc700 1 -- 172.21.15.59:0/656608041 <== osd.1 v2:172.21.15.59:6808/116665 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7fbe40071350 con 0x7fbe4006e8d0 2023-09-30T02:06:21.653 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.654+0000 7fbe4e7fc700 10 client.4321.objecter ms_dispatch 0x7fbe60001550 command_reply(tid 1: 0 ) v1 2023-09-30T02:06:21.653 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.654+0000 7fbe4e7fc700 10 client.4321.objecter _finish_command 1 = system:0 2023-09-30T02:06:21.653 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.654+0000 7fbe4e7fc700 20 client.4321.objecter put_session s=0x7fbe4006e220 osd=1 4 2023-09-30T02:06:21.653 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.654+0000 7fbe4e7fc700 15 client.4321.objecter _session_command_op_remove 1 1 2023-09-30T02:06:21.657 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.657+0000 7f43f0ff9700 1 -- 172.21.15.59:0/1297509251 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f43ec012070 con 0x7f43f4142290 2023-09-30T02:06:21.657 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.657+0000 7f43f0ff9700 1 --2- 172.21.15.59:0/1297509251 >> v2:172.21.15.110:6800/115870 conn(0x7f43e0068960 0x7f43e006ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:21.657 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.657+0000 7f43f0ff9700 1 -- 172.21.15.59:0/1297509251 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f43ec083d10 con 0x7f43f4142290 2023-09-30T02:06:21.657 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.657+0000 7f43f0ff9700 10 client.4333.objecter ms_dispatch 0x7f43f4001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:06:21.657 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.657+0000 7f43f0ff9700 3 client.4333.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:06:21.657 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.657+0000 7f43f0ff9700 3 client.4333.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:06:21.658 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.658+0000 7f43f0ff9700 10 client.4333.objecter checking command 1 2023-09-30T02:06:21.658 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.658+0000 7f43f0ff9700 1 --2- 172.21.15.59:0/1297509251 >> v2:172.21.15.110:6809/116562 conn(0x7f43e006e8d0 0x7f43e0070ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:21.658 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.658+0000 7f43f0ff9700 20 client.4333.objecter _get_session s=0x7f43e006e220 osd=5 3 2023-09-30T02:06:21.658 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.658+0000 7f43f0ff9700 20 client.4333.objecter put_session s=0x7f43e006e220 osd=5 3 2023-09-30T02:06:21.658 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.658+0000 7f43f0ff9700 15 client.4333.objecter _session_command_op_remove -1 1 2023-09-30T02:06:21.658 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.658+0000 7f43f0ff9700 20 client.4333.objecter _get_session s=0x7f43e006e220 osd=5 3 2023-09-30T02:06:21.658 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.658+0000 7f43f0ff9700 20 client.4333.objecter get_session s=0x7f43e006e220 osd=5 3 2023-09-30T02:06:21.658 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.658+0000 7f43f0ff9700 15 client.4333.objecter _session_command_op_assign 5 1 2023-09-30T02:06:21.658 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.658+0000 7f43f0ff9700 20 client.4333.objecter put_session s=0x7f43e006e220 osd=5 4 2023-09-30T02:06:21.658 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.658+0000 7f43f0ff9700 10 client.4333.objecter _send_command 1 2023-09-30T02:06:21.658 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.658+0000 7f43f0ff9700 1 -- 172.21.15.59:0/1297509251 --> v2:172.21.15.110:6809/116562 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f43e0071350 con 0x7f43e006e8d0 2023-09-30T02:06:21.658 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.658+0000 7f43f0ff9700 20 client.4333.objecter dump_active .. 0 homeless 2023-09-30T02:06:21.659 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.660+0000 7f43f0ff9700 1 -- 172.21.15.59:0/1297509251 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f43ec00ce80 con 0x7f43f4142290 2023-09-30T02:06:21.659 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.660+0000 7f43f0ff9700 10 monclient: handle_get_version_reply finishing 1 version 25 2023-09-30T02:06:21.659 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.660+0000 7f43f8c70700 1 --2- 172.21.15.59:0/1297509251 >> v2:172.21.15.110:6800/115870 conn(0x7f43e0068960 0x7f43e006ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:21.659 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.660+0000 7f43f8c70700 10 monclient: get_auth_request con 0x7f43e0068960 auth_method 0 2023-09-30T02:06:21.660 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.660+0000 7f9d199df700 10 client.4280.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-09-30T02:06:21.660 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.660+0000 7f9d199df700 15 client.4280.objecter _session_command_op_assign -1 2 2023-09-30T02:06:21.660 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.660+0000 7f9d199df700 20 client.4280.objecter _get_session s=0x7f9d0006e220 osd=2 3 2023-09-30T02:06:21.660 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.660+0000 7f9d199df700 20 client.4280.objecter put_session s=0x7f9d0006e220 osd=2 3 2023-09-30T02:06:21.660 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.660+0000 7f9d199df700 20 client.4280.objecter _get_session s=0x7f9d0006e220 osd=2 3 2023-09-30T02:06:21.660 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.660+0000 7f9d199df700 15 client.4280.objecter _session_command_op_remove -1 2 2023-09-30T02:06:21.660 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.660+0000 7f9d199df700 20 client.4280.objecter get_session s=0x7f9d0006e220 osd=2 3 2023-09-30T02:06:21.660 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.660+0000 7f9d199df700 15 client.4280.objecter _session_command_op_assign 2 2 2023-09-30T02:06:21.661 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.660+0000 7f9d199df700 20 client.4280.objecter put_session s=0x7f9d0006e220 osd=2 4 2023-09-30T02:06:21.661 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.660+0000 7f9d199df700 10 client.4280.objecter _send_command 2 2023-09-30T02:06:21.661 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.660+0000 7f9d199df700 1 -- 172.21.15.59:0/3859569758 --> v2:172.21.15.59:6804/116666 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f9d141520b0 con 0x7f9d0006e8d0 2023-09-30T02:06:21.661 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.660+0000 7f0f7e7fc700 1 -- 172.21.15.59:0/3803865518 <== osd.6 v2:172.21.15.110:6813/116563 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f0f6c071110 con 0x7f0f6c06e690 2023-09-30T02:06:21.661 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.660+0000 7f0f7e7fc700 10 client.4330.objecter ms_dispatch 0x7f0f88001550 command_reply(tid 1: 0 ) v1 2023-09-30T02:06:21.661 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.660+0000 7f0f7e7fc700 10 client.4330.objecter _finish_command 1 = system:0 2023-09-30T02:06:21.661 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.660+0000 7f0f7e7fc700 20 client.4330.objecter put_session s=0x7f0f7800bd10 osd=6 4 2023-09-30T02:06:21.661 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.660+0000 7f0f7e7fc700 15 client.4330.objecter _session_command_op_remove 6 1 2023-09-30T02:06:21.662 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.660+0000 7f43f3fff700 1 --2- 172.21.15.59:0/1297509251 >> v2:172.21.15.110:6809/116562 conn(0x7f43e006e8d0 0x7f43e0070ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:21.662 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.660+0000 7f43f0ff9700 10 client.4333.objecter ms_handle_connect 0x7f43e0068960 2023-09-30T02:06:21.662 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.660+0000 7f43f8c70700 1 --2- 172.21.15.59:0/1297509251 >> v2:172.21.15.110:6800/115870 conn(0x7f43e0068960 0x7f43e006ae10 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f43e4006fd0 tx=0x7f43e40090f0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:21.662 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.661+0000 7f43f3fff700 10 monclient: get_auth_request con 0x7f43e006e8d0 auth_method 0 2023-09-30T02:06:21.666 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.666+0000 7f291972e700 10 client.4318.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-09-30T02:06:21.666 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.666+0000 7f291972e700 15 client.4318.objecter _session_command_op_assign -1 2 2023-09-30T02:06:21.666 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.666+0000 7f291972e700 20 client.4318.objecter _get_session s=0x7f290006e220 osd=7 3 2023-09-30T02:06:21.666 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.666+0000 7f291972e700 20 client.4318.objecter put_session s=0x7f290006e220 osd=7 3 2023-09-30T02:06:21.666 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.666+0000 7f291972e700 20 client.4318.objecter _get_session s=0x7f290006e220 osd=7 3 2023-09-30T02:06:21.666 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.666+0000 7f291972e700 15 client.4318.objecter _session_command_op_remove -1 2 2023-09-30T02:06:21.666 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.666+0000 7f291972e700 20 client.4318.objecter get_session s=0x7f290006e220 osd=7 3 2023-09-30T02:06:21.666 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.666+0000 7f291972e700 15 client.4318.objecter _session_command_op_assign 7 2 2023-09-30T02:06:21.666 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.666+0000 7f291972e700 20 client.4318.objecter put_session s=0x7f290006e220 osd=7 4 2023-09-30T02:06:21.666 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.666+0000 7f291972e700 10 client.4318.objecter _send_command 2 2023-09-30T02:06:21.667 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.666+0000 7f291972e700 1 -- 172.21.15.59:0/1269097232 --> v2:172.21.15.110:6805/116561 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f29140a9de0 con 0x7f290006e8d0 2023-09-30T02:06:21.667 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.668+0000 7f43f3fff700 1 --2- 172.21.15.59:0/1297509251 >> v2:172.21.15.110:6809/116562 conn(0x7f43e006e8d0 0x7f43e0070ce0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:21.667 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.668+0000 7f43f0ff9700 10 client.4333.objecter ms_handle_connect 0x7f43e006e8d0 2023-09-30T02:06:21.674 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.675+0000 7f9cfaffd700 1 -- 172.21.15.59:0/3859569758 <== osd.2 v2:172.21.15.59:6804/116666 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f9d141520b0 con 0x7f9d0006e8d0 2023-09-30T02:06:21.674 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.675+0000 7f9cfaffd700 10 client.4280.objecter ms_dispatch 0x7f9d14001550 command_reply(tid 2: 0 ) v1 2023-09-30T02:06:21.674 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.675+0000 7f9cfaffd700 10 client.4280.objecter _finish_command 2 = system:0 2023-09-30T02:06:21.674 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.675+0000 7f9cfaffd700 20 client.4280.objecter put_session s=0x7f9d0006e220 osd=2 4 2023-09-30T02:06:21.674 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.675+0000 7f9cfaffd700 15 client.4280.objecter _session_command_op_remove 2 2 2023-09-30T02:06:21.675 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.675+0000 7f28faffd700 1 -- 172.21.15.59:0/1269097232 <== osd.7 v2:172.21.15.110:6805/116561 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f29140a9de0 con 0x7f290006e8d0 2023-09-30T02:06:21.675 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.675+0000 7f28faffd700 10 client.4318.objecter ms_dispatch 0x7f2914001550 command_reply(tid 2: 0 ) v1 2023-09-30T02:06:21.675 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.675+0000 7f28faffd700 10 client.4318.objecter _finish_command 2 = system:0 2023-09-30T02:06:21.675 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.675+0000 7f28faffd700 20 client.4318.objecter put_session s=0x7f290006e220 osd=7 4 2023-09-30T02:06:21.675 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.675+0000 7f28faffd700 15 client.4318.objecter _session_command_op_remove 7 2 2023-09-30T02:06:21.676 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.676+0000 7f291972e700 10 librados: watch_flush enter 2023-09-30T02:06:21.676 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.676+0000 7f291972e700 10 librados: watch_flush exit 2023-09-30T02:06:21.676 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.676+0000 7f291972e700 10 client.4318.objecter close_session for osd.7 2023-09-30T02:06:21.677 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.676+0000 7f291972e700 1 -- 172.21.15.59:0/1269097232 >> v2:172.21.15.110:6805/116561 conn(0x7f290006e8d0 msgr2=0x7f2900070ce0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:21.677 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.676+0000 7f291972e700 1 --2- 172.21.15.59:0/1269097232 >> v2:172.21.15.110:6805/116561 conn(0x7f290006e8d0 0x7f2900070ce0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:21.677 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.676+0000 7f291972e700 20 client.4318.objecter put_session s=0x7f290006e220 osd=7 1 2023-09-30T02:06:21.677 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.676+0000 7f291972e700 20 client.4318.objecter shutdown clearing up homeless session... 2023-09-30T02:06:21.677 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.676+0000 7f291972e700 10 client.4318.objecter successfully canceled tick 2023-09-30T02:06:21.677 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.676+0000 7f291972e700 1 -- 172.21.15.59:0/1269097232 >> v2:172.21.15.110:6800/115870 conn(0x7f2900068960 msgr2=0x7f290006ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:21.677 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.676+0000 7f291972e700 1 --2- 172.21.15.59:0/1269097232 >> v2:172.21.15.110:6800/115870 conn(0x7f2900068960 0x7f290006ae10 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f290800a8c0 tx=0x7f2908008040 comp rx=0 tx=0).stop 2023-09-30T02:06:21.677 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.676+0000 7f291972e700 10 monclient: shutdown 2023-09-30T02:06:21.677 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.676+0000 7f291972e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:21.677 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.676+0000 7f291972e700 1 -- 172.21.15.59:0/1269097232 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f291413e140 msgr2=0x7f29140aa090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:21.677 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.676+0000 7f291972e700 1 --2- 172.21.15.59:0/1269097232 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f291413e140 0x7f29140aa090 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f290c00b610 tx=0x7f290c017000 comp rx=0 tx=0).stop 2023-09-30T02:06:21.678 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.676+0000 7f29137fe700 1 -- 172.21.15.59:0/1269097232 reap_dead start 2023-09-30T02:06:21.678 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.676+0000 7f291972e700 1 -- 172.21.15.59:0/1269097232 shutdown_connections 2023-09-30T02:06:21.678 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.676+0000 7f291972e700 1 -- 172.21.15.59:0/1269097232 >> 172.21.15.59:0/1269097232 conn(0x7f29140b08e0 msgr2=0x7f29140afb20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:21.678 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.676+0000 7f291972e700 1 -- 172.21.15.59:0/1269097232 shutdown_connections 2023-09-30T02:06:21.678 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.676+0000 7f291972e700 1 -- 172.21.15.59:0/1269097232 wait complete. 2023-09-30T02:06:21.678 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.677+0000 7f291972e700 1 librados: shutdown 2023-09-30T02:06:21.679 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.677+0000 7f9d199df700 10 librados: watch_flush enter 2023-09-30T02:06:21.679 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.677+0000 7f9d199df700 10 librados: watch_flush exit 2023-09-30T02:06:21.679 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.677+0000 7f9d199df700 10 client.4280.objecter close_session for osd.2 2023-09-30T02:06:21.679 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.677+0000 7f9d199df700 1 -- 172.21.15.59:0/3859569758 >> v2:172.21.15.59:6804/116666 conn(0x7f9d0006e8d0 msgr2=0x7f9d00070ce0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:21.679 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.677+0000 7f9d199df700 1 --2- 172.21.15.59:0/3859569758 >> v2:172.21.15.59:6804/116666 conn(0x7f9d0006e8d0 0x7f9d00070ce0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:21.679 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.677+0000 7f9d199df700 20 client.4280.objecter put_session s=0x7f9d0006e220 osd=2 1 2023-09-30T02:06:21.679 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.677+0000 7f9d199df700 20 client.4280.objecter shutdown clearing up homeless session... 2023-09-30T02:06:21.680 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.677+0000 7f9d199df700 10 client.4280.objecter successfully canceled tick 2023-09-30T02:06:21.680 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.677+0000 7f9d199df700 1 -- 172.21.15.59:0/3859569758 >> v2:172.21.15.110:6800/115870 conn(0x7f9d00068960 msgr2=0x7f9d0006ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:21.680 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.677+0000 7f9d199df700 1 --2- 172.21.15.59:0/3859569758 >> v2:172.21.15.110:6800/115870 conn(0x7f9d00068960 0x7f9d0006ae10 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9d08000c00 tx=0x7f9d08015000 comp rx=0 tx=0).stop 2023-09-30T02:06:21.680 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.677+0000 7f9d199df700 10 monclient: shutdown 2023-09-30T02:06:21.680 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.677+0000 7f9d199df700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:21.680 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.677+0000 7f9d199df700 1 -- 172.21.15.59:0/3859569758 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9d141529d0 msgr2=0x7f9d1413ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:21.680 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.677+0000 7f9d199df700 1 --2- 172.21.15.59:0/3859569758 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9d141529d0 0x7f9d1413ec50 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f9d141403e0 tx=0x7f9d04008040 comp rx=0 tx=0).stop 2023-09-30T02:06:21.680 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.677+0000 7f9d137fe700 1 -- 172.21.15.59:0/3859569758 reap_dead start 2023-09-30T02:06:21.680 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.678+0000 7f9d199df700 1 -- 172.21.15.59:0/3859569758 shutdown_connections 2023-09-30T02:06:21.680 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.678+0000 7f9d199df700 1 -- 172.21.15.59:0/3859569758 >> 172.21.15.59:0/3859569758 conn(0x7f9d140631f0 msgr2=0x7f9d140ae440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:21.681 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.680+0000 7f32e8ff9700 1 -- 172.21.15.59:0/802744325 <== osd.4 v2:172.21.15.110:6801/116560 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f32d8071350 con 0x7f32d806e8d0 2023-09-30T02:06:21.681 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.680+0000 7f32e8ff9700 10 client.4293.objecter ms_dispatch 0x7f32ec001550 command_reply(tid 1: 0 ) v1 2023-09-30T02:06:21.681 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.680+0000 7f32e8ff9700 10 client.4293.objecter _finish_command 1 = system:0 2023-09-30T02:06:21.681 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.680+0000 7f32e8ff9700 20 client.4293.objecter put_session s=0x7f32d806e220 osd=4 4 2023-09-30T02:06:21.681 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.680+0000 7f32e8ff9700 15 client.4293.objecter _session_command_op_remove 4 1 2023-09-30T02:06:21.681 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.680+0000 7f9d199df700 1 -- 172.21.15.59:0/3859569758 shutdown_connections 2023-09-30T02:06:21.682 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.682+0000 7fbe64a5d700 10 client.4321.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-09-30T02:06:21.682 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.682+0000 7fbe64a5d700 15 client.4321.objecter _session_command_op_assign -1 2 2023-09-30T02:06:21.682 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.682+0000 7fbe64a5d700 20 client.4321.objecter _get_session s=0x7fbe4006e220 osd=1 3 2023-09-30T02:06:21.682 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.682+0000 7fbe64a5d700 20 client.4321.objecter put_session s=0x7fbe4006e220 osd=1 3 2023-09-30T02:06:21.682 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.682+0000 7fbe64a5d700 20 client.4321.objecter _get_session s=0x7fbe4006e220 osd=1 3 2023-09-30T02:06:21.682 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.682+0000 7fbe64a5d700 15 client.4321.objecter _session_command_op_remove -1 2 2023-09-30T02:06:21.682 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.682+0000 7fbe64a5d700 20 client.4321.objecter get_session s=0x7fbe4006e220 osd=1 3 2023-09-30T02:06:21.683 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.682+0000 7fbe64a5d700 15 client.4321.objecter _session_command_op_assign 1 2 2023-09-30T02:06:21.683 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.682+0000 7fbe64a5d700 20 client.4321.objecter put_session s=0x7fbe4006e220 osd=1 4 2023-09-30T02:06:21.683 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.682+0000 7fbe64a5d700 10 client.4321.objecter _send_command 2 2023-09-30T02:06:21.683 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.682+0000 7fbe64a5d700 1 -- 172.21.15.59:0/656608041 --> v2:172.21.15.59:6808/116665 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fbe60072b30 con 0x7fbe4006e8d0 2023-09-30T02:06:21.684 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.685+0000 7f9d199df700 1 -- 172.21.15.59:0/3859569758 wait complete. 2023-09-30T02:06:21.684 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.685+0000 7f9d199df700 1 librados: shutdown 2023-09-30T02:06:21.690 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.691+0000 7f0f9055c700 10 client.4330.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-09-30T02:06:21.690 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.691+0000 7f0f9055c700 15 client.4330.objecter _session_command_op_assign -1 2 2023-09-30T02:06:21.690 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.691+0000 7f0f9055c700 20 client.4330.objecter _get_session s=0x7f0f7800bd10 osd=6 3 2023-09-30T02:06:21.690 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.691+0000 7f0f9055c700 20 client.4330.objecter put_session s=0x7f0f7800bd10 osd=6 3 2023-09-30T02:06:21.690 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.691+0000 7f0f9055c700 20 client.4330.objecter _get_session s=0x7f0f7800bd10 osd=6 3 2023-09-30T02:06:21.691 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.691+0000 7f0f9055c700 15 client.4330.objecter _session_command_op_remove -1 2 2023-09-30T02:06:21.691 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.691+0000 7f0f9055c700 20 client.4330.objecter get_session s=0x7f0f7800bd10 osd=6 3 2023-09-30T02:06:21.691 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.691+0000 7f0f9055c700 15 client.4330.objecter _session_command_op_assign 6 2 2023-09-30T02:06:21.691 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.691+0000 7f0f9055c700 20 client.4330.objecter put_session s=0x7f0f7800bd10 osd=6 4 2023-09-30T02:06:21.691 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.691+0000 7f0f9055c700 10 client.4330.objecter _send_command 2 2023-09-30T02:06:21.691 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.691+0000 7f0f9055c700 1 -- 172.21.15.59:0/3803865518 --> v2:172.21.15.110:6813/116563 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0f88072b30 con 0x7f0f6c06e690 2023-09-30T02:06:21.695 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.696+0000 7fbe4e7fc700 1 -- 172.21.15.59:0/656608041 <== osd.1 v2:172.21.15.59:6808/116665 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fbe60072b30 con 0x7fbe4006e8d0 2023-09-30T02:06:21.695 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.696+0000 7fbe4e7fc700 10 client.4321.objecter ms_dispatch 0x7fbe60001550 command_reply(tid 2: 0 ) v1 2023-09-30T02:06:21.695 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.696+0000 7fbe4e7fc700 10 client.4321.objecter _finish_command 2 = system:0 2023-09-30T02:06:21.695 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.696+0000 7fbe4e7fc700 20 client.4321.objecter put_session s=0x7fbe4006e220 osd=1 4 2023-09-30T02:06:21.695 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.696+0000 7fbe4e7fc700 15 client.4321.objecter _session_command_op_remove 1 2 2023-09-30T02:06:21.698 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.699+0000 7f0f7e7fc700 1 -- 172.21.15.59:0/3803865518 <== osd.6 v2:172.21.15.110:6813/116563 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f0f88072b30 con 0x7f0f6c06e690 2023-09-30T02:06:21.698 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.699+0000 7f0f7e7fc700 10 client.4330.objecter ms_dispatch 0x7f0f88001550 command_reply(tid 2: 0 ) v1 2023-09-30T02:06:21.698 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.699+0000 7f0f7e7fc700 10 client.4330.objecter _finish_command 2 = system:0 2023-09-30T02:06:21.698 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.699+0000 7f0f7e7fc700 20 client.4330.objecter put_session s=0x7f0f7800bd10 osd=6 4 2023-09-30T02:06:21.699 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.699+0000 7f0f7e7fc700 15 client.4330.objecter _session_command_op_remove 6 2 2023-09-30T02:06:21.699 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.699+0000 7f0f9055c700 10 librados: watch_flush enter 2023-09-30T02:06:21.699 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.700+0000 7f0f9055c700 10 librados: watch_flush exit 2023-09-30T02:06:21.699 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.700+0000 7f0f9055c700 10 client.4330.objecter close_session for osd.6 2023-09-30T02:06:21.700 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.700+0000 7f0f9055c700 1 -- 172.21.15.59:0/3803865518 >> v2:172.21.15.110:6813/116563 conn(0x7f0f6c06e690 msgr2=0x7f0f6c070aa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:21.700 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.700+0000 7f0f9055c700 1 --2- 172.21.15.59:0/3803865518 >> v2:172.21.15.110:6813/116563 conn(0x7f0f6c06e690 0x7f0f6c070aa0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:21.700 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.700+0000 7f0f9055c700 20 client.4330.objecter put_session s=0x7f0f7800bd10 osd=6 1 2023-09-30T02:06:21.700 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.700+0000 7f0f9055c700 20 client.4330.objecter shutdown clearing up homeless session... 2023-09-30T02:06:21.700 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.700+0000 7f0f9055c700 10 client.4330.objecter successfully canceled tick 2023-09-30T02:06:21.701 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.700+0000 7fbe64a5d700 10 librados: watch_flush enter 2023-09-30T02:06:21.701 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.700+0000 7fbe64a5d700 10 librados: watch_flush exit 2023-09-30T02:06:21.701 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.700+0000 7fbe64a5d700 10 client.4321.objecter close_session for osd.1 2023-09-30T02:06:21.701 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.700+0000 7fbe64a5d700 1 -- 172.21.15.59:0/656608041 >> v2:172.21.15.59:6808/116665 conn(0x7fbe4006e8d0 msgr2=0x7fbe40070ce0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:21.701 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.700+0000 7fbe64a5d700 1 --2- 172.21.15.59:0/656608041 >> v2:172.21.15.59:6808/116665 conn(0x7fbe4006e8d0 0x7fbe40070ce0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:21.701 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.700+0000 7fbe64a5d700 20 client.4321.objecter put_session s=0x7fbe4006e220 osd=1 1 2023-09-30T02:06:21.701 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.700+0000 7fbe64a5d700 20 client.4321.objecter shutdown clearing up homeless session... 2023-09-30T02:06:21.701 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.700+0000 7fbe64a5d700 10 client.4321.objecter successfully canceled tick 2023-09-30T02:06:21.701 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.700+0000 7fbe64a5d700 1 -- 172.21.15.59:0/656608041 >> v2:172.21.15.110:6800/115870 conn(0x7fbe40068960 msgr2=0x7fbe4006ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:21.701 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.700+0000 7fbe64a5d700 1 --2- 172.21.15.59:0/656608041 >> v2:172.21.15.110:6800/115870 conn(0x7fbe40068960 0x7fbe4006ae10 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fbe54006fd0 tx=0x7fbe54008040 comp rx=0 tx=0).stop 2023-09-30T02:06:21.701 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.700+0000 7fbe64a5d700 10 monclient: shutdown 2023-09-30T02:06:21.702 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.700+0000 7fbe64a5d700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:21.702 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.700+0000 7fbe64a5d700 1 -- 172.21.15.59:0/656608041 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbe60087810 msgr2=0x7fbe600a90a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:21.702 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.700+0000 7fbe64a5d700 1 --2- 172.21.15.59:0/656608041 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbe60087810 0x7fbe600a90a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fbe5800e190 tx=0x7fbe58009920 comp rx=0 tx=0).stop 2023-09-30T02:06:21.702 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.700+0000 7fbe5ed9d700 1 -- 172.21.15.59:0/656608041 reap_dead start 2023-09-30T02:06:21.702 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.700+0000 7fbe64a5d700 1 -- 172.21.15.59:0/656608041 shutdown_connections 2023-09-30T02:06:21.702 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.700+0000 7fbe64a5d700 1 -- 172.21.15.59:0/656608041 >> 172.21.15.59:0/656608041 conn(0x7fbe600b08e0 msgr2=0x7fbe600ab6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:21.702 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.701+0000 7fbe64a5d700 1 -- 172.21.15.59:0/656608041 shutdown_connections 2023-09-30T02:06:21.702 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.701+0000 7fbe64a5d700 1 -- 172.21.15.59:0/656608041 wait complete. 2023-09-30T02:06:21.702 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.701+0000 7fbe64a5d700 1 librados: shutdown 2023-09-30T02:06:21.703 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.702+0000 7f0f9055c700 1 -- 172.21.15.59:0/3803865518 >> v2:172.21.15.110:6800/115870 conn(0x7f0f6c068890 msgr2=0x7f0f6c06ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:21.703 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.702+0000 7f0f9055c700 1 --2- 172.21.15.59:0/3803865518 >> v2:172.21.15.110:6800/115870 conn(0x7f0f6c068890 0x7f0f6c06ad40 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0f8000ad10 tx=0x7f0f80009370 comp rx=0 tx=0).stop 2023-09-30T02:06:21.703 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.702+0000 7f0f9055c700 10 monclient: shutdown 2023-09-30T02:06:21.703 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.702+0000 7f0f9055c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:21.703 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.702+0000 7f0f9055c700 1 -- 172.21.15.59:0/3803865518 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0f880ab4e0 msgr2=0x7f0f8811ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:21.703 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.702+0000 7f0f9055c700 1 --2- 172.21.15.59:0/3803865518 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0f880ab4e0 0x7f0f8811ab90 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0f78003080 tx=0x7f0f78016000 comp rx=0 tx=0).stop 2023-09-30T02:06:21.703 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.702+0000 7f0f8eaf9700 1 -- 172.21.15.59:0/3803865518 reap_dead start 2023-09-30T02:06:21.703 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.704+0000 7f0f9055c700 1 -- 172.21.15.59:0/3803865518 shutdown_connections 2023-09-30T02:06:21.704 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.704+0000 7f0f9055c700 1 -- 172.21.15.59:0/3803865518 >> 172.21.15.59:0/3803865518 conn(0x7f0f880b08e0 msgr2=0x7f0f880b0020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:21.704 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.705+0000 7f32f314d700 10 client.4293.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-09-30T02:06:21.704 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.705+0000 7f32f314d700 15 client.4293.objecter _session_command_op_assign -1 2 2023-09-30T02:06:21.705 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.705+0000 7f32f314d700 20 client.4293.objecter _get_session s=0x7f32d806e220 osd=4 3 2023-09-30T02:06:21.705 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.705+0000 7f32f314d700 20 client.4293.objecter put_session s=0x7f32d806e220 osd=4 3 2023-09-30T02:06:21.705 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.705+0000 7f32f314d700 20 client.4293.objecter _get_session s=0x7f32d806e220 osd=4 3 2023-09-30T02:06:21.705 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.705+0000 7f32f314d700 15 client.4293.objecter _session_command_op_remove -1 2 2023-09-30T02:06:21.705 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.705+0000 7f32f314d700 20 client.4293.objecter get_session s=0x7f32d806e220 osd=4 3 2023-09-30T02:06:21.705 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.705+0000 7f32f314d700 15 client.4293.objecter _session_command_op_assign 4 2 2023-09-30T02:06:21.705 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.705+0000 7f32f314d700 20 client.4293.objecter put_session s=0x7f32d806e220 osd=4 4 2023-09-30T02:06:21.705 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.705+0000 7f32f314d700 10 client.4293.objecter _send_command 2 2023-09-30T02:06:21.705 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.705+0000 7f32f314d700 1 -- 172.21.15.59:0/802744325 --> v2:172.21.15.110:6801/116560 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f32ec0649f0 con 0x7f32d806e8d0 2023-09-30T02:06:21.706 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.706+0000 7f0f9055c700 1 -- 172.21.15.59:0/3803865518 shutdown_connections 2023-09-30T02:06:21.706 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.706+0000 7f0f9055c700 1 -- 172.21.15.59:0/3803865518 wait complete. 2023-09-30T02:06:21.707 INFO:teuthology.orchestra.run.smithi059.stdout:60129542151 2023-09-30T02:06:21.707 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-09-30T02:06:21.707 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.708+0000 7f0f9055c700 1 librados: shutdown 2023-09-30T02:06:21.708 INFO:teuthology.orchestra.run.smithi059.stdout:77309411333 2023-09-30T02:06:21.709 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-09-30T02:06:21.711 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.712+0000 7ff3367fc700 1 -- 172.21.15.59:0/723961325 <== osd.0 v2:172.21.15.59:6800/116664 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7ff324071300 con 0x7ff32406e880 2023-09-30T02:06:21.711 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.712+0000 7ff3367fc700 10 client.4286.objecter ms_dispatch 0x7ff340001550 command_reply(tid 1: 0 ) v1 2023-09-30T02:06:21.711 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.712+0000 7ff3367fc700 10 client.4286.objecter _finish_command 1 = system:0 2023-09-30T02:06:21.712 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.712+0000 7ff3367fc700 20 client.4286.objecter put_session s=0x7ff32406e1d0 osd=0 4 2023-09-30T02:06:21.712 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.712+0000 7ff3367fc700 15 client.4286.objecter _session_command_op_remove 0 1 2023-09-30T02:06:21.712 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.713+0000 7f32e8ff9700 1 -- 172.21.15.59:0/802744325 <== osd.4 v2:172.21.15.110:6801/116560 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f32ec0649f0 con 0x7f32d806e8d0 2023-09-30T02:06:21.712 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.713+0000 7f32e8ff9700 10 client.4293.objecter ms_dispatch 0x7f32ec001550 command_reply(tid 2: 0 ) v1 2023-09-30T02:06:21.712 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.713+0000 7f32e8ff9700 10 client.4293.objecter _finish_command 2 = system:0 2023-09-30T02:06:21.713 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.713+0000 7f32e8ff9700 20 client.4293.objecter put_session s=0x7f32d806e220 osd=4 4 2023-09-30T02:06:21.713 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.713+0000 7f32e8ff9700 15 client.4293.objecter _session_command_op_remove 4 2 2023-09-30T02:06:21.713 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.714+0000 7f32f314d700 10 librados: watch_flush enter 2023-09-30T02:06:21.713 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.714+0000 7f32f314d700 10 librados: watch_flush exit 2023-09-30T02:06:21.713 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.714+0000 7f32f314d700 10 client.4293.objecter close_session for osd.4 2023-09-30T02:06:21.714 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.714+0000 7f32f314d700 1 -- 172.21.15.59:0/802744325 >> v2:172.21.15.110:6801/116560 conn(0x7f32d806e8d0 msgr2=0x7f32d8070ce0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:21.714 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.714+0000 7f32f314d700 1 --2- 172.21.15.59:0/802744325 >> v2:172.21.15.110:6801/116560 conn(0x7f32d806e8d0 0x7f32d8070ce0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:21.714 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.714+0000 7f32f314d700 20 client.4293.objecter put_session s=0x7f32d806e220 osd=4 1 2023-09-30T02:06:21.714 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.714+0000 7f32f314d700 20 client.4293.objecter shutdown clearing up homeless session... 2023-09-30T02:06:21.714 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.714+0000 7f32f314d700 10 client.4293.objecter successfully canceled tick 2023-09-30T02:06:21.714 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.714+0000 7f32f314d700 1 -- 172.21.15.59:0/802744325 >> v2:172.21.15.110:6800/115870 conn(0x7f32d8068960 msgr2=0x7f32d806ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:21.714 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.714+0000 7f32f314d700 1 --2- 172.21.15.59:0/802744325 >> v2:172.21.15.110:6800/115870 conn(0x7f32d8068960 0x7f32d806ae10 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f32e40020c0 tx=0x7f32e4001a10 comp rx=0 tx=0).stop 2023-09-30T02:06:21.714 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.714+0000 7f32f314d700 10 monclient: shutdown 2023-09-30T02:06:21.714 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.714+0000 7f32f314d700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:21.714 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.714+0000 7f32f314d700 1 -- 172.21.15.59:0/802744325 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32ec0b13f0 msgr2=0x7f32ec11a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:21.714 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.714+0000 7f32f314d700 1 --2- 172.21.15.59:0/802744325 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f32ec0b13f0 0x7f32ec11a0b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f32dc00d890 tx=0x7f32dc008040 comp rx=0 tx=0).stop 2023-09-30T02:06:21.714 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.714+0000 7f32f16ea700 1 -- 172.21.15.59:0/802744325 reap_dead start 2023-09-30T02:06:21.715 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.715+0000 7f32f314d700 1 -- 172.21.15.59:0/802744325 shutdown_connections 2023-09-30T02:06:21.715 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.715+0000 7f32f314d700 1 -- 172.21.15.59:0/802744325 >> 172.21.15.59:0/802744325 conn(0x7f32ec0b08e0 msgr2=0x7f32ec0b0cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:21.718 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.718+0000 7f32f314d700 1 -- 172.21.15.59:0/802744325 shutdown_connections 2023-09-30T02:06:21.718 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.718+0000 7f32f314d700 1 -- 172.21.15.59:0/802744325 wait complete. 2023-09-30T02:06:21.718 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.719+0000 7f32f314d700 1 librados: shutdown 2023-09-30T02:06:21.719 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.719+0000 7f43f0ff9700 1 -- 172.21.15.59:0/1297509251 <== osd.5 v2:172.21.15.110:6809/116562 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f43e0071350 con 0x7f43e006e8d0 2023-09-30T02:06:21.719 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.719+0000 7f43f0ff9700 10 client.4333.objecter ms_dispatch 0x7f43f4001550 command_reply(tid 1: 0 ) v1 2023-09-30T02:06:21.719 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.719+0000 7f43f0ff9700 10 client.4333.objecter _finish_command 1 = system:0 2023-09-30T02:06:21.719 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.719+0000 7f43f0ff9700 20 client.4333.objecter put_session s=0x7f43e006e220 osd=5 4 2023-09-30T02:06:21.719 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.719+0000 7f43f0ff9700 15 client.4333.objecter _session_command_op_remove 5 1 2023-09-30T02:06:21.721 INFO:teuthology.orchestra.run.smithi059.stdout:77309411333 2023-09-30T02:06:21.722 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-09-30T02:06:21.730 INFO:teuthology.orchestra.run.smithi059.stdout:55834574854 2023-09-30T02:06:21.730 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-09-30T02:06:21.744 INFO:teuthology.orchestra.run.smithi059.stdout:64424509447 2023-09-30T02:06:21.744 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-09-30T02:06:21.754 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.746+0000 7ff3483d3700 10 client.4286.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-09-30T02:06:21.754 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.746+0000 7ff3483d3700 15 client.4286.objecter _session_command_op_assign -1 2 2023-09-30T02:06:21.754 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.746+0000 7ff3483d3700 20 client.4286.objecter _get_session s=0x7ff32406e1d0 osd=0 3 2023-09-30T02:06:21.754 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.746+0000 7ff3483d3700 20 client.4286.objecter put_session s=0x7ff32406e1d0 osd=0 3 2023-09-30T02:06:21.754 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.746+0000 7ff3483d3700 20 client.4286.objecter _get_session s=0x7ff32406e1d0 osd=0 3 2023-09-30T02:06:21.754 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.746+0000 7ff3483d3700 15 client.4286.objecter _session_command_op_remove -1 2 2023-09-30T02:06:21.754 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.746+0000 7ff3483d3700 20 client.4286.objecter get_session s=0x7ff32406e1d0 osd=0 3 2023-09-30T02:06:21.754 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.746+0000 7ff3483d3700 15 client.4286.objecter _session_command_op_assign 0 2 2023-09-30T02:06:21.754 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.746+0000 7ff3483d3700 20 client.4286.objecter put_session s=0x7ff32406e1d0 osd=0 4 2023-09-30T02:06:21.754 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.746+0000 7ff3483d3700 10 client.4286.objecter _send_command 2 2023-09-30T02:06:21.755 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.746+0000 7ff3483d3700 1 -- 172.21.15.59:0/723961325 --> v2:172.21.15.59:6800/116664 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff3280011e0 con 0x7ff32406e880 2023-09-30T02:06:21.761 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.761+0000 7ff3367fc700 1 -- 172.21.15.59:0/723961325 <== osd.0 v2:172.21.15.59:6800/116664 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7ff3280011e0 con 0x7ff32406e880 2023-09-30T02:06:21.761 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.761+0000 7ff3367fc700 10 client.4286.objecter ms_dispatch 0x7ff340001550 command_reply(tid 2: 0 ) v1 2023-09-30T02:06:21.761 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.761+0000 7ff3367fc700 10 client.4286.objecter _finish_command 2 = system:0 2023-09-30T02:06:21.761 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.761+0000 7ff3367fc700 20 client.4286.objecter put_session s=0x7ff32406e1d0 osd=0 4 2023-09-30T02:06:21.761 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.761+0000 7ff3367fc700 15 client.4286.objecter _session_command_op_remove 0 2 2023-09-30T02:06:21.761 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.761+0000 7ff3483d3700 10 librados: watch_flush enter 2023-09-30T02:06:21.761 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.761+0000 7ff3483d3700 10 librados: watch_flush exit 2023-09-30T02:06:21.761 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.761+0000 7f43faed4700 10 client.4333.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2023-09-30T02:06:21.762 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.761+0000 7f43faed4700 15 client.4333.objecter _session_command_op_assign -1 2 2023-09-30T02:06:21.762 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.761+0000 7f43faed4700 20 client.4333.objecter _get_session s=0x7f43e006e220 osd=5 3 2023-09-30T02:06:21.762 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.761+0000 7f43faed4700 20 client.4333.objecter put_session s=0x7f43e006e220 osd=5 3 2023-09-30T02:06:21.762 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.761+0000 7f43faed4700 20 client.4333.objecter _get_session s=0x7f43e006e220 osd=5 3 2023-09-30T02:06:21.762 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.761+0000 7f43faed4700 15 client.4333.objecter _session_command_op_remove -1 2 2023-09-30T02:06:21.762 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.761+0000 7f43faed4700 20 client.4333.objecter get_session s=0x7f43e006e220 osd=5 3 2023-09-30T02:06:21.762 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.761+0000 7f43faed4700 15 client.4333.objecter _session_command_op_assign 5 2 2023-09-30T02:06:21.762 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.761+0000 7f43faed4700 20 client.4333.objecter put_session s=0x7f43e006e220 osd=5 4 2023-09-30T02:06:21.762 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.761+0000 7f43faed4700 10 client.4333.objecter _send_command 2 2023-09-30T02:06:21.762 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.761+0000 7f43faed4700 1 -- 172.21.15.59:0/1297509251 --> v2:172.21.15.110:6809/116562 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f43f413c790 con 0x7f43e006e8d0 2023-09-30T02:06:21.763 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.761+0000 7ff3483d3700 10 client.4286.objecter close_session for osd.0 2023-09-30T02:06:21.763 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.761+0000 7ff3483d3700 1 -- 172.21.15.59:0/723961325 >> v2:172.21.15.59:6800/116664 conn(0x7ff32406e880 msgr2=0x7ff324070c90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:21.763 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.761+0000 7ff3483d3700 1 --2- 172.21.15.59:0/723961325 >> v2:172.21.15.59:6800/116664 conn(0x7ff32406e880 0x7ff324070c90 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:21.763 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.761+0000 7ff3483d3700 20 client.4286.objecter put_session s=0x7ff32406e1d0 osd=0 1 2023-09-30T02:06:21.763 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.761+0000 7ff3483d3700 20 client.4286.objecter shutdown clearing up homeless session... 2023-09-30T02:06:21.763 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.761+0000 7ff3483d3700 10 client.4286.objecter successfully canceled tick 2023-09-30T02:06:21.764 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.761+0000 7ff3483d3700 1 -- 172.21.15.59:0/723961325 >> v2:172.21.15.110:6800/115870 conn(0x7ff324068910 msgr2=0x7ff32406adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:21.764 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.761+0000 7ff3483d3700 1 --2- 172.21.15.59:0/723961325 >> v2:172.21.15.110:6800/115870 conn(0x7ff324068910 0x7ff32406adc0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff33c0058d0 tx=0x7ff33c0079d0 comp rx=0 tx=0).stop 2023-09-30T02:06:21.764 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.761+0000 7ff3483d3700 10 monclient: shutdown 2023-09-30T02:06:21.764 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.761+0000 7ff3483d3700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:21.765 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.761+0000 7ff3483d3700 1 -- 172.21.15.59:0/723961325 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff34006ba90 msgr2=0x7ff3400a9370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:21.765 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.761+0000 7ff3483d3700 1 --2- 172.21.15.59:0/723961325 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff34006ba90 0x7ff3400a9370 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff33000ecb0 tx=0x7ff3300091a0 comp rx=0 tx=0).stop 2023-09-30T02:06:21.765 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.762+0000 7ff346970700 1 -- 172.21.15.59:0/723961325 reap_dead start 2023-09-30T02:06:21.765 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.762+0000 7ff3483d3700 1 -- 172.21.15.59:0/723961325 shutdown_connections 2023-09-30T02:06:21.765 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.762+0000 7ff3483d3700 1 -- 172.21.15.59:0/723961325 >> 172.21.15.59:0/723961325 conn(0x7ff3400b08e0 msgr2=0x7ff340144b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:21.765 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.764+0000 7ff3483d3700 1 -- 172.21.15.59:0/723961325 shutdown_connections 2023-09-30T02:06:21.765 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.764+0000 7ff3483d3700 1 -- 172.21.15.59:0/723961325 wait complete. 2023-09-30T02:06:21.765 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.764+0000 7ff3483d3700 1 librados: shutdown 2023-09-30T02:06:21.767 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.768+0000 7f43f0ff9700 1 -- 172.21.15.59:0/1297509251 <== osd.5 v2:172.21.15.110:6809/116562 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f43f413c790 con 0x7f43e006e8d0 2023-09-30T02:06:21.767 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.768+0000 7f43f0ff9700 10 client.4333.objecter ms_dispatch 0x7f43f4001550 command_reply(tid 2: 0 ) v1 2023-09-30T02:06:21.767 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.768+0000 7f43f0ff9700 10 client.4333.objecter _finish_command 2 = system:0 2023-09-30T02:06:21.767 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.768+0000 7f43f0ff9700 20 client.4333.objecter put_session s=0x7f43e006e220 osd=5 4 2023-09-30T02:06:21.767 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.768+0000 7f43f0ff9700 15 client.4333.objecter _session_command_op_remove 5 2 2023-09-30T02:06:21.769 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.768+0000 7f43faed4700 10 librados: watch_flush enter 2023-09-30T02:06:21.769 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.769+0000 7f43faed4700 10 librados: watch_flush exit 2023-09-30T02:06:21.769 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.769+0000 7f43faed4700 10 client.4333.objecter close_session for osd.5 2023-09-30T02:06:21.769 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.769+0000 7f43faed4700 1 -- 172.21.15.59:0/1297509251 >> v2:172.21.15.110:6809/116562 conn(0x7f43e006e8d0 msgr2=0x7f43e0070ce0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:21.769 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.769+0000 7f43faed4700 1 --2- 172.21.15.59:0/1297509251 >> v2:172.21.15.110:6809/116562 conn(0x7f43e006e8d0 0x7f43e0070ce0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:21.770 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.769+0000 7f43faed4700 20 client.4333.objecter put_session s=0x7f43e006e220 osd=5 1 2023-09-30T02:06:21.770 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.769+0000 7f43faed4700 20 client.4333.objecter shutdown clearing up homeless session... 2023-09-30T02:06:21.770 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.769+0000 7f43faed4700 10 client.4333.objecter successfully canceled tick 2023-09-30T02:06:21.770 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.769+0000 7f43faed4700 1 -- 172.21.15.59:0/1297509251 >> v2:172.21.15.110:6800/115870 conn(0x7f43e0068960 msgr2=0x7f43e006ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:21.770 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.769+0000 7f43faed4700 1 --2- 172.21.15.59:0/1297509251 >> v2:172.21.15.110:6800/115870 conn(0x7f43e0068960 0x7f43e006ae10 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f43e4006fd0 tx=0x7f43e40090f0 comp rx=0 tx=0).stop 2023-09-30T02:06:21.770 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.769+0000 7f43faed4700 10 monclient: shutdown 2023-09-30T02:06:21.770 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.769+0000 7f43faed4700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:21.770 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.769+0000 7f43faed4700 1 -- 172.21.15.59:0/1297509251 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f43f4142290 msgr2=0x7f43f416f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:21.770 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.769+0000 7f43faed4700 1 --2- 172.21.15.59:0/1297509251 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f43f4142290 0x7f43f416f180 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f43ec00bc30 tx=0x7f43ec009840 comp rx=0 tx=0).stop 2023-09-30T02:06:21.770 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.769+0000 7f43f9471700 1 -- 172.21.15.59:0/1297509251 reap_dead start 2023-09-30T02:06:21.770 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.769+0000 7f43faed4700 1 -- 172.21.15.59:0/1297509251 shutdown_connections 2023-09-30T02:06:21.770 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.769+0000 7f43faed4700 1 -- 172.21.15.59:0/1297509251 >> 172.21.15.59:0/1297509251 conn(0x7f43f40631f0 msgr2=0x7f43f4141b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:21.771 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.769+0000 7f43faed4700 1 -- 172.21.15.59:0/1297509251 shutdown_connections 2023-09-30T02:06:21.771 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.769+0000 7f43faed4700 1 -- 172.21.15.59:0/1297509251 wait complete. 2023-09-30T02:06:21.771 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.769+0000 7f43faed4700 1 librados: shutdown 2023-09-30T02:06:21.792 INFO:teuthology.orchestra.run.smithi059.stdout:77309411334 2023-09-30T02:06:21.792 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-09-30T02:06:21.798 INFO:teuthology.orchestra.run.smithi059.stdout:64424509446 2023-09-30T02:06:21.798 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-09-30T02:06:21.835 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.835+0000 7fa044ff9700 1 -- 172.21.15.59:0/2373407985 <== osd.3 v2:172.21.15.59:6812/116667 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa048139af0 con 0x7fa02806e800 2023-09-30T02:06:21.835 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.835+0000 7fa044ff9700 10 client.4290.objecter ms_dispatch 0x7fa048001550 command_reply(tid 2: 0 ) v1 2023-09-30T02:06:21.835 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.835+0000 7fa044ff9700 10 client.4290.objecter _finish_command 2 = system:0 2023-09-30T02:06:21.835 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.835+0000 7fa044ff9700 20 client.4290.objecter put_session s=0x7fa02806e150 osd=3 4 2023-09-30T02:06:21.835 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.835+0000 7fa044ff9700 15 client.4290.objecter _session_command_op_remove 3 2 2023-09-30T02:06:21.836 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.837+0000 7fa04e664700 10 librados: watch_flush enter 2023-09-30T02:06:21.840 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.837+0000 7fa04e664700 10 librados: watch_flush exit 2023-09-30T02:06:21.840 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.837+0000 7fa04e664700 10 client.4290.objecter close_session for osd.3 2023-09-30T02:06:21.840 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.837+0000 7fa04e664700 1 -- 172.21.15.59:0/2373407985 >> v2:172.21.15.59:6812/116667 conn(0x7fa02806e800 msgr2=0x7fa028070c10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:21.840 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.837+0000 7fa04e664700 1 --2- 172.21.15.59:0/2373407985 >> v2:172.21.15.59:6812/116667 conn(0x7fa02806e800 0x7fa028070c10 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:21.840 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.837+0000 7fa04e664700 20 client.4290.objecter put_session s=0x7fa02806e150 osd=3 1 2023-09-30T02:06:21.840 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.837+0000 7fa04e664700 20 client.4290.objecter shutdown clearing up homeless session... 2023-09-30T02:06:21.840 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.837+0000 7fa04e664700 10 client.4290.objecter successfully canceled tick 2023-09-30T02:06:21.841 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.841+0000 7fa04e664700 1 -- 172.21.15.59:0/2373407985 >> v2:172.21.15.110:6800/115870 conn(0x7fa028068890 msgr2=0x7fa02806ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:21.841 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.841+0000 7fa04e664700 1 --2- 172.21.15.59:0/2373407985 >> v2:172.21.15.110:6800/115870 conn(0x7fa028068890 0x7fa02806ad40 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa03000c010 tx=0x7fa030017000 comp rx=0 tx=0).stop 2023-09-30T02:06:21.841 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.841+0000 7fa04e664700 10 monclient: shutdown 2023-09-30T02:06:21.841 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.841+0000 7fa04e664700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:21.841 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.841+0000 7fa04e664700 1 -- 172.21.15.59:0/2373407985 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0480b12c0 msgr2=0x7fa04806b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:21.842 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.841+0000 7fa04e664700 1 --2- 172.21.15.59:0/2373407985 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0480b12c0 0x7fa04806b7e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa03800bc30 tx=0x7fa038009840 comp rx=0 tx=0).stop 2023-09-30T02:06:21.842 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.841+0000 7fa04cc01700 1 -- 172.21.15.59:0/2373407985 reap_dead start 2023-09-30T02:06:21.842 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.841+0000 7fa04e664700 1 -- 172.21.15.59:0/2373407985 shutdown_connections 2023-09-30T02:06:21.842 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.841+0000 7fa04e664700 1 -- 172.21.15.59:0/2373407985 >> 172.21.15.59:0/2373407985 conn(0x7fa0480631f0 msgr2=0x7fa0480afb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:21.842 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.841+0000 7fa04e664700 1 -- 172.21.15.59:0/2373407985 shutdown_connections 2023-09-30T02:06:21.842 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.842+0000 7fa04e664700 1 -- 172.21.15.59:0/2373407985 wait complete. 2023-09-30T02:06:21.843 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.844+0000 7fa04e664700 1 librados: shutdown 2023-09-30T02:06:21.871 INFO:teuthology.orchestra.run.smithi059.stdout:81604378630 2023-09-30T02:06:21.871 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-09-30T02:06:21.931 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.930+0000 7f13ac6c3700 10 monclient: get_monmap_and_config 2023-09-30T02:06:21.931 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.930+0000 7f13ac6c3700 10 monclient: build_initial_monmap 2023-09-30T02:06:21.931 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.930+0000 7f13ac6c3700 10 monclient: monmap: 2023-09-30T02:06:21.931 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:21.931 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:21.931 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:06:21.931168+0000 2023-09-30T02:06:21.931 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:06:21.931168+0000 2023-09-30T02:06:21.931 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:21.932 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:21.932 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:06:21.932 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:06:21.932 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:06:21.932 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:06:21.932 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:06:21.932 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:21.938 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.934+0000 7f13ac6c3700 1 Processor -- start 2023-09-30T02:06:21.940 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.938+0000 7f13ac6c3700 1 -- start start 2023-09-30T02:06:21.940 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.938+0000 7f13ac6c3700 10 monclient: init 2023-09-30T02:06:21.940 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.939+0000 7f13ac6c3700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:21.941 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.939+0000 7f13ac6c3700 10 monclient: _add_conns ranks=[3,1,4,2,0] 2023-09-30T02:06:21.941 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.939+0000 7f13ac6c3700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f13a40a65b0 0x7f13a40a6980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:21.941 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.939+0000 7f13ac6c3700 10 monclient(hunting): picked mon.noname-a con 0x7f13a40a65b0 addr v2:172.21.15.59:3300/0 2023-09-30T02:06:21.941 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.939+0000 7f13ac6c3700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f13a40a9940 addr v1:172.21.15.110:6789/0 2023-09-30T02:06:21.941 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.939+0000 7f13ac6c3700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f13a40ab5e0 0x7f13a40aba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:21.941 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.939+0000 7f13ac6c3700 10 monclient(hunting): picked mon.noname-c con 0x7f13a40ab5e0 addr v2:172.21.15.110:3300/0 2023-09-30T02:06:21.941 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.939+0000 7f13ac6c3700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13a40ab150 con 0x7f13a40a9940 2023-09-30T02:06:21.941 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.939+0000 7f13ac6c3700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:21.941 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.939+0000 7f13ac6c3700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13a40a6ec0 con 0x7f13a40a65b0 2023-09-30T02:06:21.941 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.939+0000 7f13ac6c3700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:21.941 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.939+0000 7f13ac6c3700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13a40a7840 con 0x7f13a40ab5e0 2023-09-30T02:06:21.942 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.939+0000 7f13ac6c3700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:21.942 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.939+0000 7f13ac6c3700 10 monclient(hunting): authenticate will time out at 1827.241886s 2023-09-30T02:06:21.942 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.941+0000 7f13aac60700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f13a40ab5e0 0x7f13a40aba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:21.942 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.942+0000 7f13aa45f700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f13a40a65b0 0x7f13a40a6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:21.942 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.942+0000 7f13a9c5e700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f13a40a9940 0x7f13a40a9da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.59:54828/0 (socket says 172.21.15.59:54828) 2023-09-30T02:06:21.942 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.942+0000 7f13a9c5e700 1 -- 172.21.15.59:0/3532691214 learned_addr learned my addr 172.21.15.59:0/3532691214 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:06:21.943 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.944+0000 7f13aa45f700 10 monclient(hunting): get_auth_request con 0x7f13a40a65b0 auth_method 0 2023-09-30T02:06:21.943 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.944+0000 7f13aa45f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:21.943 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.944+0000 7f13aa45f700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:21.943 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.944+0000 7f13aa45f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:21.945 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.946+0000 7f13aac60700 10 monclient(hunting): get_auth_request con 0x7f13a40ab5e0 auth_method 0 2023-09-30T02:06:21.945 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.946+0000 7f13aac60700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:21.945 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.946+0000 7f13aac60700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:21.945 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.946+0000 7f13aac60700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:21.945 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.946+0000 7f13aa45f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:21.945 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.946+0000 7f13aa45f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:21.945 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.946+0000 7f13aa45f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:21.946 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.943+0000 7f6135a5d700 10 monclient: get_monmap_and_config 2023-09-30T02:06:21.946 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.943+0000 7f6135a5d700 10 monclient: build_initial_monmap 2023-09-30T02:06:21.946 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.943+0000 7f6135a5d700 10 monclient: monmap: 2023-09-30T02:06:21.946 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:21.946 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:21.946 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:06:21.943752+0000 2023-09-30T02:06:21.946 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:06:21.943752+0000 2023-09-30T02:06:21.947 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:21.947 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:21.947 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:06:21.947 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:06:21.947 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:06:21.947 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:06:21.947 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:06:21.947 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:21.947 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.945+0000 7f6135a5d700 1 Processor -- start 2023-09-30T02:06:21.947 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.945+0000 7f6135a5d700 1 -- start start 2023-09-30T02:06:21.947 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.945+0000 7f6135a5d700 10 monclient: init 2023-09-30T02:06:21.947 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.946+0000 7f6135a5d700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:21.948 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.946+0000 7f6135a5d700 10 monclient: _add_conns ranks=[2,1,0,4,3] 2023-09-30T02:06:21.948 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.946+0000 7f6135a5d700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6130069b10 0x7f613006b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:21.948 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.946+0000 7f6135a5d700 10 monclient(hunting): picked mon.noname-b con 0x7f6130069b10 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:21.948 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.946+0000 7f6135a5d700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6130087810 addr v1:172.21.15.110:6789/0 2023-09-30T02:06:21.948 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.946+0000 7f6135a5d700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f613006bde0 addr v1:172.21.15.59:6789/0 2023-09-30T02:06:21.948 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.946+0000 7f6135a5d700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6130069ee0 con 0x7f613006bde0 2023-09-30T02:06:21.948 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.946+0000 7f6135a5d700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61300ac0f0 con 0x7f6130087810 2023-09-30T02:06:21.949 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.948+0000 7f6135a5d700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:21.949 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.948+0000 7f6135a5d700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f613006c260 con 0x7f6130069b10 2023-09-30T02:06:21.949 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.948+0000 7f6135a5d700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:21.949 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.948+0000 7f6135a5d700 10 monclient(hunting): authenticate will time out at 1827.250115s 2023-09-30T02:06:21.949 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.948+0000 7f612effd700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6130069b10 0x7f613006b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:21.949 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.949+0000 7f612effd700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6130069b10 0x7f613006b8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:47618/0 (socket says 172.21.15.59:47618) 2023-09-30T02:06:21.949 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.949+0000 7f612effd700 1 -- 172.21.15.59:0/16576379 learned_addr learned my addr 172.21.15.59:0/16576379 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:06:21.949 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.949+0000 7f612effd700 10 monclient(hunting): get_auth_request con 0x7f6130069b10 auth_method 0 2023-09-30T02:06:21.949 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.949+0000 7f612effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:21.949 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.949+0000 7f612effd700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:21.949 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.949+0000 7f612effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:21.950 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.949+0000 7f13a945d700 1 -- 172.21.15.59:0/3532691214 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1012066328 0 0) 0x7f13a40ab150 con 0x7f13a40a9940 2023-09-30T02:06:21.950 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.949+0000 7f13a945d700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:21.950 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.949+0000 7f13a945d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:21.950 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.949+0000 7f13a945d700 10 monclient(hunting): my global_id is 4339 2023-09-30T02:06:21.950 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.949+0000 7f13a945d700 1 -- 172.21.15.59:0/3532691214 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1390003570 con 0x7f13a40a9940 2023-09-30T02:06:21.950 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.949+0000 7f13aac60700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:21.950 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.949+0000 7f13aac60700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:21.950 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.949+0000 7f13aac60700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:21.951 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.950+0000 7f612effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:21.951 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.950+0000 7f612effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:21.951 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.950+0000 7f612effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:21.951 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.950+0000 7f13aa45f700 10 monclient(hunting): handle_auth_done global_id 4299 payload 274 2023-09-30T02:06:21.951 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.950+0000 7f13aa45f700 1 -- 172.21.15.59:0/3532691214 >> v2:172.21.15.110:3300/0 conn(0x7f13a40ab5e0 msgr2=0x7f13a40aba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:06:21.951 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.950+0000 7f13aa45f700 1 --2- 172.21.15.59:0/3532691214 >> v2:172.21.15.110:3300/0 conn(0x7f13a40ab5e0 0x7f13a40aba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:21.951 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.950+0000 7f13aa45f700 1 -- 172.21.15.59:0/3532691214 >> v1:172.21.15.110:6789/0 conn(0x7f13a40a9940 legacy=0x7f13a40a9da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:21.951 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.950+0000 7f13aa45f700 10 monclient: _finish_hunting 0 2023-09-30T02:06:21.952 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.950+0000 7f13aa45f700 1 monclient: found mon.noname-a 2023-09-30T02:06:21.952 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.950+0000 7f13aa45f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:21.952 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.950+0000 7f13aa45f700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.59:3300/0 2023-09-30T02:06:21.952 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.950+0000 7f13aa45f700 1 -- 172.21.15.59:0/3532691214 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13a40a7f00 con 0x7f13a40a65b0 2023-09-30T02:06:21.952 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.953+0000 7f612effd700 10 monclient(hunting): handle_auth_done global_id 4295 payload 274 2023-09-30T02:06:21.952 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.953+0000 7f612effd700 1 -- 172.21.15.59:0/16576379 >> v1:172.21.15.110:6789/0 conn(0x7f6130087810 legacy=0x7f61300ab2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:06:21.952 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.953+0000 7f612effd700 1 -- 172.21.15.59:0/16576379 >> v1:172.21.15.59:6789/0 conn(0x7f613006bde0 legacy=0x7f61300ab9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:06:21.952 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.953+0000 7f612effd700 10 monclient: _finish_hunting 0 2023-09-30T02:06:21.952 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.953+0000 7f612effd700 1 monclient: found mon.noname-b 2023-09-30T02:06:21.953 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.953+0000 7f612effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:21.953 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.953+0000 7f612effd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.59:3301/0 2023-09-30T02:06:21.953 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.953+0000 7f612effd700 1 -- 172.21.15.59:0/16576379 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61300ace90 con 0x7f6130069b10 2023-09-30T02:06:21.955 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.955+0000 7f612effd700 1 --2- 172.21.15.59:0/16576379 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6130069b10 0x7f613006b8a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6130063c80 tx=0x7f612000b4e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3550b58db63cdf09 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:21.955 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.956+0000 7f13aa45f700 1 --2- 172.21.15.59:0/3532691214 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13a40a65b0 0x7f13a40a6980 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f13a40a8600 tx=0x7f139c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46cc2e4372b3393b server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:21.964 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.964+0000 7f612dffb700 1 -- 172.21.15.59:0/16576379 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f612000dac0 con 0x7f6130069b10 2023-09-30T02:06:21.964 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.964+0000 7f612dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:21.964 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.964+0000 7f612dffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:06:21.964 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.964+0000 7f612dffb700 10 monclient: dump: 2023-09-30T02:06:21.964 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:21.964 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:21.964 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:21.964 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:21.964 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:21.965 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:21.965 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:21.965 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:21.965 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:21.965 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:21.965 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.964+0000 7f612dffb700 10 monclient: _finish_auth 0 2023-09-30T02:06:21.965 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.964+0000 7f612dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:21.965 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.964+0000 7f612dffb700 1 -- 172.21.15.59:0/16576379 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f612000dc20 con 0x7f6130069b10 2023-09-30T02:06:21.965 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.964+0000 7f612dffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:21.965 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.964+0000 7f612dffb700 1 -- 172.21.15.59:0/16576379 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6120017410 con 0x7f6130069b10 2023-09-30T02:06:21.965 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.965+0000 7f612dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:21.965 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.965+0000 7f612dffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:06:21.965 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.965+0000 7f612dffb700 10 monclient: dump: 2023-09-30T02:06:21.966 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:21.966 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:21.966 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:21.966 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:21.966 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:21.966 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:21.966 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:21.966 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:21.966 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:21.966 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:21.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.965+0000 7f6135a5d700 5 monclient: authenticate success, global_id 4295 2023-09-30T02:06:21.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.965+0000 7f6135a5d700 10 monclient: get_monmap_and_config success 2023-09-30T02:06:21.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.965+0000 7f6135a5d700 4 set_mon_vals no callback set 2023-09-30T02:06:21.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.966+0000 7f6135a5d700 10 monclient: shutdown 2023-09-30T02:06:21.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.966+0000 7f6135a5d700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:21.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.966+0000 7f6135a5d700 1 -- 172.21.15.59:0/16576379 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6130069b10 msgr2=0x7f613006b8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:21.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.966+0000 7f6135a5d700 1 --2- 172.21.15.59:0/16576379 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6130069b10 0x7f613006b8a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6130063c80 tx=0x7f612000b4e0 comp rx=0 tx=0).stop 2023-09-30T02:06:21.968 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.967+0000 7f13a945d700 1 -- 172.21.15.59:0/3532691214 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f139c00e070 con 0x7f13a40a65b0 2023-09-30T02:06:21.968 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.967+0000 7f13a945d700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:21.968 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.967+0000 7f13a945d700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T02:06:21.968 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.967+0000 7f13a945d700 10 monclient: dump: 2023-09-30T02:06:21.968 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:21.968 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:21.968 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:21.968 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:21.969 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:21.969 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:21.969 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:21.969 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:21.969 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:21.969 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:21.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.967+0000 7f13a945d700 10 monclient: _finish_auth 0 2023-09-30T02:06:21.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.967+0000 7f13a945d700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:21.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.967+0000 7f13a945d700 1 -- 172.21.15.59:0/3532691214 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f139c00bab0 con 0x7f13a40a65b0 2023-09-30T02:06:21.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.967+0000 7f13ac6c3700 5 monclient: authenticate success, global_id 4299 2023-09-30T02:06:21.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.967+0000 7f13ac6c3700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:06:21.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.967+0000 7f13a945d700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:21.970 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.967+0000 7f13a945d700 1 -- 172.21.15.59:0/3532691214 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f139c00be70 con 0x7f13a40a65b0 2023-09-30T02:06:21.970 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.967+0000 7f13a945d700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:21.970 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.967+0000 7f13a945d700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T02:06:21.970 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.967+0000 7f13a945d700 10 monclient: dump: 2023-09-30T02:06:21.970 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:21.970 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:21.970 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:21.970 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:21.970 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:21.970 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:21.970 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:21.971 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:21.971 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:21.971 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:21.971 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.967+0000 7f13ac6c3700 10 monclient: get_monmap_and_config success 2023-09-30T02:06:21.971 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.967+0000 7f13ac6c3700 4 set_mon_vals no callback set 2023-09-30T02:06:21.971 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.967+0000 7f6135a5d700 1 -- 172.21.15.59:0/16576379 shutdown_connections 2023-09-30T02:06:21.971 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.967+0000 7f6135a5d700 1 --2- 172.21.15.59:0/16576379 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6130069b10 0x7f613006b8a0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:21.971 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.967+0000 7f6135a5d700 1 -- 172.21.15.59:0/16576379 >> 172.21.15.59:0/16576379 conn(0x7f61300b08e0 msgr2=0x7f61300b0ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:21.972 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.968+0000 7f13ac6c3700 10 monclient: shutdown 2023-09-30T02:06:21.972 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.968+0000 7f13ac6c3700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:21.972 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.968+0000 7f13ac6c3700 1 -- 172.21.15.59:0/3532691214 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13a40a65b0 msgr2=0x7f13a40a6980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:21.972 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.968+0000 7f13ac6c3700 1 --2- 172.21.15.59:0/3532691214 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13a40a65b0 0x7f13a40a6980 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f13a40a8600 tx=0x7f139c009780 comp rx=0 tx=0).stop 2023-09-30T02:06:21.974 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.975+0000 7f6135a5d700 1 -- 172.21.15.59:0/16576379 shutdown_connections 2023-09-30T02:06:21.974 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.975+0000 7f13ac6c3700 1 -- 172.21.15.59:0/3532691214 shutdown_connections 2023-09-30T02:06:21.975 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.975+0000 7f13ac6c3700 1 --2- 172.21.15.59:0/3532691214 >> v2:172.21.15.110:3300/0 conn(0x7f13a40ab5e0 0x7f13a40aba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:21.975 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.975+0000 7f13ac6c3700 1 --2- 172.21.15.59:0/3532691214 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13a40a65b0 0x7f13a40a6980 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:21.975 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.975+0000 7f13ac6c3700 1 -- 172.21.15.59:0/3532691214 >> 172.21.15.59:0/3532691214 conn(0x7f13a40b0d30 msgr2=0x7f13a40ad250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:21.975 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.976+0000 7f6135a5d700 1 -- 172.21.15.59:0/16576379 wait complete. 2023-09-30T02:06:21.976 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.977+0000 7f6135a5d700 10 monclient: build_initial_monmap 2023-09-30T02:06:21.976 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.977+0000 7f6135a5d700 10 monclient: monmap: 2023-09-30T02:06:21.976 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:21.976 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:06:21.976 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:06:21.976 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:06:21.976 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:21.977 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:06:21.977 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:06:21.977 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:06:21.977 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:06:21.977 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:21.978 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.977+0000 7f6135a5d700 1 librados: starting msgr at 2023-09-30T02:06:21.978 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.977+0000 7f6135a5d700 1 librados: starting objecter 2023-09-30T02:06:21.978 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.977+0000 7f6135a5d700 1 Processor -- start 2023-09-30T02:06:21.978 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.977+0000 7f6135a5d700 1 -- start start 2023-09-30T02:06:21.978 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.977+0000 7f6135a5d700 1 librados: setting wanted keys 2023-09-30T02:06:21.978 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.977+0000 7f6135a5d700 1 librados: calling monclient init 2023-09-30T02:06:21.978 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.977+0000 7f6135a5d700 10 monclient: init 2023-09-30T02:06:21.978 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.978+0000 7f6135a5d700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:21.978 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.978+0000 7f6135a5d700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T02:06:21.978 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.978+0000 7f6135a5d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6130069b10 0x7f6130081400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:21.979 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.978+0000 7f6135a5d700 10 monclient(hunting): picked mon.noname-b con 0x7f6130069b10 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:06:21.979 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.978+0000 7f6135a5d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f613006bde0 0x7f6130081940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:21.979 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.978+0000 7f6135a5d700 10 monclient(hunting): picked mon.noname-a con 0x7f613006bde0 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:06:21.979 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.978+0000 7f6135a5d700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6130087810 0x7f61301638f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:21.979 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.978+0000 7f6135a5d700 10 monclient(hunting): picked mon.noname-c con 0x7f6130087810 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:21.979 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.978+0000 7f6135a5d700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:21.979 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.978+0000 7f6135a5d700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6130163460 con 0x7f613006bde0 2023-09-30T02:06:21.979 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.978+0000 7f6135a5d700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:21.979 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.978+0000 7f6135a5d700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f613007fc40 con 0x7f6130069b10 2023-09-30T02:06:21.979 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.978+0000 7f6135a5d700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:21.979 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.978+0000 7f6135a5d700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f613007fdb0 con 0x7f6130087810 2023-09-30T02:06:21.980 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.978+0000 7f6135a5d700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:21.980 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.978+0000 7f6135a5d700 10 monclient(hunting): authenticate will time out at 1827.280860s 2023-09-30T02:06:21.981 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.981+0000 7f41ac516700 10 monclient: get_monmap_and_config 2023-09-30T02:06:21.981 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.981+0000 7f41ac516700 10 monclient: build_initial_monmap 2023-09-30T02:06:21.981 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.981+0000 7f41ac516700 10 monclient: monmap: 2023-09-30T02:06:21.981 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:21.981 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:21.982 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:06:21.981778+0000 2023-09-30T02:06:21.982 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:06:21.981778+0000 2023-09-30T02:06:21.982 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:21.982 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:21.982 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:06:21.982 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:06:21.982 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:06:21.982 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:06:21.982 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:06:21.982 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:21.983 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.980+0000 7f612f7fe700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6130087810 0x7f61301638f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:21.983 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.980+0000 7f612f7fe700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6130087810 0x7f61301638f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:47628/0 (socket says 172.21.15.59:47628) 2023-09-30T02:06:21.983 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.980+0000 7f612f7fe700 1 -- 172.21.15.59:0/4256937319 learned_addr learned my addr 172.21.15.59:0/4256937319 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:06:21.983 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.980+0000 7f612f7fe700 10 monclient(hunting): get_auth_request con 0x7f6130087810 auth_method 0 2023-09-30T02:06:21.983 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.980+0000 7f612f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:21.983 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.980+0000 7f612f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:21.983 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.980+0000 7f612f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:21.983 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.981+0000 7f612effd700 1 --2- 172.21.15.59:0/4256937319 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6130069b10 0x7f6130081400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:21.983 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.982+0000 7f612effd700 10 monclient(hunting): get_auth_request con 0x7f6130069b10 auth_method 0 2023-09-30T02:06:21.983 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.982+0000 7f612effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:21.984 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.982+0000 7f612effd700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:21.984 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.982+0000 7f612effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:21.984 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.983+0000 7f13ac6c3700 1 -- 172.21.15.59:0/3532691214 shutdown_connections 2023-09-30T02:06:21.984 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.983+0000 7f13ac6c3700 1 -- 172.21.15.59:0/3532691214 wait complete. 2023-09-30T02:06:21.984 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.983+0000 7f612f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:21.984 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.983+0000 7f612f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:21.984 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.983+0000 7f612f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:21.985 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.983+0000 7f612effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:21.985 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.983+0000 7f612effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:21.985 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.983+0000 7f612effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:21.993 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.993+0000 7f41ac516700 1 Processor -- start 2023-09-30T02:06:21.993 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.984+0000 7f13ac6c3700 10 monclient: build_initial_monmap 2023-09-30T02:06:21.993 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.984+0000 7f13ac6c3700 10 monclient: monmap: 2023-09-30T02:06:21.993 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:21.993 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:06:21.993 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:06:21.994 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:06:21.994 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:21.994 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:06:21.994 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:06:21.994 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:06:21.994 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:06:21.994 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:21.994 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.984+0000 7f13ac6c3700 1 librados: starting msgr at 2023-09-30T02:06:21.994 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.984+0000 7f13ac6c3700 1 librados: starting objecter 2023-09-30T02:06:21.994 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.985+0000 7f13ac6c3700 1 Processor -- start 2023-09-30T02:06:21.994 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.985+0000 7f13ac6c3700 1 -- start start 2023-09-30T02:06:21.994 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.985+0000 7f13ac6c3700 1 librados: setting wanted keys 2023-09-30T02:06:21.995 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.985+0000 7f13ac6c3700 1 librados: calling monclient init 2023-09-30T02:06:21.995 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.985+0000 7f13ac6c3700 10 monclient: init 2023-09-30T02:06:21.995 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.986+0000 7f13ac6c3700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:21.995 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.986+0000 7f13ac6c3700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T02:06:21.995 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.986+0000 7f13ac6c3700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f13a40a65b0 0x7f13a4136e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:21.995 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.986+0000 7f13ac6c3700 10 monclient(hunting): picked mon.noname-c con 0x7f13a40a65b0 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:21.995 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.986+0000 7f13ac6c3700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f13a40a9940 0x7f13a415ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:21.995 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.986+0000 7f13ac6c3700 10 monclient(hunting): picked mon.noname-b con 0x7f13a40a9940 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:06:21.995 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.986+0000 7f13ac6c3700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13a40ab5e0 0x7f13a415b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:21.995 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.986+0000 7f13ac6c3700 10 monclient(hunting): picked mon.noname-a con 0x7f13a40ab5e0 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:06:21.995 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.986+0000 7f13ac6c3700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:21.995 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.986+0000 7f13ac6c3700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13a4137830 con 0x7f13a40ab5e0 2023-09-30T02:06:21.996 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.986+0000 7f13ac6c3700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:21.996 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.986+0000 7f13ac6c3700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13a41387e0 con 0x7f13a40a9940 2023-09-30T02:06:21.996 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.986+0000 7f13ac6c3700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:21.996 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.986+0000 7f13ac6c3700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13a415b8b0 con 0x7f13a40a65b0 2023-09-30T02:06:21.996 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.986+0000 7f13ac6c3700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:21.996 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.986+0000 7f13ac6c3700 10 monclient(hunting): authenticate will time out at 1827.288270s 2023-09-30T02:06:21.996 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.990+0000 7f13aac60700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13a40ab5e0 0x7f13a415b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:21.996 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.990+0000 7f13aac60700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13a40ab5e0 0x7f13a415b370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51936/0 (socket says 172.21.15.59:51936) 2023-09-30T02:06:21.996 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.990+0000 7f13aac60700 1 -- 172.21.15.59:0/3513182853 learned_addr learned my addr 172.21.15.59:0/3513182853 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:06:21.996 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.990+0000 7f13aac60700 10 monclient(hunting): get_auth_request con 0x7f13a40ab5e0 auth_method 0 2023-09-30T02:06:21.996 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.990+0000 7f13aac60700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:21.996 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.990+0000 7f13aac60700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:21.997 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.990+0000 7f13aac60700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:21.997 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.992+0000 7f13aac60700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:21.997 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.992+0000 7f13aac60700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:21.997 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.992+0000 7f13aac60700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:21.997 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.987+0000 7f612effd700 10 monclient(hunting): handle_auth_done global_id 4342 payload 824 2023-09-30T02:06:21.997 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.987+0000 7f612effd700 1 -- 172.21.15.59:0/4256937319 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6130087810 msgr2=0x7f61301638f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:21.997 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.987+0000 7f612effd700 1 --2- 172.21.15.59:0/4256937319 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6130087810 0x7f61301638f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:21.997 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.987+0000 7f612effd700 1 -- 172.21.15.59:0/4256937319 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f613006bde0 msgr2=0x7f6130081940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:21.997 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.987+0000 7f612effd700 1 --2- 172.21.15.59:0/4256937319 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f613006bde0 0x7f6130081940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:21.997 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.987+0000 7f612effd700 10 monclient: _finish_hunting 0 2023-09-30T02:06:21.998 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.987+0000 7f612effd700 1 monclient: found mon.noname-b 2023-09-30T02:06:21.998 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.987+0000 7f612effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:21.998 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.987+0000 7f612effd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:06:21.998 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.987+0000 7f612effd700 1 -- 172.21.15.59:0/4256937319 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f612000b780 con 0x7f6130069b10 2023-09-30T02:06:21.998 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.992+0000 7f612effd700 1 --2- 172.21.15.59:0/4256937319 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6130069b10 0x7f6130081400 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6120018000 tx=0x7f6120019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:21.999 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.993+0000 7f41ac516700 1 -- start start 2023-09-30T02:06:21.999 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.993+0000 7f41ac516700 10 monclient: init 2023-09-30T02:06:21.999 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.994+0000 7f41ac516700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:21.999 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.994+0000 7f41ac516700 10 monclient: _add_conns ranks=[2,0,4,3,1] 2023-09-30T02:06:21.999 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.994+0000 7f41ac516700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f41a406a060 0x7f41a406bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:21.999 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.994+0000 7f41ac516700 10 monclient(hunting): picked mon.noname-b con 0x7f41a406a060 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:21.999 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.994+0000 7f41ac516700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f41a413dc00 addr v1:172.21.15.59:6789/0 2023-09-30T02:06:21.999 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.994+0000 7f41ac516700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f41a413a750 0x7f41a413abf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:22.000 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.994+0000 7f41ac516700 10 monclient(hunting): picked mon.noname-c con 0x7f41a413a750 addr v2:172.21.15.110:3300/0 2023-09-30T02:06:22.000 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.994+0000 7f41ac516700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41a406a430 con 0x7f41a413dc00 2023-09-30T02:06:22.000 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.994+0000 7f41ac516700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:22.000 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.994+0000 7f41ac516700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41a406db80 con 0x7f41a413a750 2023-09-30T02:06:22.000 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.994+0000 7f41ac516700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:22.000 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.994+0000 7f41ac516700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41a406c560 con 0x7f41a406a060 2023-09-30T02:06:22.000 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.994+0000 7f41ac516700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:22.000 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.994+0000 7f41ac516700 10 monclient(hunting): authenticate will time out at 1827.296811s 2023-09-30T02:06:22.000 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.999+0000 7f41aaab3700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f41a413a750 0x7f41a413abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:22.000 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.996+0000 7f1f2cd82700 10 monclient: get_monmap_and_config 2023-09-30T02:06:22.001 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.996+0000 7f1f2cd82700 10 monclient: build_initial_monmap 2023-09-30T02:06:22.001 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.996+0000 7f1f2cd82700 10 monclient: monmap: 2023-09-30T02:06:22.001 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:22.001 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:22.001 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:06:21.996644+0000 2023-09-30T02:06:22.001 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:06:21.996644+0000 2023-09-30T02:06:22.001 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:22.001 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:22.001 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:06:22.001 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:06:22.001 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:06:22.002 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:06:22.002 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:06:22.002 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:22.002 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.997+0000 7f1f2cd82700 1 Processor -- start 2023-09-30T02:06:22.002 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.997+0000 7f1f2cd82700 1 -- start start 2023-09-30T02:06:22.002 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.997+0000 7f1f2cd82700 10 monclient: init 2023-09-30T02:06:22.002 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.998+0000 7f1f2cd82700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:22.002 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.998+0000 7f1f2cd82700 10 monclient: _add_conns ranks=[0,3,4,2,1] 2023-09-30T02:06:22.002 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.998+0000 7f1f2cd82700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1f280ae430 addr v1:172.21.15.59:6789/0 2023-09-30T02:06:22.002 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.998+0000 7f1f2cd82700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f1f280ab210 0x7f1f2813e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:22.002 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.998+0000 7f1f2cd82700 10 monclient(hunting): picked mon.noname-a con 0x7f1f280ab210 addr v2:172.21.15.59:3300/0 2023-09-30T02:06:22.003 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.998+0000 7f1f2cd82700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f1f2813ebc0 0x7f1f2813f000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:22.003 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.998+0000 7f1f2cd82700 10 monclient(hunting): picked mon.noname-c con 0x7f1f2813ebc0 addr v2:172.21.15.110:3300/0 2023-09-30T02:06:22.003 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.998+0000 7f1f2cd82700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f280acaf0 con 0x7f1f280ae430 2023-09-30T02:06:22.003 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.998+0000 7f1f2cd82700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:22.003 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.998+0000 7f1f2cd82700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f280ace10 con 0x7f1f280ab210 2023-09-30T02:06:22.003 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.998+0000 7f1f2cd82700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:22.003 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.998+0000 7f1f2cd82700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f2813f540 con 0x7f1f2813ebc0 2023-09-30T02:06:22.003 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.998+0000 7f1f2cd82700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:22.003 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.998+0000 7f1f2cd82700 10 monclient(hunting): authenticate will time out at 1827.300717s 2023-09-30T02:06:22.003 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.997+0000 7f13a9c5e700 1 --2- 172.21.15.59:0/3513182853 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f13a40a9940 0x7f13a415ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:22.004 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.998+0000 7f13a9c5e700 10 monclient(hunting): get_auth_request con 0x7f13a40a9940 auth_method 0 2023-09-30T02:06:22.004 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.998+0000 7f13a9c5e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:22.004 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.998+0000 7f13a9c5e700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:22.004 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.998+0000 7f13a9c5e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:22.004 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.998+0000 7f13aac60700 10 monclient(hunting): handle_auth_done global_id 4302 payload 824 2023-09-30T02:06:22.004 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.998+0000 7f13aac60700 1 -- 172.21.15.59:0/3513182853 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f13a40a65b0 msgr2=0x7f13a4136e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:22.004 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.998+0000 7f13aac60700 1 --2- 172.21.15.59:0/3513182853 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f13a40a65b0 0x7f13a4136e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:22.004 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.998+0000 7f13aac60700 1 -- 172.21.15.59:0/3513182853 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f13a40a9940 msgr2=0x7f13a415ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:22.004 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.998+0000 7f13aac60700 1 --2- 172.21.15.59:0/3513182853 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f13a40a9940 0x7f13a415ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:22.004 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.998+0000 7f13aac60700 10 monclient: _finish_hunting 0 2023-09-30T02:06:22.004 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.998+0000 7f13aac60700 1 monclient: found mon.noname-a 2023-09-30T02:06:22.004 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.998+0000 7f13aac60700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:22.005 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.998+0000 7f13aac60700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.59:3300/0 2023-09-30T02:06:22.005 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.998+0000 7f13aac60700 1 -- 172.21.15.59:0/3513182853 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f139c009a20 con 0x7f13a40ab5e0 2023-09-30T02:06:22.005 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.998+0000 7f6116ffd700 10 client.?.objecter ms_handle_connect 0x7f6130069b10 2023-09-30T02:06:22.005 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.998+0000 7f6116ffd700 10 client.?.objecter resend_mon_ops 2023-09-30T02:06:22.005 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.994+0000 7fb98b07e700 10 monclient: get_monmap_and_config 2023-09-30T02:06:22.005 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.994+0000 7fb98b07e700 10 monclient: build_initial_monmap 2023-09-30T02:06:22.006 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.994+0000 7fb98b07e700 10 monclient: monmap: 2023-09-30T02:06:22.006 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:22.006 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:22.006 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:06:21.995296+0000 2023-09-30T02:06:22.006 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:06:21.995296+0000 2023-09-30T02:06:22.006 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:22.006 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:22.006 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:06:22.006 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:06:22.006 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:06:22.006 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:06:22.006 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:06:22.007 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:22.007 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.997+0000 7fb98b07e700 1 Processor -- start 2023-09-30T02:06:22.007 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.998+0000 7fb98b07e700 1 -- start start 2023-09-30T02:06:22.007 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.998+0000 7fb98b07e700 10 monclient: init 2023-09-30T02:06:22.007 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.999+0000 7fb98b07e700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:22.007 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.999+0000 7fb98b07e700 10 monclient: _add_conns ranks=[3,0,2,1,4] 2023-09-30T02:06:22.007 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.999+0000 7fb98b07e700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fb984069b10 0x7fb98406b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:22.007 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.999+0000 7fb98b07e700 10 monclient(hunting): picked mon.noname-a con 0x7fb984069b10 addr v2:172.21.15.59:3300/0 2023-09-30T02:06:22.007 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.999+0000 7fb98b07e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb984087810 addr v1:172.21.15.59:6789/0 2023-09-30T02:06:22.007 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.999+0000 7fb98b07e700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb98406bde0 0x7fb9840ab9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:22.007 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.999+0000 7fb98b07e700 10 monclient(hunting): picked mon.noname-b con 0x7fb98406bde0 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:22.007 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.999+0000 7fb98b07e700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb984069ee0 con 0x7fb984087810 2023-09-30T02:06:22.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.999+0000 7fb98b07e700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:22.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.999+0000 7fb98b07e700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb98406c260 con 0x7fb984069b10 2023-09-30T02:06:22.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.999+0000 7fb98b07e700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:22.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.999+0000 7fb98b07e700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9840ace90 con 0x7fb98406bde0 2023-09-30T02:06:22.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.999+0000 7fb98b07e700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:22.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:21.999+0000 7fb98b07e700 10 monclient(hunting): authenticate will time out at 1827.301376s 2023-09-30T02:06:22.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.002+0000 7f6116ffd700 1 -- 172.21.15.59:0/4256937319 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6120009040 con 0x7f6130069b10 2023-09-30T02:06:22.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.002+0000 7f6116ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:22.009 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.002+0000 7f6116ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:06:22.009 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.002+0000 7f6116ffd700 10 monclient: dump: 2023-09-30T02:06:22.009 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:22.009 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:22.009 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:22.009 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:22.009 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:22.009 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:22.009 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:22.009 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:22.009 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:22.009 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:22.010 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.002+0000 7f6116ffd700 10 monclient: _finish_auth 0 2023-09-30T02:06:22.010 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.002+0000 7f6116ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:22.010 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.002+0000 7f6116ffd700 1 -- 172.21.15.59:0/4256937319 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6120000c90 con 0x7f6130069b10 2023-09-30T02:06:22.010 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.002+0000 7f6116ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:22.010 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.002+0000 7f6116ffd700 1 -- 172.21.15.59:0/4256937319 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f612001f830 con 0x7f6130069b10 2023-09-30T02:06:22.010 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.002+0000 7f6116ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:22.010 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.002+0000 7f6116ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:06:22.010 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.002+0000 7f6116ffd700 10 monclient: dump: 2023-09-30T02:06:22.010 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:22.010 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:22.010 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:22.011 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:22.011 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:22.011 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:22.011 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:22.011 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:22.011 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:22.011 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:22.011 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.002+0000 7f612cff9700 4 set_mon_vals no callback set 2023-09-30T02:06:22.011 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.002+0000 7f41aaab3700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f41a413a750 0x7f41a413abf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:39022/0 (socket says 172.21.15.59:39022) 2023-09-30T02:06:22.012 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.002+0000 7f41aaab3700 1 -- 172.21.15.59:0/20148303 learned_addr learned my addr 172.21.15.59:0/20148303 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:06:22.012 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.002+0000 7f41aaab3700 10 monclient(hunting): get_auth_request con 0x7f41a413a750 auth_method 0 2023-09-30T02:06:22.012 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.002+0000 7f41aaab3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:22.012 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.002+0000 7f41aaab3700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:22.012 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.002+0000 7f41aaab3700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:22.012 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.003+0000 7f41aaab3700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:22.012 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.003+0000 7f41aaab3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:22.012 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.003+0000 7f41aaab3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:22.013 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.003+0000 7f6135a5d700 5 monclient: authenticate success, global_id 4342 2023-09-30T02:06:22.013 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.003+0000 7f6135a5d700 10 monclient: _renew_subs 2023-09-30T02:06:22.013 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.003+0000 7f6135a5d700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:22.013 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.003+0000 7f6135a5d700 1 -- 172.21.15.59:0/4256937319 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61300a8ec0 con 0x7f6130069b10 2023-09-30T02:06:22.013 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.003+0000 7f6135a5d700 10 client.4342.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:06:22.013 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.003+0000 7f6135a5d700 10 monclient: _renew_subs 2023-09-30T02:06:22.013 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.003+0000 7f6135a5d700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:22.013 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.003+0000 7f6135a5d700 1 -- 172.21.15.59:0/4256937319 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6130163e30 con 0x7f6130069b10 2023-09-30T02:06:22.013 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.003+0000 7f6135a5d700 1 librados: init done 2023-09-30T02:06:22.013 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.004+0000 7f6135a5d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:22.014 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.004+0000 7f6135a5d700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:22.014 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.004+0000 7f6135a5d700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:22.014 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.004+0000 7f6135a5d700 1 -- 172.21.15.59:0/4256937319 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6130072b30 con 0x7f6130069b10 2023-09-30T02:06:22.014 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.006+0000 7f41aaab3700 10 monclient(hunting): handle_auth_done global_id 4348 payload 274 2023-09-30T02:06:22.014 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.006+0000 7f41aaab3700 1 -- 172.21.15.59:0/20148303 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f41a406a060 msgr2=0x7f41a406bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:06:22.014 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.006+0000 7f41aaab3700 1 --2- 172.21.15.59:0/20148303 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f41a406a060 0x7f41a406bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:22.014 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.006+0000 7f41aaab3700 1 -- 172.21.15.59:0/20148303 >> v1:172.21.15.59:6789/0 conn(0x7f41a413dc00 legacy=0x7f41a413e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:06:22.014 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.006+0000 7f41aaab3700 10 monclient: _finish_hunting 0 2023-09-30T02:06:22.014 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.006+0000 7f41aaab3700 1 monclient: found mon.noname-c 2023-09-30T02:06:22.015 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.006+0000 7f41aaab3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:22.015 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.006+0000 7f41aaab3700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3300/0 2023-09-30T02:06:22.015 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.006+0000 7f41aaab3700 1 -- 172.21.15.59:0/20148303 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41a406d480 con 0x7f41a413a750 2023-09-30T02:06:22.015 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.006+0000 7fb98961b700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb98406bde0 0x7fb9840ab9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:22.015 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.007+0000 7f13aac60700 1 --2- 172.21.15.59:0/3513182853 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13a40ab5e0 0x7f13a415b370 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f13a000e9a0 tx=0x7f13a000a330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:22.015 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.007+0000 7f139a7fc700 10 client.?.objecter ms_handle_connect 0x7f13a40ab5e0 2023-09-30T02:06:22.016 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.007+0000 7f139a7fc700 10 client.?.objecter resend_mon_ops 2023-09-30T02:06:22.016 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.009+0000 7fb98961b700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb98406bde0 0x7fb9840ab9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:47662/0 (socket says 172.21.15.59:47662) 2023-09-30T02:06:22.016 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.009+0000 7fb98961b700 1 -- 172.21.15.59:0/1472033880 learned_addr learned my addr 172.21.15.59:0/1472033880 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:06:22.016 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.009+0000 7fb98961b700 10 monclient(hunting): get_auth_request con 0x7fb98406bde0 auth_method 0 2023-09-30T02:06:22.016 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.009+0000 7fb98961b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:22.016 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.009+0000 7fb98961b700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:22.016 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.009+0000 7fb98961b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:22.016 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.010+0000 7fb98961b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:22.017 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.010+0000 7fb98961b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:22.017 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.010+0000 7fb98961b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:22.017 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.013+0000 7f1f25d9b700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f1f280ab210 0x7f1f2813e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:22.017 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.013+0000 7f1f25d9b700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f1f280ab210 0x7f1f2813e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51948/0 (socket says 172.21.15.59:51948) 2023-09-30T02:06:22.017 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.013+0000 7f1f25d9b700 1 -- 172.21.15.59:0/3598850354 learned_addr learned my addr 172.21.15.59:0/3598850354 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:06:22.017 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.013+0000 7f1f25d9b700 10 monclient(hunting): get_auth_request con 0x7f1f280ab210 auth_method 0 2023-09-30T02:06:22.017 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.013+0000 7f1f25d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:22.017 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.013+0000 7f1f25d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:22.018 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.013+0000 7f1f25d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:22.018 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.014+0000 7fb98961b700 10 monclient(hunting): handle_auth_done global_id 4301 payload 274 2023-09-30T02:06:22.018 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.014+0000 7fb98961b700 1 -- 172.21.15.59:0/1472033880 >> v2:172.21.15.59:3300/0 conn(0x7fb984069b10 msgr2=0x7fb98406b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:06:22.018 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.014+0000 7fb98961b700 1 --2- 172.21.15.59:0/1472033880 >> v2:172.21.15.59:3300/0 conn(0x7fb984069b10 0x7fb98406b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:22.018 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.014+0000 7fb98961b700 1 -- 172.21.15.59:0/1472033880 >> v1:172.21.15.59:6789/0 conn(0x7fb984087810 legacy=0x7fb9840ab2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:06:22.018 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.014+0000 7fb98961b700 10 monclient: _finish_hunting 0 2023-09-30T02:06:22.018 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.014+0000 7fb98961b700 1 monclient: found mon.noname-b 2023-09-30T02:06:22.018 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.014+0000 7fb98961b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:22.018 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.014+0000 7fb98961b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.59:3301/0 2023-09-30T02:06:22.019 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.014+0000 7fb98961b700 1 -- 172.21.15.59:0/1472033880 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb98406cd50 con 0x7fb98406bde0 2023-09-30T02:06:22.019 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.015+0000 7f41aaab3700 1 --2- 172.21.15.59:0/20148303 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f41a413a750 0x7f41a413abf0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f419c009d10 tx=0x7f419c009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cf622a49ae073af5 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:22.019 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.015+0000 7f1f25d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:22.019 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.015+0000 7f1f25d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:22.019 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.015+0000 7f1f25d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:22.019 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.016+0000 7f1f26d9d700 1 --2- 172.21.15.59:0/3598850354 >> v2:172.21.15.110:3300/0 conn(0x7f1f2813ebc0 0x7f1f2813f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:22.020 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.016+0000 7fb98961b700 1 --2- 172.21.15.59:0/1472033880 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb98406bde0 0x7fb9840ab9e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb97c0061a0 tx=0x7fb97c007620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5906b6802b89f34 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:22.020 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.019+0000 7f1f25d9b700 10 monclient(hunting): handle_auth_done global_id 4305 payload 274 2023-09-30T02:06:22.020 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.019+0000 7f1f25d9b700 1 -- 172.21.15.59:0/3598850354 >> v2:172.21.15.110:3300/0 conn(0x7f1f2813ebc0 msgr2=0x7f1f2813f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:06:22.020 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.019+0000 7f1f25d9b700 1 --2- 172.21.15.59:0/3598850354 >> v2:172.21.15.110:3300/0 conn(0x7f1f2813ebc0 0x7f1f2813f000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:22.020 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.019+0000 7f1f25d9b700 1 -- 172.21.15.59:0/3598850354 >> v1:172.21.15.59:6789/0 conn(0x7f1f280ae430 legacy=0x7f1f280aab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:06:22.020 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.019+0000 7f1f25d9b700 10 monclient: _finish_hunting 0 2023-09-30T02:06:22.020 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.019+0000 7f1f25d9b700 1 monclient: found mon.noname-a 2023-09-30T02:06:22.021 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.019+0000 7f1f25d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:22.021 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.019+0000 7f1f25d9b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.59:3300/0 2023-09-30T02:06:22.021 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.019+0000 7f1f25d9b700 1 -- 172.21.15.59:0/3598850354 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f280ae800 con 0x7f1f280ab210 2023-09-30T02:06:22.023 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.023+0000 7fb9837fe700 1 -- 172.21.15.59:0/1472033880 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb97c007e60 con 0x7fb98406bde0 2023-09-30T02:06:22.023 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.023+0000 7fb9837fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:22.023 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.023+0000 7fb9837fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:06:22.023 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.023+0000 7fb9837fe700 10 monclient: dump: 2023-09-30T02:06:22.023 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:22.023 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:22.023 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:22.024 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:22.024 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:22.024 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:22.024 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:22.024 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:22.024 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:22.024 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:22.024 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.023+0000 7fb9837fe700 10 monclient: _finish_auth 0 2023-09-30T02:06:22.024 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.023+0000 7fb9837fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:22.024 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.023+0000 7fb9837fe700 1 -- 172.21.15.59:0/1472033880 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb97c0146e0 con 0x7fb98406bde0 2023-09-30T02:06:22.024 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.023+0000 7fb98b07e700 5 monclient: authenticate success, global_id 4301 2023-09-30T02:06:22.024 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.023+0000 7fb98b07e700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:06:22.024 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.023+0000 7fb9837fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:22.025 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.023+0000 7fb98b07e700 10 monclient: get_monmap_and_config success 2023-09-30T02:06:22.025 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.023+0000 7fb98b07e700 4 set_mon_vals no callback set 2023-09-30T02:06:22.025 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.023+0000 7fb9837fe700 1 -- 172.21.15.59:0/1472033880 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb97c014aa0 con 0x7fb98406bde0 2023-09-30T02:06:22.025 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.023+0000 7fb9837fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:22.025 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.023+0000 7fb9837fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:06:22.025 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.023+0000 7fb9837fe700 10 monclient: dump: 2023-09-30T02:06:22.025 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:22.025 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:22.025 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:22.025 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:22.025 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:22.025 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:22.025 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:22.026 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:22.026 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:22.026 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:22.026 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.024+0000 7fb98b07e700 10 monclient: shutdown 2023-09-30T02:06:22.026 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.024+0000 7fb98b07e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:22.026 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.024+0000 7fb98b07e700 1 -- 172.21.15.59:0/1472033880 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb98406bde0 msgr2=0x7fb9840ab9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:22.026 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.024+0000 7fb98b07e700 1 --2- 172.21.15.59:0/1472033880 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb98406bde0 0x7fb9840ab9e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb97c0061a0 tx=0x7fb97c007620 comp rx=0 tx=0).stop 2023-09-30T02:06:22.027 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.025+0000 7fb98b07e700 1 -- 172.21.15.59:0/1472033880 shutdown_connections 2023-09-30T02:06:22.027 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.025+0000 7fb98b07e700 1 --2- 172.21.15.59:0/1472033880 >> v2:172.21.15.59:3300/0 conn(0x7fb984069b10 0x7fb98406b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:22.027 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.025+0000 7fb98b07e700 1 --2- 172.21.15.59:0/1472033880 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb98406bde0 0x7fb9840ab9e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:22.027 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.025+0000 7fb98b07e700 1 -- 172.21.15.59:0/1472033880 >> 172.21.15.59:0/1472033880 conn(0x7fb9840b08e0 msgr2=0x7fb9840b0ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:22.028 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.025+0000 7fb98b07e700 1 -- 172.21.15.59:0/1472033880 shutdown_connections 2023-09-30T02:06:22.028 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.025+0000 7fb98b07e700 1 -- 172.21.15.59:0/1472033880 wait complete. 2023-09-30T02:06:22.028 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.026+0000 7fb98b07e700 10 monclient: build_initial_monmap 2023-09-30T02:06:22.028 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.026+0000 7fb98b07e700 10 monclient: monmap: 2023-09-30T02:06:22.028 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:22.028 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:06:22.028 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:06:22.028 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:06:22.028 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:22.028 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:06:22.028 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:06:22.028 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:06:22.029 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:06:22.029 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:22.029 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.026+0000 7fb98b07e700 1 librados: starting msgr at 2023-09-30T02:06:22.029 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.026+0000 7fb98b07e700 1 librados: starting objecter 2023-09-30T02:06:22.029 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.027+0000 7fb98b07e700 1 Processor -- start 2023-09-30T02:06:22.029 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.027+0000 7fb98b07e700 1 -- start start 2023-09-30T02:06:22.029 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.027+0000 7fb98b07e700 1 librados: setting wanted keys 2023-09-30T02:06:22.029 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.027+0000 7fb98b07e700 1 librados: calling monclient init 2023-09-30T02:06:22.029 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.027+0000 7fb98b07e700 10 monclient: init 2023-09-30T02:06:22.029 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.028+0000 7fb98b07e700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:22.029 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.028+0000 7fb98b07e700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T02:06:22.029 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.028+0000 7fb98b07e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb984069b10 0x7fb984163900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:22.029 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.028+0000 7fb98b07e700 10 monclient(hunting): picked mon.noname-a con 0x7fb984069b10 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:06:22.030 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.028+0000 7fb98b07e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb98406bde0 0x7fb9840a8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:22.030 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.028+0000 7fb98b07e700 10 monclient(hunting): picked mon.noname-b con 0x7fb98406bde0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:06:22.030 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.028+0000 7fb98b07e700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb984087810 0x7fb9840a90d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:22.030 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.028+0000 7fb98b07e700 10 monclient(hunting): picked mon.noname-c con 0x7fb984087810 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:22.030 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.028+0000 7fb98b07e700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:22.030 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.028+0000 7fb98b07e700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9840a8700 con 0x7fb984069b10 2023-09-30T02:06:22.030 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.028+0000 7fb98b07e700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:22.030 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.028+0000 7fb98b07e700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9840a7360 con 0x7fb98406bde0 2023-09-30T02:06:22.030 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.028+0000 7fb98b07e700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:22.030 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.028+0000 7fb98b07e700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9840a7520 con 0x7fb984087810 2023-09-30T02:06:22.030 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.028+0000 7fb98b07e700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:22.030 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.028+0000 7fb98b07e700 10 monclient(hunting): authenticate will time out at 1827.330481s 2023-09-30T02:06:22.031 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.030+0000 7fb98961b700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb984087810 0x7fb9840a90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:22.032 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.033+0000 7f1f25d9b700 1 --2- 172.21.15.59:0/3598850354 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f280ab210 0x7f1f2813e680 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f1f280ac200 tx=0x7f1f1c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a50907e353a82717 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:22.034 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.034+0000 7f6116ffd700 1 -- 172.21.15.59:0/4256937319 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f6120015070 con 0x7f6130069b10 2023-09-30T02:06:22.034 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.035+0000 7f6116ffd700 1 --2- 172.21.15.59:0/4256937319 >> v2:172.21.15.110:6800/115870 conn(0x7f611c068960 0x7f611c06ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:22.034 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.035+0000 7f612e7fc700 1 --2- 172.21.15.59:0/4256937319 >> v2:172.21.15.110:6800/115870 conn(0x7f611c068960 0x7f611c06ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:22.034 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.035+0000 7f612e7fc700 10 monclient: get_auth_request con 0x7f611c068960 auth_method 0 2023-09-30T02:06:22.035 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.036+0000 7f612e7fc700 1 --2- 172.21.15.59:0/4256937319 >> v2:172.21.15.110:6800/115870 conn(0x7f611c068960 0x7f611c06ae10 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f61301611a0 tx=0x7f6128009350 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:22.035 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.036+0000 7fb98961b700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb984087810 0x7fb9840a90d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:47674/0 (socket says 172.21.15.59:47674) 2023-09-30T02:06:22.035 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.036+0000 7fb98961b700 1 -- 172.21.15.59:0/3534270498 learned_addr learned my addr 172.21.15.59:0/3534270498 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:06:22.035 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.036+0000 7fb98961b700 10 monclient(hunting): get_auth_request con 0x7fb984087810 auth_method 0 2023-09-30T02:06:22.036 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.036+0000 7fb98961b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:22.036 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.036+0000 7fb98961b700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:22.036 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.036+0000 7fb98961b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:22.036 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.036+0000 7f6116ffd700 10 client.4342.objecter ms_handle_connect 0x7f611c068960 2023-09-30T02:06:22.036 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.036+0000 7fb98961b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:22.036 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.036+0000 7fb98961b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:22.036 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.036+0000 7fb98961b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:22.044 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.041+0000 7fb983fff700 1 --2- 172.21.15.59:0/3534270498 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb98406bde0 0x7fb9840a8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:22.044 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.041+0000 7fb988e1a700 1 --2- 172.21.15.59:0/3534270498 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb984069b10 0x7fb984163900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:22.046 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.047+0000 7fe2fb2fc700 10 monclient: get_monmap_and_config 2023-09-30T02:06:22.046 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.047+0000 7fe2fb2fc700 10 monclient: build_initial_monmap 2023-09-30T02:06:22.046 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.047+0000 7fe2fb2fc700 10 monclient: monmap: 2023-09-30T02:06:22.046 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:22.046 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:22.047 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:06:22.047452+0000 2023-09-30T02:06:22.047 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:06:22.047452+0000 2023-09-30T02:06:22.047 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:22.047 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:22.047 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:06:22.047 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:06:22.047 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:06:22.047 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:06:22.047 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:06:22.047 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:22.048 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.047+0000 7fe2fb2fc700 1 Processor -- start 2023-09-30T02:06:22.048 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.048+0000 7fe2fb2fc700 1 -- start start 2023-09-30T02:06:22.048 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.048+0000 7fe2fb2fc700 10 monclient: init 2023-09-30T02:06:22.048 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.049+0000 7fe2fb2fc700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:22.048 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.049+0000 7fe2fb2fc700 10 monclient: _add_conns ranks=[4,2,1,0,3] 2023-09-30T02:06:22.049 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.049+0000 7fb98961b700 10 monclient(hunting): handle_auth_done global_id 4304 payload 824 2023-09-30T02:06:22.049 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.049+0000 7fb98961b700 1 -- 172.21.15.59:0/3534270498 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb98406bde0 msgr2=0x7fb9840a8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:22.049 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.049+0000 7fb98961b700 1 --2- 172.21.15.59:0/3534270498 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb98406bde0 0x7fb9840a8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:22.049 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.049+0000 7fb98961b700 1 -- 172.21.15.59:0/3534270498 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb984069b10 msgr2=0x7fb984163900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:22.049 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.049+0000 7fb98961b700 1 --2- 172.21.15.59:0/3534270498 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb984069b10 0x7fb984163900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:22.049 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.049+0000 7fb98961b700 10 monclient: _finish_hunting 0 2023-09-30T02:06:22.049 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.049+0000 7fb98961b700 1 monclient: found mon.noname-c 2023-09-30T02:06:22.049 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.049+0000 7fb98961b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:22.049 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.049+0000 7fb98961b700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.59:3301/0 2023-09-30T02:06:22.049 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.049+0000 7fb98961b700 1 -- 172.21.15.59:0/3534270498 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb97c0078a0 con 0x7fb984087810 2023-09-30T02:06:22.050 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.049+0000 7fe2fb2fc700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe2f40b2f70 0x7fe2f406e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:22.050 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.049+0000 7fe2fb2fc700 10 monclient(hunting): picked mon.noname-c con 0x7fe2f40b2f70 addr v2:172.21.15.110:3300/0 2023-09-30T02:06:22.050 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.049+0000 7fe2fb2fc700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe2f406dd20 0x7fe2f413f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:22.050 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.049+0000 7fe2fb2fc700 10 monclient(hunting): picked mon.noname-b con 0x7fe2f406dd20 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:22.050 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.049+0000 7fe2fb2fc700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe2f40b3340 addr v1:172.21.15.110:6789/0 2023-09-30T02:06:22.050 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.049+0000 7fe2fb2fc700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2f4139cc0 con 0x7fe2f40b3340 2023-09-30T02:06:22.050 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.049+0000 7fe2fb2fc700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:22.050 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.049+0000 7fe2fb2fc700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2f406f1c0 con 0x7fe2f40b2f70 2023-09-30T02:06:22.051 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.049+0000 7fe2fb2fc700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:22.051 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.049+0000 7fe2fb2fc700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f406f3c0 con 0x7fe2f406dd20 2023-09-30T02:06:22.051 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.049+0000 7fe2fb2fc700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:22.051 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.049+0000 7fe2fb2fc700 10 monclient(hunting): authenticate will time out at 1827.351278s 2023-09-30T02:06:22.052 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.050+0000 7f228890e700 10 monclient: get_monmap_and_config 2023-09-30T02:06:22.052 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.050+0000 7f228890e700 10 monclient: build_initial_monmap 2023-09-30T02:06:22.052 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.050+0000 7f228890e700 10 monclient: monmap: 2023-09-30T02:06:22.052 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:22.052 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:22.052 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:06:22.051046+0000 2023-09-30T02:06:22.052 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:06:22.051046+0000 2023-09-30T02:06:22.053 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:22.053 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:22.053 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:06:22.053 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:06:22.053 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:06:22.053 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:06:22.053 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:06:22.053 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:22.053 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.051+0000 7f228890e700 1 Processor -- start 2023-09-30T02:06:22.053 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.051+0000 7f228890e700 1 -- start start 2023-09-30T02:06:22.053 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.051+0000 7f228890e700 10 monclient: init 2023-09-30T02:06:22.053 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.052+0000 7f228890e700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:22.054 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.052+0000 7f228890e700 10 monclient: _add_conns ranks=[4,0,2,1,3] 2023-09-30T02:06:22.054 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.052+0000 7f228890e700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f22840a5b70 0x7f22840b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:22.054 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.052+0000 7f228890e700 10 monclient(hunting): picked mon.noname-c con 0x7f22840a5b70 addr v2:172.21.15.110:3300/0 2023-09-30T02:06:22.054 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.052+0000 7f228890e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f22840a5f40 addr v1:172.21.15.59:6789/0 2023-09-30T02:06:22.054 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.052+0000 7f228890e700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2284142290 0x7f228413af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:22.054 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.052+0000 7f228890e700 10 monclient(hunting): picked mon.noname-b con 0x7f2284142290 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:22.054 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.052+0000 7f228890e700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22840a75a0 con 0x7f22840a5f40 2023-09-30T02:06:22.054 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.052+0000 7f228890e700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:22.054 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.052+0000 7f228890e700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22841426d0 con 0x7f22840a5b70 2023-09-30T02:06:22.054 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.052+0000 7f228890e700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:22.054 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.052+0000 7f228890e700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22840a63e0 con 0x7f2284142290 2023-09-30T02:06:22.054 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.052+0000 7f228890e700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:22.055 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.052+0000 7f228890e700 10 monclient(hunting): authenticate will time out at 1827.354369s 2023-09-30T02:06:22.055 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.055+0000 7fb98961b700 1 --2- 172.21.15.59:0/3534270498 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb984087810 0x7fb9840a90d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb97c00ba70 tx=0x7fb97c01b000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:22.055 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.055+0000 7fb980ff9700 10 client.?.objecter ms_handle_connect 0x7fb984087810 2023-09-30T02:06:22.055 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.055+0000 7fb980ff9700 10 client.?.objecter resend_mon_ops 2023-09-30T02:06:22.057 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.057+0000 7fe2f8897700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe2f406dd20 0x7fe2f413f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:22.057 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.058+0000 7f2282d9d700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2284142290 0x7f228413af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:22.058 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.058+0000 7fe2f8897700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe2f406dd20 0x7fe2f413f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:47688/0 (socket says 172.21.15.59:47688) 2023-09-30T02:06:22.058 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.058+0000 7fe2f8897700 1 -- 172.21.15.59:0/2673347399 learned_addr learned my addr 172.21.15.59:0/2673347399 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:06:22.058 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.058+0000 7fe2f8897700 10 monclient(hunting): get_auth_request con 0x7fe2f406dd20 auth_method 0 2023-09-30T02:06:22.058 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.058+0000 7fe2f8897700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:22.058 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.058+0000 7fe2f8897700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:22.058 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.058+0000 7fe2f8897700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:22.059 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.059+0000 7f2282d9d700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2284142290 0x7f228413af30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:47698/0 (socket says 172.21.15.59:47698) 2023-09-30T02:06:22.059 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.059+0000 7f2282d9d700 1 -- 172.21.15.59:0/1253880262 learned_addr learned my addr 172.21.15.59:0/1253880262 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:06:22.059 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.059+0000 7f2282d9d700 10 monclient(hunting): get_auth_request con 0x7f2284142290 auth_method 0 2023-09-30T02:06:22.059 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.059+0000 7f2282d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:22.059 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.059+0000 7f2282d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:22.059 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.059+0000 7f2282d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:22.060 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.059+0000 7fe2f8897700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:22.060 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.059+0000 7fe2f8897700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:22.060 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.059+0000 7fe2f8897700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:22.060 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.060+0000 7f2282d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:22.060 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.060+0000 7f2282d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:22.060 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.060+0000 7f2282d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:22.061 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.062+0000 7f2282d9d700 10 monclient(hunting): handle_auth_done global_id 4310 payload 274 2023-09-30T02:06:22.061 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.062+0000 7f2282d9d700 1 -- 172.21.15.59:0/1253880262 >> v2:172.21.15.110:3300/0 conn(0x7f22840a5b70 msgr2=0x7f22840b1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:06:22.061 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.062+0000 7f2282d9d700 1 --2- 172.21.15.59:0/1253880262 >> v2:172.21.15.110:3300/0 conn(0x7f22840a5b70 0x7f22840b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:22.061 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.062+0000 7f2282d9d700 1 -- 172.21.15.59:0/1253880262 >> v1:172.21.15.59:6789/0 conn(0x7f22840a5f40 legacy=0x7f2284141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:06:22.061 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.062+0000 7f2282d9d700 10 monclient: _finish_hunting 0 2023-09-30T02:06:22.061 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.062+0000 7f2282d9d700 1 monclient: found mon.noname-b 2023-09-30T02:06:22.061 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.062+0000 7f2282d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:22.061 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.062+0000 7f2282d9d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.59:3301/0 2023-09-30T02:06:22.062 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.062+0000 7f2282d9d700 1 -- 172.21.15.59:0/1253880262 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22840af3f0 con 0x7f2284142290 2023-09-30T02:06:22.063 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.063+0000 7fe2f8897700 10 monclient(hunting): handle_auth_done global_id 4307 payload 274 2023-09-30T02:06:22.063 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.063+0000 7fe2f8897700 1 -- 172.21.15.59:0/2673347399 >> v2:172.21.15.110:3300/0 conn(0x7fe2f40b2f70 msgr2=0x7fe2f406e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:06:22.063 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.063+0000 7fe2f8897700 1 --2- 172.21.15.59:0/2673347399 >> v2:172.21.15.110:3300/0 conn(0x7fe2f40b2f70 0x7fe2f406e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:22.063 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.063+0000 7fe2f8897700 1 -- 172.21.15.59:0/2673347399 >> v1:172.21.15.110:6789/0 conn(0x7fe2f40b3340 legacy=0x7fe2f40abd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:06:22.063 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.063+0000 7fe2f8897700 10 monclient: _finish_hunting 0 2023-09-30T02:06:22.063 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.063+0000 7fe2f8897700 1 monclient: found mon.noname-b 2023-09-30T02:06:22.063 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.063+0000 7fe2f8897700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:22.063 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.063+0000 7fe2f8897700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.59:3301/0 2023-09-30T02:06:22.063 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.063+0000 7fe2f8897700 1 -- 172.21.15.59:0/2673347399 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2f406f6a0 con 0x7fe2f406dd20 2023-09-30T02:06:22.067 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.068+0000 7f2282d9d700 1 --2- 172.21.15.59:0/1253880262 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2284142290 0x7f228413af30 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f227c009f20 tx=0x7f227c009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c694f2df9f6c246 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:22.068 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.069+0000 7fe2f8897700 1 --2- 172.21.15.59:0/2673347399 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe2f406dd20 0x7fe2f413f600 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe2e40099b0 tx=0x7fe2e4009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a7e6ae3613d6b706 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:22.129 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.128+0000 7f1f2559a700 1 -- 172.21.15.59:0/3598850354 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f1c00b8b0 con 0x7f1f280ab210 2023-09-30T02:06:22.130 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.129+0000 7f1f2559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:22.130 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.129+0000 7f1f2559a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T02:06:22.130 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.129+0000 7f1f2559a700 10 monclient: dump: 2023-09-30T02:06:22.130 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:22.130 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:22.130 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:22.130 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:22.130 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:22.130 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:22.130 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:22.130 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:22.131 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:22.131 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:22.131 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.129+0000 7f1f2559a700 10 monclient: _finish_auth 0 2023-09-30T02:06:22.131 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.129+0000 7f1f2559a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:22.131 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.129+0000 7f1f2559a700 1 -- 172.21.15.59:0/3598850354 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f1c00ba10 con 0x7f1f280ab210 2023-09-30T02:06:22.131 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.129+0000 7f1f2559a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:22.131 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.129+0000 7f1f2559a700 1 -- 172.21.15.59:0/3598850354 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f1c00bdb0 con 0x7f1f280ab210 2023-09-30T02:06:22.131 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.129+0000 7f1f2559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:22.131 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.129+0000 7f1f2559a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T02:06:22.131 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.129+0000 7f1f2559a700 10 monclient: dump: 2023-09-30T02:06:22.131 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:22.131 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:22.131 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:22.132 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:22.132 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:22.132 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:22.132 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:22.132 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:22.132 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:22.132 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:22.132 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.129+0000 7f1f2cd82700 5 monclient: authenticate success, global_id 4305 2023-09-30T02:06:22.132 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.129+0000 7f1f2cd82700 10 monclient: get_monmap_and_config success 2023-09-30T02:06:22.132 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.129+0000 7f1f2cd82700 4 set_mon_vals no callback set 2023-09-30T02:06:22.132 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.130+0000 7f1f2cd82700 10 monclient: shutdown 2023-09-30T02:06:22.132 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.130+0000 7f1f2cd82700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:22.132 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.130+0000 7f1f2cd82700 1 -- 172.21.15.59:0/3598850354 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f280ab210 msgr2=0x7f1f2813e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:22.133 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.130+0000 7f1f2cd82700 1 --2- 172.21.15.59:0/3598850354 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f280ab210 0x7f1f2813e680 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f1f280ac200 tx=0x7f1f1c009250 comp rx=0 tx=0).stop 2023-09-30T02:06:22.133 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.130+0000 7f139a7fc700 1 -- 172.21.15.59:0/3513182853 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13a000a7c0 con 0x7f13a40ab5e0 2023-09-30T02:06:22.133 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.130+0000 7f139a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:22.133 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.130+0000 7f139a7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T02:06:22.133 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.130+0000 7f139a7fc700 10 monclient: dump: 2023-09-30T02:06:22.133 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:22.133 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:22.133 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:22.133 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:22.133 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:22.133 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:22.134 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:22.134 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:22.134 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:22.134 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:22.134 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.130+0000 7f139a7fc700 10 monclient: _finish_auth 0 2023-09-30T02:06:22.134 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.130+0000 7f139a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:22.134 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.130+0000 7f139a7fc700 1 -- 172.21.15.59:0/3513182853 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13a000c950 con 0x7f13a40ab5e0 2023-09-30T02:06:22.134 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.130+0000 7f13ac6c3700 5 monclient: authenticate success, global_id 4302 2023-09-30T02:06:22.134 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.130+0000 7f13ac6c3700 10 monclient: _renew_subs 2023-09-30T02:06:22.134 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.130+0000 7f13ac6c3700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:06:22.134 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.130+0000 7f13ac6c3700 1 -- 172.21.15.59:0/3513182853 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13a4136650 con 0x7f13a40ab5e0 2023-09-30T02:06:22.134 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.130+0000 7f139a7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:22.135 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.130+0000 7f139a7fc700 1 -- 172.21.15.59:0/3513182853 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13a000cd10 con 0x7f13a40ab5e0 2023-09-30T02:06:22.135 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.130+0000 7f139a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:22.135 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.130+0000 7f139a7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T02:06:22.135 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.130+0000 7f139a7fc700 10 monclient: dump: 2023-09-30T02:06:22.135 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:22.135 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:22.135 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:22.135 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:22.135 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:22.135 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:22.135 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:22.135 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:22.135 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:22.135 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:22.136 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.130+0000 7f13ac6c3700 10 client.4302.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:06:22.136 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.130+0000 7f13ac6c3700 10 monclient: _renew_subs 2023-09-30T02:06:22.136 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.130+0000 7f13ac6c3700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:06:22.136 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.130+0000 7f13ac6c3700 1 -- 172.21.15.59:0/3513182853 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13a4138c00 con 0x7f13a40ab5e0 2023-09-30T02:06:22.136 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.130+0000 7f13ac6c3700 1 librados: init done 2023-09-30T02:06:22.136 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.130+0000 7f1f2cd82700 1 -- 172.21.15.59:0/3598850354 shutdown_connections 2023-09-30T02:06:22.136 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.130+0000 7f1f2cd82700 1 --2- 172.21.15.59:0/3598850354 >> v2:172.21.15.110:3300/0 conn(0x7f1f2813ebc0 0x7f1f2813f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:22.136 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.130+0000 7f1f2cd82700 1 --2- 172.21.15.59:0/3598850354 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f280ab210 0x7f1f2813e680 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:22.137 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.130+0000 7f1f2cd82700 1 -- 172.21.15.59:0/3598850354 >> 172.21.15.59:0/3598850354 conn(0x7f1f2806a030 msgr2=0x7f1f28068730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:22.137 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.130+0000 7f139bfff700 4 set_mon_vals no callback set 2023-09-30T02:06:22.137 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.131+0000 7f1f2cd82700 1 -- 172.21.15.59:0/3598850354 shutdown_connections 2023-09-30T02:06:22.138 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.131+0000 7f13ac6c3700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:22.138 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.131+0000 7f13ac6c3700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:22.138 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.131+0000 7f13ac6c3700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:06:22.138 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.131+0000 7f13ac6c3700 1 -- 172.21.15.59:0/3513182853 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f138c0050b0 con 0x7f13a40ab5e0 2023-09-30T02:06:22.138 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.131+0000 7f1f2cd82700 1 -- 172.21.15.59:0/3598850354 wait complete. 2023-09-30T02:06:22.138 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.132+0000 7f1f2cd82700 10 monclient: build_initial_monmap 2023-09-30T02:06:22.138 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.132+0000 7f1f2cd82700 10 monclient: monmap: 2023-09-30T02:06:22.139 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:22.139 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:06:22.139 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:06:22.139 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:06:22.139 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:22.139 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:06:22.139 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:06:22.139 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:06:22.139 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:06:22.139 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:22.140 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.132+0000 7fbccc958700 10 monclient: get_monmap_and_config 2023-09-30T02:06:22.140 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.132+0000 7fbccc958700 10 monclient: build_initial_monmap 2023-09-30T02:06:22.140 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.132+0000 7fbccc958700 10 monclient: monmap: 2023-09-30T02:06:22.140 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:22.140 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:22.140 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:06:22.133164+0000 2023-09-30T02:06:22.140 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:06:22.133164+0000 2023-09-30T02:06:22.140 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:22.140 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:22.141 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:06:22.141 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:06:22.141 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:06:22.141 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:06:22.141 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:06:22.141 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:22.141 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.133+0000 7fbccc958700 1 Processor -- start 2023-09-30T02:06:22.142 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.132+0000 7f1f2cd82700 1 librados: starting msgr at 2023-09-30T02:06:22.142 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.132+0000 7f1f2cd82700 1 librados: starting objecter 2023-09-30T02:06:22.142 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.133+0000 7f1f2cd82700 1 Processor -- start 2023-09-30T02:06:22.142 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.133+0000 7f1f2cd82700 1 -- start start 2023-09-30T02:06:22.142 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.133+0000 7f1f2cd82700 1 librados: setting wanted keys 2023-09-30T02:06:22.143 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.133+0000 7f1f2cd82700 1 librados: calling monclient init 2023-09-30T02:06:22.143 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.133+0000 7f1f2cd82700 10 monclient: init 2023-09-30T02:06:22.143 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.133+0000 7f1f2cd82700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:22.143 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.133+0000 7f1f2cd82700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T02:06:22.143 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.133+0000 7f1f2cd82700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f280ab210 0x7f1f2813e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:22.143 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.133+0000 7f1f2cd82700 10 monclient(hunting): picked mon.noname-a con 0x7f1f280ab210 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:06:22.143 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.133+0000 7f1f2cd82700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1f280ae430 0x7f1f28155a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:22.143 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.133+0000 7f1f2cd82700 10 monclient(hunting): picked mon.noname-b con 0x7f1f280ae430 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:06:22.143 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.133+0000 7f1f2cd82700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1f2813ebc0 0x7f1f280a7200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:22.143 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.133+0000 7f1f2cd82700 10 monclient(hunting): picked mon.noname-c con 0x7f1f2813ebc0 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:22.143 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.133+0000 7f1f2cd82700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:22.144 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.133+0000 7f1f2cd82700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f280a79f0 con 0x7f1f280ab210 2023-09-30T02:06:22.144 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.133+0000 7f1f2cd82700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:22.144 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.133+0000 7f1f2cd82700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f280a7740 con 0x7f1f280ae430 2023-09-30T02:06:22.144 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.133+0000 7f1f2cd82700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:22.144 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.133+0000 7f1f2cd82700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f280a9580 con 0x7f1f2813ebc0 2023-09-30T02:06:22.144 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.133+0000 7f1f2cd82700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:22.144 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.133+0000 7f1f2cd82700 10 monclient(hunting): authenticate will time out at 1827.435975s 2023-09-30T02:06:22.144 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.133+0000 7fbccc958700 1 -- start start 2023-09-30T02:06:22.145 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.133+0000 7fbccc958700 10 monclient: init 2023-09-30T02:06:22.145 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.134+0000 7fbccc958700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:22.145 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.134+0000 7fbccc958700 10 monclient: _add_conns ranks=[0,1,3,4,2] 2023-09-30T02:06:22.145 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.134+0000 7fbccc958700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbcc80b2f70 addr v1:172.21.15.59:6789/0 2023-09-30T02:06:22.145 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.134+0000 7fbccc958700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbcc806dd20 addr v1:172.21.15.110:6789/0 2023-09-30T02:06:22.145 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.134+0000 7fbccc958700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fbcc80b3340 0x7fbcc806e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:22.145 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.134+0000 7fbccc958700 10 monclient(hunting): picked mon.noname-a con 0x7fbcc80b3340 addr v2:172.21.15.59:3300/0 2023-09-30T02:06:22.145 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.134+0000 7fbccc958700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcc8139cc0 con 0x7fbcc80b2f70 2023-09-30T02:06:22.145 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.134+0000 7fbccc958700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcc806f3c0 con 0x7fbcc806dd20 2023-09-30T02:06:22.145 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.134+0000 7fbccc958700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:22.145 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.134+0000 7fbccc958700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcc806f6a0 con 0x7fbcc80b3340 2023-09-30T02:06:22.145 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.134+0000 7fbccc958700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:22.146 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.134+0000 7fbccc958700 10 monclient(hunting): authenticate will time out at 1827.436918s 2023-09-30T02:06:22.146 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.136+0000 7fb980ff9700 1 -- 172.21.15.59:0/3534270498 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb97c005e80 con 0x7fb984087810 2023-09-30T02:06:22.147 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.136+0000 7fb980ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:22.147 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.137+0000 7fb980ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:06:22.147 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.137+0000 7fb980ff9700 10 monclient: dump: 2023-09-30T02:06:22.147 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:22.147 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:22.147 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:22.147 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:22.147 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:22.147 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:22.147 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:22.147 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:22.148 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:22.148 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:22.148 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.137+0000 7fb980ff9700 10 monclient: _finish_auth 0 2023-09-30T02:06:22.148 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.137+0000 7fb980ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:22.148 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.137+0000 7fb98b07e700 5 monclient: authenticate success, global_id 4304 2023-09-30T02:06:22.148 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.137+0000 7fb98b07e700 10 monclient: _renew_subs 2023-09-30T02:06:22.148 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.137+0000 7fb98b07e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:06:22.148 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.137+0000 7fb98b07e700 1 -- 172.21.15.59:0/3534270498 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9840a54e0 con 0x7fb984087810 2023-09-30T02:06:22.148 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.137+0000 7fb98b07e700 10 client.4304.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:06:22.148 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.137+0000 7fb98b07e700 10 monclient: _renew_subs 2023-09-30T02:06:22.148 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.137+0000 7fb98b07e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:06:22.148 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.137+0000 7fb98b07e700 1 -- 172.21.15.59:0/3534270498 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9840a80c0 con 0x7fb984087810 2023-09-30T02:06:22.148 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.137+0000 7fb98b07e700 1 librados: init done 2023-09-30T02:06:22.149 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.138+0000 7fb98b07e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:22.149 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.138+0000 7fb98b07e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:22.149 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.138+0000 7fb98b07e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:06:22.149 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.138+0000 7fb98b07e700 1 -- 172.21.15.59:0/3534270498 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9840a8870 con 0x7fb984087810 2023-09-30T02:06:22.149 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.142+0000 7f1f26d9d700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1f2813ebc0 0x7f1f280a7200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:22.149 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.146+0000 7f1f26d9d700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1f2813ebc0 0x7f1f280a7200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:47704/0 (socket says 172.21.15.59:47704) 2023-09-30T02:06:22.149 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.146+0000 7f1f26d9d700 1 -- 172.21.15.59:0/4148507962 learned_addr learned my addr 172.21.15.59:0/4148507962 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:06:22.150 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.146+0000 7f1f26d9d700 10 monclient(hunting): get_auth_request con 0x7f1f2813ebc0 auth_method 0 2023-09-30T02:06:22.150 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.146+0000 7f1f26d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:22.150 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.146+0000 7f1f26d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:22.150 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.146+0000 7f1f26d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:22.157 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.158+0000 7f1f2659c700 1 --2- 172.21.15.59:0/4148507962 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f280ab210 0x7f1f2813e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:22.187 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.188+0000 7f6116ffd700 1 -- 172.21.15.59:0/4256937319 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f6120056ba0 con 0x7f6130069b10 2023-09-30T02:06:22.187 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.188+0000 7f6116ffd700 10 client.4342.objecter ms_dispatch 0x7f6130001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:06:22.187 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.188+0000 7f6116ffd700 3 client.4342.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:06:22.187 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.188+0000 7f6116ffd700 3 client.4342.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:06:22.187 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.188+0000 7f6116ffd700 20 client.4342.objecter dump_active .. 0 homeless 2023-09-30T02:06:22.203 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.204+0000 7f1f26d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:22.203 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.204+0000 7f1f26d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:22.204 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.204+0000 7f1f26d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:22.206 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.207+0000 7fe2ebfff700 1 -- 172.21.15.59:0/2673347399 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2e400b820 con 0x7fe2f406dd20 2023-09-30T02:06:22.206 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.207+0000 7fe2ebfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:22.206 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.207+0000 7fe2ebfff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:06:22.206 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.207+0000 7fe2ebfff700 10 monclient: dump: 2023-09-30T02:06:22.206 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:22.206 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:22.207 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:22.207 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:22.207 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:22.207 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:22.207 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:22.207 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:22.207 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:22.207 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:22.207 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.207+0000 7fe2ebfff700 10 monclient: _finish_auth 0 2023-09-30T02:06:22.207 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.207+0000 7fe2ebfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:22.207 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.207+0000 7fe2ebfff700 1 -- 172.21.15.59:0/2673347399 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2e400b980 con 0x7fe2f406dd20 2023-09-30T02:06:22.207 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.207+0000 7fe2ebfff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:22.207 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.207+0000 7fe2ebfff700 1 -- 172.21.15.59:0/2673347399 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2e400bd20 con 0x7fe2f406dd20 2023-09-30T02:06:22.208 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.207+0000 7fe2fb2fc700 5 monclient: authenticate success, global_id 4307 2023-09-30T02:06:22.208 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.207+0000 7fe2fb2fc700 10 monclient: get_monmap_and_config success 2023-09-30T02:06:22.208 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.207+0000 7fe2ebfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:22.208 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.207+0000 7fe2fb2fc700 4 set_mon_vals no callback set 2023-09-30T02:06:22.208 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.207+0000 7fe2ebfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:06:22.208 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.207+0000 7fe2ebfff700 10 monclient: dump: 2023-09-30T02:06:22.208 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:22.208 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:22.208 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:22.208 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:22.208 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:22.208 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:22.208 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:22.209 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:22.209 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:22.209 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:22.210 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.207+0000 7fe2fb2fc700 10 monclient: shutdown 2023-09-30T02:06:22.211 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.207+0000 7fe2fb2fc700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:22.211 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.207+0000 7fe2fb2fc700 1 -- 172.21.15.59:0/2673347399 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe2f406dd20 msgr2=0x7fe2f413f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:22.211 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.207+0000 7fe2fb2fc700 1 --2- 172.21.15.59:0/2673347399 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe2f406dd20 0x7fe2f413f600 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe2e40099b0 tx=0x7fe2e4009250 comp rx=0 tx=0).stop 2023-09-30T02:06:22.211 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.208+0000 7fe2fb2fc700 1 -- 172.21.15.59:0/2673347399 shutdown_connections 2023-09-30T02:06:22.211 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.208+0000 7fe2fb2fc700 1 --2- 172.21.15.59:0/2673347399 >> v2:172.21.15.110:3300/0 conn(0x7fe2f40b2f70 0x7fe2f406e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:22.211 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.208+0000 7fe2fb2fc700 1 --2- 172.21.15.59:0/2673347399 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe2f406dd20 0x7fe2f413f600 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:22.211 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.208+0000 7fe2fb2fc700 1 -- 172.21.15.59:0/2673347399 >> 172.21.15.59:0/2673347399 conn(0x7fe2f40637c0 msgr2=0x7fe2f40b2860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:22.211 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.208+0000 7fe2fb2fc700 1 -- 172.21.15.59:0/2673347399 shutdown_connections 2023-09-30T02:06:22.211 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.208+0000 7fe2fb2fc700 1 -- 172.21.15.59:0/2673347399 wait complete. 2023-09-30T02:06:22.211 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.208+0000 7fe2fb2fc700 10 monclient: build_initial_monmap 2023-09-30T02:06:22.211 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.208+0000 7fe2fb2fc700 10 monclient: monmap: 2023-09-30T02:06:22.211 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:22.212 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:06:22.212 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:06:22.212 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:06:22.212 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:22.212 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:06:22.212 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:06:22.212 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:06:22.212 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:06:22.212 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:22.212 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.209+0000 7fe2fb2fc700 1 librados: starting msgr at 2023-09-30T02:06:22.212 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.209+0000 7fe2fb2fc700 1 librados: starting objecter 2023-09-30T02:06:22.212 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.209+0000 7fe2fb2fc700 1 Processor -- start 2023-09-30T02:06:22.213 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.209+0000 7fe2fb2fc700 1 -- start start 2023-09-30T02:06:22.213 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.209+0000 7fe2fb2fc700 1 librados: setting wanted keys 2023-09-30T02:06:22.213 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.209+0000 7fe2fb2fc700 1 librados: calling monclient init 2023-09-30T02:06:22.213 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.209+0000 7fe2fb2fc700 10 monclient: init 2023-09-30T02:06:22.213 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.209+0000 7fe2fb2fc700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:22.213 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.209+0000 7fe2fb2fc700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T02:06:22.213 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.209+0000 7fe2fb2fc700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2f406dd20 0x7fe2f40a6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:22.213 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.209+0000 7fe2fb2fc700 10 monclient(hunting): picked mon.noname-a con 0x7fe2f406dd20 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:06:22.213 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.209+0000 7fe2fb2fc700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe2f40b2f70 0x7fe2f40a64d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:22.213 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.209+0000 7fe2fb2fc700 10 monclient(hunting): picked mon.noname-c con 0x7fe2f40b2f70 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:22.213 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.210+0000 7fe2fb2fc700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe2f40b3340 0x7fe2f414bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:22.213 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.210+0000 7fe2fb2fc700 10 monclient(hunting): picked mon.noname-b con 0x7fe2f40b3340 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:06:22.213 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.210+0000 7fe2fb2fc700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:22.214 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.210+0000 7fe2fb2fc700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f40a7810 con 0x7fe2f406dd20 2023-09-30T02:06:22.214 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.210+0000 7fe2fb2fc700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:22.214 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.210+0000 7fe2fb2fc700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f40a8700 con 0x7fe2f40b3340 2023-09-30T02:06:22.214 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.210+0000 7fe2fb2fc700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:22.214 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.210+0000 7fe2fb2fc700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2f40a8970 con 0x7fe2f40b2f70 2023-09-30T02:06:22.214 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.210+0000 7fe2fb2fc700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:22.214 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.210+0000 7fe2fb2fc700 10 monclient(hunting): authenticate will time out at 1827.512044s 2023-09-30T02:06:22.225 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.224+0000 7f228159a700 1 -- 172.21.15.59:0/1253880262 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f227c00e070 con 0x7f2284142290 2023-09-30T02:06:22.225 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.224+0000 7f228159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:22.225 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.224+0000 7f228159a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:06:22.225 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.224+0000 7f228159a700 10 monclient: dump: 2023-09-30T02:06:22.225 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:22.225 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:22.225 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:22.225 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:22.225 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:22.225 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:22.226 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:22.226 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:22.226 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:22.226 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:22.226 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.224+0000 7f228159a700 10 monclient: _finish_auth 0 2023-09-30T02:06:22.226 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.224+0000 7f228159a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:22.226 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.224+0000 7f228890e700 5 monclient: authenticate success, global_id 4310 2023-09-30T02:06:22.226 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.224+0000 7f228890e700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:06:22.226 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.224+0000 7f228159a700 1 -- 172.21.15.59:0/1253880262 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f227c00ba40 con 0x7f2284142290 2023-09-30T02:06:22.226 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.224+0000 7f228159a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:22.226 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.224+0000 7f228890e700 10 monclient: get_monmap_and_config success 2023-09-30T02:06:22.226 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.224+0000 7f228159a700 1 -- 172.21.15.59:0/1253880262 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f227c0092e0 con 0x7f2284142290 2023-09-30T02:06:22.227 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.224+0000 7f228890e700 4 set_mon_vals no callback set 2023-09-30T02:06:22.227 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.224+0000 7f228159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:22.227 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.224+0000 7f228159a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:06:22.227 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.224+0000 7f228159a700 10 monclient: dump: 2023-09-30T02:06:22.227 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:22.227 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:22.227 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:22.227 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:22.227 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:22.227 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:22.227 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:22.227 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:22.227 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:22.228 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:22.228 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.225+0000 7f228890e700 10 monclient: shutdown 2023-09-30T02:06:22.228 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.225+0000 7f228890e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:22.228 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.225+0000 7f228890e700 1 -- 172.21.15.59:0/1253880262 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2284142290 msgr2=0x7f228413af30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:22.228 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.225+0000 7f228890e700 1 --2- 172.21.15.59:0/1253880262 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2284142290 0x7f228413af30 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f227c009f20 tx=0x7f227c009840 comp rx=0 tx=0).stop 2023-09-30T02:06:22.228 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.225+0000 7f228890e700 1 -- 172.21.15.59:0/1253880262 shutdown_connections 2023-09-30T02:06:22.228 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.225+0000 7f228890e700 1 --2- 172.21.15.59:0/1253880262 >> v2:172.21.15.110:3300/0 conn(0x7f22840a5b70 0x7f22840b1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:22.228 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.225+0000 7f228890e700 1 --2- 172.21.15.59:0/1253880262 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2284142290 0x7f228413af30 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:22.228 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.225+0000 7f228890e700 1 -- 172.21.15.59:0/1253880262 >> 172.21.15.59:0/1253880262 conn(0x7f22840631f0 msgr2=0x7f22840635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:22.228 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.225+0000 7f228890e700 1 -- 172.21.15.59:0/1253880262 shutdown_connections 2023-09-30T02:06:22.228 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.225+0000 7f228890e700 1 -- 172.21.15.59:0/1253880262 wait complete. 2023-09-30T02:06:22.230 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.226+0000 7f228890e700 10 monclient: build_initial_monmap 2023-09-30T02:06:22.230 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.226+0000 7f228890e700 10 monclient: monmap: 2023-09-30T02:06:22.230 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:22.230 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:06:22.230 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:06:22.230 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:06:22.231 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:22.231 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:06:22.231 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:06:22.231 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:06:22.231 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:06:22.231 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:22.231 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.227+0000 7f228890e700 1 librados: starting msgr at 2023-09-30T02:06:22.231 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.227+0000 7f228890e700 1 librados: starting objecter 2023-09-30T02:06:22.231 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.227+0000 7f228890e700 1 Processor -- start 2023-09-30T02:06:22.231 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.227+0000 7f228890e700 1 -- start start 2023-09-30T02:06:22.231 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.227+0000 7f228890e700 1 librados: setting wanted keys 2023-09-30T02:06:22.231 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.227+0000 7f228890e700 1 librados: calling monclient init 2023-09-30T02:06:22.232 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.227+0000 7f228890e700 10 monclient: init 2023-09-30T02:06:22.232 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.228+0000 7f228890e700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:22.232 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.228+0000 7f228890e700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T02:06:22.232 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.228+0000 7f228890e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f22840a5b70 0x7f228413ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:22.232 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.228+0000 7f228890e700 10 monclient(hunting): picked mon.noname-a con 0x7f22840a5b70 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:06:22.232 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.228+0000 7f228890e700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f22840a5f40 0x7f22840a7a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:22.232 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.228+0000 7f228890e700 10 monclient(hunting): picked mon.noname-c con 0x7f22840a5f40 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:22.232 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.228+0000 7f228890e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2284142290 0x7f2284137880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:22.232 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.228+0000 7f228890e700 10 monclient(hunting): picked mon.noname-b con 0x7f2284142290 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:06:22.232 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.228+0000 7f228890e700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:22.232 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.228+0000 7f228890e700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22840a9350 con 0x7f22840a5b70 2023-09-30T02:06:22.232 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.228+0000 7f228890e700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:22.232 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.228+0000 7f228890e700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f228413c160 con 0x7f2284142290 2023-09-30T02:06:22.233 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.228+0000 7f228890e700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:22.233 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.228+0000 7f228890e700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22840a9550 con 0x7f22840a5f40 2023-09-30T02:06:22.233 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.228+0000 7f228890e700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:22.233 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.228+0000 7f228890e700 10 monclient(hunting): authenticate will time out at 1827.530530s 2023-09-30T02:06:22.263 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.264+0000 7f1f26d9d700 10 monclient(hunting): handle_auth_done global_id 4313 payload 824 2023-09-30T02:06:22.263 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.264+0000 7f1f26d9d700 1 -- 172.21.15.59:0/4148507962 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1f280ae430 msgr2=0x7f1f28155a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:22.263 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.264+0000 7f1f26d9d700 1 --2- 172.21.15.59:0/4148507962 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1f280ae430 0x7f1f28155a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:22.264 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.264+0000 7f1f26d9d700 1 -- 172.21.15.59:0/4148507962 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f280ab210 msgr2=0x7f1f2813e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:22.264 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.264+0000 7f1f26d9d700 1 --2- 172.21.15.59:0/4148507962 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f280ab210 0x7f1f2813e1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:22.264 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.264+0000 7f1f26d9d700 10 monclient: _finish_hunting 0 2023-09-30T02:06:22.264 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.264+0000 7f1f26d9d700 1 monclient: found mon.noname-c 2023-09-30T02:06:22.264 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.264+0000 7f1f26d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:22.264 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.264+0000 7f1f26d9d700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.59:3301/0 2023-09-30T02:06:22.264 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.264+0000 7f1f26d9d700 1 -- 172.21.15.59:0/4148507962 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f1c0094f0 con 0x7f1f2813ebc0 2023-09-30T02:06:22.287 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.288+0000 7fe2f8897700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe2f40b2f70 0x7fe2f40a64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:22.288 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.289+0000 7fbcc6d9d700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fbcc80b3340 0x7fbcc806e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:22.292 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.292+0000 7f6116ffd700 1 -- 172.21.15.59:0/4256937319 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f61200527e0 con 0x7f6130069b10 2023-09-30T02:06:22.292 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.292+0000 7f6116ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:22.292 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.292+0000 7f6116ffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:06:22.294 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.294+0000 7fbcc6d9d700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fbcc80b3340 0x7fbcc806e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51980/0 (socket says 172.21.15.59:51980) 2023-09-30T02:06:22.294 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.294+0000 7fbcc6d9d700 1 -- 172.21.15.59:0/2239380756 learned_addr learned my addr 172.21.15.59:0/2239380756 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:06:22.294 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.294+0000 7fbcc6d9d700 10 monclient(hunting): get_auth_request con 0x7fbcc80b3340 auth_method 0 2023-09-30T02:06:22.294 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.294+0000 7fbcc6d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:22.294 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.294+0000 7fbcc6d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:22.294 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.294+0000 7fbcc6d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:22.296 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.297+0000 7fe2f8897700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe2f40b2f70 0x7fe2f40a64d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:47720/0 (socket says 172.21.15.59:47720) 2023-09-30T02:06:22.296 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.297+0000 7fe2f8897700 1 -- 172.21.15.59:0/1585332057 learned_addr learned my addr 172.21.15.59:0/1585332057 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:06:22.296 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.297+0000 7fe2f8897700 10 monclient(hunting): get_auth_request con 0x7fe2f40b2f70 auth_method 0 2023-09-30T02:06:22.297 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.297+0000 7fe2f8897700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:22.297 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.297+0000 7fe2f8897700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:22.297 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.297+0000 7fe2f8897700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:22.297 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.298+0000 7fe2f9899700 1 --2- 172.21.15.59:0/1585332057 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe2f40b3340 0x7fe2f414bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:22.298 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.299+0000 7fe2f9899700 10 monclient(hunting): get_auth_request con 0x7fe2f40b3340 auth_method 0 2023-09-30T02:06:22.298 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.299+0000 7fe2f9899700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:22.298 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.299+0000 7fe2f9899700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:22.298 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.299+0000 7fe2f9899700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:22.299 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.299+0000 7f2282d9d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2284142290 0x7f2284137880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:22.299 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.299+0000 7f2282d9d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2284142290 0x7f2284137880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:39074/0 (socket says 172.21.15.59:39074) 2023-09-30T02:06:22.299 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.299+0000 7f2282d9d700 1 -- 172.21.15.59:0/1828135727 learned_addr learned my addr 172.21.15.59:0/1828135727 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:06:22.299 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.299+0000 7f2282d9d700 10 monclient(hunting): get_auth_request con 0x7f2284142290 auth_method 0 2023-09-30T02:06:22.299 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.299+0000 7f2282d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:22.299 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.299+0000 7f2282d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:22.299 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.299+0000 7f2282d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:22.309 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.310+0000 7fe2f9899700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:22.309 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.310+0000 7fe2f9899700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:22.309 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.310+0000 7fe2f9899700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:22.311 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.312+0000 7f139a7fc700 1 -- 172.21.15.59:0/3513182853 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f13a00113f0 con 0x7f13a40ab5e0 2023-09-30T02:06:22.312 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.312+0000 7f139a7fc700 1 --2- 172.21.15.59:0/3513182853 >> v2:172.21.15.110:6800/115870 conn(0x7f1388068890 0x7f138806ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:22.312 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.313+0000 7f139a7fc700 1 -- 172.21.15.59:0/3513182853 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f13a0015030 con 0x7f13a40ab5e0 2023-09-30T02:06:22.312 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.313+0000 7f139a7fc700 10 client.4302.objecter ms_dispatch 0x7f13a4001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:06:22.312 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.313+0000 7f139a7fc700 3 client.4302.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:06:22.312 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.313+0000 7f139a7fc700 3 client.4302.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:06:22.312 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.313+0000 7f139a7fc700 20 client.4302.objecter dump_active .. 0 homeless 2023-09-30T02:06:22.312 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.313+0000 7f13aa45f700 1 --2- 172.21.15.59:0/3513182853 >> v2:172.21.15.110:6800/115870 conn(0x7f1388068890 0x7f138806ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:22.312 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.313+0000 7f13aa45f700 10 monclient: get_auth_request con 0x7f1388068890 auth_method 0 2023-09-30T02:06:22.313 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.313+0000 7f139a7fc700 10 client.4302.objecter ms_handle_connect 0x7f1388068890 2023-09-30T02:06:22.313 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.313+0000 7f13aa45f700 1 --2- 172.21.15.59:0/3513182853 >> v2:172.21.15.110:6800/115870 conn(0x7f1388068890 0x7f138806ad40 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f139c0097c0 tx=0x7f139c019000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:22.355 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.355+0000 7fe2f9899700 10 monclient(hunting): handle_auth_done global_id 4351 payload 824 2023-09-30T02:06:22.355 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.355+0000 7fe2f9899700 1 -- 172.21.15.59:0/1585332057 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe2f40b2f70 msgr2=0x7fe2f40a64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:22.355 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.355+0000 7fe2f9899700 1 --2- 172.21.15.59:0/1585332057 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe2f40b2f70 0x7fe2f40a64d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:22.355 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.355+0000 7fe2f9899700 1 -- 172.21.15.59:0/1585332057 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2f406dd20 msgr2=0x7fe2f40a6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:22.355 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.355+0000 7fe2f9899700 1 --2- 172.21.15.59:0/1585332057 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2f406dd20 0x7fe2f40a6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:22.355 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.355+0000 7fe2f9899700 10 monclient: _finish_hunting 0 2023-09-30T02:06:22.356 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.355+0000 7fe2f9899700 1 monclient: found mon.noname-b 2023-09-30T02:06:22.356 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.355+0000 7fe2f9899700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:22.356 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.355+0000 7fe2f9899700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:06:22.356 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.355+0000 7fe2f9899700 1 -- 172.21.15.59:0/1585332057 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2e40094f0 con 0x7fe2f40b3340 2023-09-30T02:06:22.358 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.358+0000 7f1f26d9d700 1 --2- 172.21.15.59:0/4148507962 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1f2813ebc0 0x7f1f280a7200 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f1f2000cdb0 tx=0x7f1f200077d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:22.358 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.358+0000 7f1f167fc700 10 client.?.objecter ms_handle_connect 0x7f1f2813ebc0 2023-09-30T02:06:22.358 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.359+0000 7f1f167fc700 10 client.?.objecter resend_mon_ops 2023-09-30T02:06:22.362 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.363+0000 7f2282d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:22.362 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.363+0000 7f2282d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:22.362 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.363+0000 7f2282d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:22.368 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.369+0000 7f2282d9d700 10 monclient(hunting): handle_auth_done global_id 4354 payload 824 2023-09-30T02:06:22.369 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.369+0000 7f2282d9d700 1 -- 172.21.15.59:0/1828135727 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f22840a5f40 msgr2=0x7f22840a7a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:22.369 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.369+0000 7f2282d9d700 1 --2- 172.21.15.59:0/1828135727 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f22840a5f40 0x7f22840a7a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:22.369 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.369+0000 7f2282d9d700 1 -- 172.21.15.59:0/1828135727 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f22840a5b70 msgr2=0x7f228413ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:22.369 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.369+0000 7f2282d9d700 1 --2- 172.21.15.59:0/1828135727 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f22840a5b70 0x7f228413ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:22.369 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.369+0000 7f2282d9d700 10 monclient: _finish_hunting 0 2023-09-30T02:06:22.369 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.369+0000 7f2282d9d700 1 monclient: found mon.noname-b 2023-09-30T02:06:22.369 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.369+0000 7f2282d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:22.369 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.369+0000 7f2282d9d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:06:22.369 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.369+0000 7f2282d9d700 1 -- 172.21.15.59:0/1828135727 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f227c009ae0 con 0x7f2284142290 2023-09-30T02:06:22.370 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.370+0000 7fe2f9899700 1 --2- 172.21.15.59:0/1585332057 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe2f40b3340 0x7fe2f414bbc0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec00bb00 tx=0x7fe2ec009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:22.370 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.370+0000 7fe2e97fa700 10 client.?.objecter ms_handle_connect 0x7fe2f40b3340 2023-09-30T02:06:22.370 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.370+0000 7fe2e97fa700 10 client.?.objecter resend_mon_ops 2023-09-30T02:06:22.376 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.377+0000 7f2282d9d700 1 --2- 172.21.15.59:0/1828135727 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2284142290 0x7f2284137880 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f227c00b6c0 tx=0x7f227c017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:22.376 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.377+0000 7f22727fc700 10 client.?.objecter ms_handle_connect 0x7f2284142290 2023-09-30T02:06:22.376 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.377+0000 7f22727fc700 10 client.?.objecter resend_mon_ops 2023-09-30T02:06:22.386 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.386+0000 7f41a92b0700 1 -- 172.21.15.59:0/20148303 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f419c00dc20 con 0x7f41a413a750 2023-09-30T02:06:22.386 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.386+0000 7f41a92b0700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:22.386 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.386+0000 7f41a92b0700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:06:22.386 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.386+0000 7f41a92b0700 10 monclient: dump: 2023-09-30T02:06:22.386 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:22.386 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:22.387 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:22.387 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:22.387 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:22.387 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:22.387 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:22.387 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:22.387 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:22.387 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:22.387 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.386+0000 7f41a92b0700 10 monclient: _finish_auth 0 2023-09-30T02:06:22.387 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.386+0000 7f41a92b0700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:22.387 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.386+0000 7f41a92b0700 1 -- 172.21.15.59:0/20148303 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f419c00dd80 con 0x7f41a413a750 2023-09-30T02:06:22.387 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.386+0000 7f41ac516700 5 monclient: authenticate success, global_id 4348 2023-09-30T02:06:22.388 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.386+0000 7f41a92b0700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:22.388 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.386+0000 7f41a92b0700 1 -- 172.21.15.59:0/20148303 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f419c00f670 con 0x7f41a413a750 2023-09-30T02:06:22.388 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.386+0000 7f41a92b0700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:22.388 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.386+0000 7f41a92b0700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:06:22.388 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.386+0000 7f41a92b0700 10 monclient: dump: 2023-09-30T02:06:22.388 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:22.388 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:22.388 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:22.388 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:22.388 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:22.388 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:22.388 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:22.389 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:22.389 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:22.389 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:22.389 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.387+0000 7f41ac516700 10 monclient: get_monmap_and_config success 2023-09-30T02:06:22.389 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.387+0000 7f41ac516700 4 set_mon_vals no callback set 2023-09-30T02:06:22.391 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.388+0000 7f41ac516700 10 monclient: shutdown 2023-09-30T02:06:22.391 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.388+0000 7f41ac516700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:22.391 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.388+0000 7f41ac516700 1 -- 172.21.15.59:0/20148303 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f41a413a750 msgr2=0x7f41a413abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:22.391 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.388+0000 7f41ac516700 1 --2- 172.21.15.59:0/20148303 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f41a413a750 0x7f41a413abf0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f419c009d10 tx=0x7f419c009f90 comp rx=0 tx=0).stop 2023-09-30T02:06:22.391 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.388+0000 7f41ac516700 1 -- 172.21.15.59:0/20148303 shutdown_connections 2023-09-30T02:06:22.391 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.388+0000 7f41ac516700 1 --2- 172.21.15.59:0/20148303 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f41a413a750 0x7f41a413abf0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:22.391 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.388+0000 7f41ac516700 1 --2- 172.21.15.59:0/20148303 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f41a406a060 0x7f41a406bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:22.391 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.388+0000 7f41ac516700 1 -- 172.21.15.59:0/20148303 >> 172.21.15.59:0/20148303 conn(0x7f41a40b08e0 msgr2=0x7f41a40b0ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:22.392 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.388+0000 7f41ac516700 1 -- 172.21.15.59:0/20148303 shutdown_connections 2023-09-30T02:06:22.392 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.388+0000 7f41ac516700 1 -- 172.21.15.59:0/20148303 wait complete. 2023-09-30T02:06:22.392 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.388+0000 7f41ac516700 10 monclient: build_initial_monmap 2023-09-30T02:06:22.392 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.388+0000 7f41ac516700 10 monclient: monmap: 2023-09-30T02:06:22.392 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:22.392 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:06:22.392 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:06:22.392 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:06:22.392 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:22.392 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:06:22.392 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:06:22.392 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:06:22.393 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:06:22.393 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:22.393 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.389+0000 7f41ac516700 1 librados: starting msgr at 2023-09-30T02:06:22.393 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.389+0000 7f41ac516700 1 librados: starting objecter 2023-09-30T02:06:22.393 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.389+0000 7f41ac516700 1 Processor -- start 2023-09-30T02:06:22.393 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.389+0000 7f41ac516700 1 -- start start 2023-09-30T02:06:22.393 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.389+0000 7f41ac516700 1 librados: setting wanted keys 2023-09-30T02:06:22.393 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.389+0000 7f41ac516700 1 librados: calling monclient init 2023-09-30T02:06:22.393 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.389+0000 7f41ac516700 10 monclient: init 2023-09-30T02:06:22.393 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.389+0000 7f41ac516700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:22.393 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.389+0000 7f41ac516700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T02:06:22.393 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.390+0000 7f41ac516700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f41a406a060 0x7f41a4142170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:22.394 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.390+0000 7f41ac516700 10 monclient(hunting): picked mon.noname-b con 0x7f41a406a060 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:06:22.394 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.390+0000 7f41ac516700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f41a413a750 0x7f41a4142c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:22.394 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.390+0000 7f41ac516700 10 monclient(hunting): picked mon.noname-a con 0x7f41a413a750 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:06:22.394 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.390+0000 7f41ac516700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f41a413dc00 0x7f41a414bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:22.394 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.390+0000 7f41ac516700 10 monclient(hunting): picked mon.noname-c con 0x7f41a413dc00 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:22.394 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.390+0000 7f41ac516700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:22.394 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.390+0000 7f41ac516700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41a41431d0 con 0x7f41a413a750 2023-09-30T02:06:22.394 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.390+0000 7f41ac516700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:22.394 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.390+0000 7f41ac516700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41a41494f0 con 0x7f41a406a060 2023-09-30T02:06:22.394 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.390+0000 7f41ac516700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:22.394 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.390+0000 7f41ac516700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41a41412e0 con 0x7f41a413dc00 2023-09-30T02:06:22.394 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.390+0000 7f41ac516700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:22.395 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.390+0000 7f41ac516700 10 monclient(hunting): authenticate will time out at 1827.692098s 2023-09-30T02:06:22.395 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.390+0000 7fe2e97fa700 1 -- 172.21.15.59:0/1585332057 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2ec00c900 con 0x7fe2f40b3340 2023-09-30T02:06:22.396 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.390+0000 7fe2e97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:22.396 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.390+0000 7fe2e97fa700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:06:22.396 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.390+0000 7fe2e97fa700 10 monclient: dump: 2023-09-30T02:06:22.396 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:22.396 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:22.396 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:22.396 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:22.396 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:22.396 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:22.396 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:22.396 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:22.396 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:22.397 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:22.397 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.390+0000 7fe2e97fa700 10 monclient: _finish_auth 0 2023-09-30T02:06:22.397 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.390+0000 7fe2e97fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:22.397 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.390+0000 7fe2e97fa700 1 -- 172.21.15.59:0/1585332057 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2ec00ca60 con 0x7fe2f40b3340 2023-09-30T02:06:22.397 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.390+0000 7fe2fb2fc700 5 monclient: authenticate success, global_id 4351 2023-09-30T02:06:22.397 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.390+0000 7fe2fb2fc700 10 monclient: _renew_subs 2023-09-30T02:06:22.397 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.390+0000 7fe2fb2fc700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:22.397 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.390+0000 7fe2fb2fc700 1 -- 172.21.15.59:0/1585332057 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2f40ab6a0 con 0x7fe2f40b3340 2023-09-30T02:06:22.397 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.390+0000 7fe2e97fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:22.397 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.390+0000 7fe2e97fa700 1 -- 172.21.15.59:0/1585332057 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2ec00ce20 con 0x7fe2f40b3340 2023-09-30T02:06:22.397 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.390+0000 7fe2e97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:22.397 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.390+0000 7fe2e97fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:06:22.398 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.390+0000 7fe2e97fa700 10 monclient: dump: 2023-09-30T02:06:22.398 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:22.398 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:22.398 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:22.398 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:22.398 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:22.398 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:22.398 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:22.398 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:22.398 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:22.398 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:22.398 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.390+0000 7fe2fb2fc700 10 client.4351.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:06:22.399 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.390+0000 7fe2fb2fc700 10 monclient: _renew_subs 2023-09-30T02:06:22.399 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.390+0000 7fe2fb2fc700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:22.399 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.390+0000 7fe2eb7fe700 4 set_mon_vals no callback set 2023-09-30T02:06:22.399 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.390+0000 7fe2fb2fc700 1 -- 172.21.15.59:0/1585332057 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2f414c180 con 0x7fe2f40b3340 2023-09-30T02:06:22.399 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.390+0000 7fe2fb2fc700 1 librados: init done 2023-09-30T02:06:22.400 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.391+0000 7f22727fc700 1 -- 172.21.15.59:0/1828135727 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f227c012e90 con 0x7f2284142290 2023-09-30T02:06:22.400 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.391+0000 7f22727fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:22.400 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.391+0000 7f22727fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:06:22.400 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.391+0000 7f22727fc700 10 monclient: dump: 2023-09-30T02:06:22.400 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:22.400 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:22.400 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:22.400 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:22.400 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:22.401 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:22.401 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:22.401 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:22.401 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:22.401 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:22.401 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.391+0000 7f22727fc700 10 monclient: _finish_auth 0 2023-09-30T02:06:22.401 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.391+0000 7f22727fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:22.401 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.391+0000 7f22727fc700 1 -- 172.21.15.59:0/1828135727 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f227c0092e0 con 0x7f2284142290 2023-09-30T02:06:22.401 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.391+0000 7f22727fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:22.401 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.391+0000 7f22727fc700 1 -- 172.21.15.59:0/1828135727 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f227c01d670 con 0x7f2284142290 2023-09-30T02:06:22.401 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.391+0000 7f22727fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:22.401 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.391+0000 7f22727fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:06:22.402 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.391+0000 7f22727fc700 10 monclient: dump: 2023-09-30T02:06:22.402 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:22.402 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:22.402 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:22.402 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:22.402 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:22.402 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:22.402 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:22.402 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:22.402 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:22.402 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:22.402 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.391+0000 7f228890e700 5 monclient: authenticate success, global_id 4354 2023-09-30T02:06:22.403 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.391+0000 7f228890e700 10 monclient: _renew_subs 2023-09-30T02:06:22.403 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.391+0000 7f228890e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:22.403 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.391+0000 7f228890e700 1 -- 172.21.15.59:0/1828135727 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22840a9830 con 0x7f2284142290 2023-09-30T02:06:22.403 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.391+0000 7f228890e700 10 client.4354.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:06:22.403 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.391+0000 7f228890e700 10 monclient: _renew_subs 2023-09-30T02:06:22.403 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.391+0000 7f228890e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:22.403 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.391+0000 7f228890e700 1 -- 172.21.15.59:0/1828135727 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2284137dc0 con 0x7f2284142290 2023-09-30T02:06:22.403 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.391+0000 7f228890e700 1 librados: init done 2023-09-30T02:06:22.403 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.391+0000 7fe2fb2fc700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:22.403 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.391+0000 7fe2fb2fc700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:22.404 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.391+0000 7fe2fb2fc700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:22.404 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.391+0000 7fe2fb2fc700 1 -- 172.21.15.59:0/1585332057 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2f40a7510 con 0x7fe2f40b3340 2023-09-30T02:06:22.404 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.392+0000 7f41aa2b2700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f41a406a060 0x7f41a4142170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:22.404 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.391+0000 7f2280d99700 4 set_mon_vals no callback set 2023-09-30T02:06:22.405 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.392+0000 7f228890e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:22.405 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.392+0000 7f228890e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:22.405 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.392+0000 7f228890e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:22.405 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.392+0000 7f228890e700 1 -- 172.21.15.59:0/1828135727 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22680050b0 con 0x7f2284142290 2023-09-30T02:06:22.416 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.416+0000 7fbcc6d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:22.416 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.416+0000 7fbcc6d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:22.416 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.416+0000 7fbcc6d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:22.426 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.427+0000 7fe2e97fa700 1 -- 172.21.15.59:0/1585332057 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fe2ec014620 con 0x7fe2f40b3340 2023-09-30T02:06:22.427 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.428+0000 7fe2e97fa700 1 --2- 172.21.15.59:0/1585332057 >> v2:172.21.15.110:6800/115870 conn(0x7fe2d8068960 0x7fe2d806ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:22.427 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.428+0000 7fe2f9098700 1 --2- 172.21.15.59:0/1585332057 >> v2:172.21.15.110:6800/115870 conn(0x7fe2d8068960 0x7fe2d806ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:22.427 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.428+0000 7fe2f9098700 10 monclient: get_auth_request con 0x7fe2d8068960 auth_method 0 2023-09-30T02:06:22.428 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.429+0000 7fe2f9098700 1 --2- 172.21.15.59:0/1585332057 >> v2:172.21.15.110:6800/115870 conn(0x7fe2d8068960 0x7fe2d806ae10 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe2f0006fd0 tx=0x7fe2f0006db0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:22.428 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.429+0000 7fe2e97fa700 10 client.4351.objecter ms_handle_connect 0x7fe2d8068960 2023-09-30T02:06:22.430 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.431+0000 7fe2e97fa700 1 -- 172.21.15.59:0/1585332057 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fe2ec056360 con 0x7fe2f40b3340 2023-09-30T02:06:22.430 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.431+0000 7fe2e97fa700 10 client.4351.objecter ms_dispatch 0x7fe2f4001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:06:22.430 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.431+0000 7fe2e97fa700 3 client.4351.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:06:22.430 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.431+0000 7fe2e97fa700 3 client.4351.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:06:22.431 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.431+0000 7f41aa2b2700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f41a406a060 0x7f41a4142170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:39090/0 (socket says 172.21.15.59:39090) 2023-09-30T02:06:22.431 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.431+0000 7f41aa2b2700 1 -- 172.21.15.59:0/2484915535 learned_addr learned my addr 172.21.15.59:0/2484915535 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:06:22.431 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.431+0000 7f41aa2b2700 10 monclient(hunting): get_auth_request con 0x7f41a406a060 auth_method 0 2023-09-30T02:06:22.431 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.431+0000 7f41aa2b2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:22.431 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.431+0000 7f41aa2b2700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:22.431 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.431+0000 7f41aa2b2700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:22.431 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.431+0000 7fe2e97fa700 20 client.4351.objecter dump_active .. 0 homeless 2023-09-30T02:06:22.443 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.444+0000 7f22727fc700 1 -- 172.21.15.59:0/1828135727 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f227c01da20 con 0x7f2284142290 2023-09-30T02:06:22.443 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.444+0000 7f22727fc700 1 --2- 172.21.15.59:0/1828135727 >> v2:172.21.15.110:6800/115870 conn(0x7f2264068890 0x7f226406ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:22.444 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.444+0000 7f228259c700 1 --2- 172.21.15.59:0/1828135727 >> v2:172.21.15.110:6800/115870 conn(0x7f2264068890 0x7f226406ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:22.444 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.445+0000 7f228259c700 10 monclient: get_auth_request con 0x7f2264068890 auth_method 0 2023-09-30T02:06:22.444 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.445+0000 7f22727fc700 10 client.4354.objecter ms_handle_connect 0x7f2264068890 2023-09-30T02:06:22.444 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.445+0000 7f228259c700 1 --2- 172.21.15.59:0/1828135727 >> v2:172.21.15.110:6800/115870 conn(0x7f2264068890 0x7f226406ad40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2274006fd0 tx=0x7f2274006db0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:22.445 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.445+0000 7fbcc6d9d700 10 monclient(hunting): handle_auth_done global_id 4308 payload 274 2023-09-30T02:06:22.445 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.445+0000 7fbcc6d9d700 1 -- 172.21.15.59:0/2239380756 >> v1:172.21.15.110:6789/0 conn(0x7fbcc806dd20 legacy=0x7fbcc80abd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:22.445 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.445+0000 7fbcc6d9d700 1 -- 172.21.15.59:0/2239380756 >> v1:172.21.15.59:6789/0 conn(0x7fbcc80b2f70 legacy=0x7fbcc813f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:06:22.445 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.445+0000 7fbcc6d9d700 10 monclient: _finish_hunting 0 2023-09-30T02:06:22.445 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.445+0000 7fbcc6d9d700 1 monclient: found mon.noname-a 2023-09-30T02:06:22.445 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.445+0000 7fbcc6d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:22.446 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.445+0000 7fbcc6d9d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.59:3300/0 2023-09-30T02:06:22.446 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.445+0000 7fbcc6d9d700 1 -- 172.21.15.59:0/2239380756 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcc806f980 con 0x7fbcc80b3340 2023-09-30T02:06:22.491 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.492+0000 7f22727fc700 1 -- 172.21.15.59:0/1828135727 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f227c057810 con 0x7f2284142290 2023-09-30T02:06:22.491 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.492+0000 7f22727fc700 10 client.4354.objecter ms_dispatch 0x7f2284001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:06:22.491 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.492+0000 7f22727fc700 3 client.4354.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:06:22.492 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.492+0000 7f22727fc700 3 client.4354.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:06:22.492 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.492+0000 7f22727fc700 20 client.4354.objecter dump_active .. 0 homeless 2023-09-30T02:06:22.492 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.492+0000 7f139a7fc700 1 -- 172.21.15.59:0/3513182853 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f13a0051d80 con 0x7f13a40ab5e0 2023-09-30T02:06:22.492 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.492+0000 7f139a7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:22.492 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.492+0000 7f139a7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:06:22.493 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.492+0000 7fb980ff9700 1 -- 172.21.15.59:0/3534270498 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb97c014a60 con 0x7fb984087810 2023-09-30T02:06:22.493 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.492+0000 7fb980ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:22.493 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.492+0000 7fb982ffd700 4 set_mon_vals no callback set 2023-09-30T02:06:22.493 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.492+0000 7fb980ff9700 1 -- 172.21.15.59:0/3534270498 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb97c014e10 con 0x7fb984087810 2023-09-30T02:06:22.493 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.492+0000 7fb980ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:22.493 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.492+0000 7fb980ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:06:22.493 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.492+0000 7fb980ff9700 10 monclient: dump: 2023-09-30T02:06:22.493 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:22.493 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:22.493 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:22.494 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:22.494 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:22.494 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:22.494 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:22.494 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:22.494 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:22.494 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:22.498 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.499+0000 7fbcc6d9d700 1 --2- 172.21.15.59:0/2239380756 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbcc80b3340 0x7fbcc806e6d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fbcc0009f20 tx=0x7fbcc0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6d63f38fc4b3a36 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:22.512 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.512+0000 7f41a9ab1700 1 --2- 172.21.15.59:0/2484915535 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f41a413a750 0x7f41a4142c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:22.514 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.515+0000 7fb980ff9700 1 -- 172.21.15.59:0/3534270498 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fb97c013420 con 0x7fb984087810 2023-09-30T02:06:22.515 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.515+0000 7f41a9ab1700 10 monclient(hunting): get_auth_request con 0x7f41a413a750 auth_method 0 2023-09-30T02:06:22.515 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.515+0000 7f41a9ab1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:22.515 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.515+0000 7f41a9ab1700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:22.515 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.515+0000 7f41a9ab1700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:22.516 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.516+0000 7fb980ff9700 1 --2- 172.21.15.59:0/3534270498 >> v2:172.21.15.110:6800/115870 conn(0x7fb968068890 0x7fb96806ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:22.516 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.516+0000 7fb980ff9700 1 -- 172.21.15.59:0/3534270498 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fb97c0869d0 con 0x7fb984087810 2023-09-30T02:06:22.516 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.516+0000 7fb980ff9700 10 client.4304.objecter ms_dispatch 0x7fb984001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:06:22.516 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.516+0000 7fb980ff9700 3 client.4304.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:06:22.516 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.516+0000 7fb980ff9700 3 client.4304.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:06:22.516 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.516+0000 7fb980ff9700 20 client.4304.objecter dump_active .. 0 homeless 2023-09-30T02:06:22.516 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.516+0000 7fb988e1a700 1 --2- 172.21.15.59:0/3534270498 >> v2:172.21.15.110:6800/115870 conn(0x7fb968068890 0x7fb96806ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:22.516 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.516+0000 7fb988e1a700 10 monclient: get_auth_request con 0x7fb968068890 auth_method 0 2023-09-30T02:06:22.516 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.516+0000 7fb980ff9700 1 -- 172.21.15.59:0/3534270498 <== mon.2 v2:172.21.15.59:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb97c0b1640 con 0x7fb984087810 2023-09-30T02:06:22.516 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.516+0000 7fb980ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:22.517 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.516+0000 7fb980ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:06:22.517 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.516+0000 7fb980ff9700 10 client.4304.objecter ms_handle_connect 0x7fb968068890 2023-09-30T02:06:22.517 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.516+0000 7fb988e1a700 1 --2- 172.21.15.59:0/3534270498 >> v2:172.21.15.110:6800/115870 conn(0x7fb968068890 0x7fb96806ad40 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb970006fd0 tx=0x7fb970008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:22.520 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.520+0000 7f1f167fc700 1 -- 172.21.15.59:0/4148507962 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f2000aa80 con 0x7f1f2813ebc0 2023-09-30T02:06:22.520 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.520+0000 7f1f167fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:22.520 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.520+0000 7f1f167fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:06:22.520 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.520+0000 7f1f167fc700 10 monclient: dump: 2023-09-30T02:06:22.520 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:22.520 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:22.520 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:22.520 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:22.520 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:22.520 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:22.520 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:22.521 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:22.521 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:22.521 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:22.521 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.520+0000 7f1f167fc700 10 monclient: _finish_auth 0 2023-09-30T02:06:22.521 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.520+0000 7f1f167fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:22.521 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.520+0000 7f1f167fc700 1 -- 172.21.15.59:0/4148507962 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f2000abe0 con 0x7f1f2813ebc0 2023-09-30T02:06:22.521 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.520+0000 7f1f167fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:22.521 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.520+0000 7f1f167fc700 1 -- 172.21.15.59:0/4148507962 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f20018480 con 0x7f1f2813ebc0 2023-09-30T02:06:22.521 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.520+0000 7f1f167fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:22.521 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.520+0000 7f1f167fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:06:22.521 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.520+0000 7f1f167fc700 10 monclient: dump: 2023-09-30T02:06:22.521 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:22.521 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:22.522 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:22.522 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:22.522 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:22.522 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:22.522 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:22.522 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:22.522 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:22.522 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:22.523 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.520+0000 7f6135a5d700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2023-09-30T02:06:22.523 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.520+0000 7f6135a5d700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-09-30T02:06:22.523 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.520+0000 7f6135a5d700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:22.523 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.520+0000 7f6135a5d700 1 -- 172.21.15.59:0/4256937319 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f6130072b30 con 0x7f6130069b10 2023-09-30T02:06:22.523 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.520+0000 7f1f24d99700 4 set_mon_vals no callback set 2023-09-30T02:06:22.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.521+0000 7fbcc559a700 1 -- 172.21.15.59:0/2239380756 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc000e070 con 0x7fbcc80b3340 2023-09-30T02:06:22.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.521+0000 7fbcc559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:22.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.521+0000 7fbcc559a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T02:06:22.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.521+0000 7fbcc559a700 10 monclient: dump: 2023-09-30T02:06:22.524 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:22.524 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:22.524 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:22.524 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:22.525 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:22.525 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:22.525 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:22.525 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:22.525 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:22.525 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:22.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.521+0000 7fbcc559a700 10 monclient: _finish_auth 0 2023-09-30T02:06:22.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.521+0000 7fbcc559a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:22.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.521+0000 7fbcc559a700 1 -- 172.21.15.59:0/2239380756 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcc000ba40 con 0x7fbcc80b3340 2023-09-30T02:06:22.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.521+0000 7fbcc559a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:22.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.521+0000 7fbcc559a700 1 -- 172.21.15.59:0/2239380756 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc00092e0 con 0x7fbcc80b3340 2023-09-30T02:06:22.526 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.521+0000 7fbcc559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:22.526 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.521+0000 7fbcc559a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T02:06:22.526 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.521+0000 7fbcc559a700 10 monclient: dump: 2023-09-30T02:06:22.526 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:22.526 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:22.526 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:22.526 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:22.526 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:22.526 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:22.526 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:22.526 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:22.526 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:22.526 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:22.527 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.521+0000 7fbccc958700 5 monclient: authenticate success, global_id 4308 2023-09-30T02:06:22.527 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.521+0000 7fbccc958700 10 monclient: get_monmap_and_config success 2023-09-30T02:06:22.527 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.521+0000 7fbccc958700 4 set_mon_vals no callback set 2023-09-30T02:06:22.527 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.521+0000 7f1f2cd82700 5 monclient: authenticate success, global_id 4313 2023-09-30T02:06:22.527 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.521+0000 7f1f2cd82700 10 monclient: _renew_subs 2023-09-30T02:06:22.527 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.521+0000 7f1f2cd82700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:06:22.528 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.521+0000 7f1f2cd82700 1 -- 172.21.15.59:0/4148507962 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f280a6620 con 0x7f1f2813ebc0 2023-09-30T02:06:22.528 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.521+0000 7f1f2cd82700 10 client.4313.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:06:22.528 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.521+0000 7f1f2cd82700 10 monclient: _renew_subs 2023-09-30T02:06:22.528 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.521+0000 7f1f2cd82700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:06:22.528 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.521+0000 7f1f2cd82700 1 -- 172.21.15.59:0/4148507962 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f280a8170 con 0x7f1f2813ebc0 2023-09-30T02:06:22.528 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.521+0000 7f1f2cd82700 1 librados: init done 2023-09-30T02:06:22.528 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.522+0000 7fbccc958700 10 monclient: shutdown 2023-09-30T02:06:22.529 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.522+0000 7fbccc958700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:22.529 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.522+0000 7fbccc958700 1 -- 172.21.15.59:0/2239380756 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbcc80b3340 msgr2=0x7fbcc806e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:22.529 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.522+0000 7fbccc958700 1 --2- 172.21.15.59:0/2239380756 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbcc80b3340 0x7fbcc806e6d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fbcc0009f20 tx=0x7fbcc0009840 comp rx=0 tx=0).stop 2023-09-30T02:06:22.529 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.522+0000 7fbccc958700 1 -- 172.21.15.59:0/2239380756 shutdown_connections 2023-09-30T02:06:22.529 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.522+0000 7fbccc958700 1 --2- 172.21.15.59:0/2239380756 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbcc80b3340 0x7fbcc806e6d0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:22.529 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.522+0000 7fbccc958700 1 -- 172.21.15.59:0/2239380756 >> 172.21.15.59:0/2239380756 conn(0x7fbcc80637c0 msgr2=0x7fbcc80b2860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:22.529 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.522+0000 7fbccc958700 1 -- 172.21.15.59:0/2239380756 shutdown_connections 2023-09-30T02:06:22.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.522+0000 7f1f2cd82700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:22.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.522+0000 7f1f2cd82700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:22.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.522+0000 7f1f2cd82700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:06:22.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.522+0000 7f1f2cd82700 1 -- 172.21.15.59:0/4148507962 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f280a7c50 con 0x7f1f2813ebc0 2023-09-30T02:06:22.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.523+0000 7fbccc958700 1 -- 172.21.15.59:0/2239380756 wait complete. 2023-09-30T02:06:22.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.524+0000 7f41a9ab1700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:22.531 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.524+0000 7f41a9ab1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:22.531 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.524+0000 7f41a9ab1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:22.531 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.524+0000 7fbccc958700 10 monclient: build_initial_monmap 2023-09-30T02:06:22.531 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.524+0000 7fbccc958700 10 monclient: monmap: 2023-09-30T02:06:22.532 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:22.532 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:06:22.532 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:06:22.532 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:06:22.532 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:22.532 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:06:22.532 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:06:22.532 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:06:22.532 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:06:22.532 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:22.532 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.524+0000 7fbccc958700 1 librados: starting msgr at 2023-09-30T02:06:22.532 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.524+0000 7fbccc958700 1 librados: starting objecter 2023-09-30T02:06:22.533 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.524+0000 7fbccc958700 1 Processor -- start 2023-09-30T02:06:22.533 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.524+0000 7fbccc958700 1 -- start start 2023-09-30T02:06:22.533 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.524+0000 7fbccc958700 1 librados: setting wanted keys 2023-09-30T02:06:22.533 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.524+0000 7fbccc958700 1 librados: calling monclient init 2023-09-30T02:06:22.533 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.524+0000 7fbccc958700 10 monclient: init 2023-09-30T02:06:22.533 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.525+0000 7f41aaab3700 1 --2- 172.21.15.59:0/2484915535 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f41a413dc00 0x7f41a414bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:22.534 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.525+0000 7fbccc958700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:22.534 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.525+0000 7fbccc958700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T02:06:22.534 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.525+0000 7fbccc958700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbcc806dd20 0x7fbcc8137020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:22.534 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.525+0000 7fbccc958700 10 monclient(hunting): picked mon.noname-b con 0x7fbcc806dd20 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:06:22.534 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.525+0000 7fbccc958700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fbcc80b2f70 0x7fbcc8151350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:22.534 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.525+0000 7fbccc958700 10 monclient(hunting): picked mon.noname-c con 0x7fbcc80b2f70 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:22.534 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.525+0000 7fbccc958700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbcc80b3340 0x7fbcc813d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:22.534 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.525+0000 7fbccc958700 10 monclient(hunting): picked mon.noname-a con 0x7fbcc80b3340 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:06:22.534 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.525+0000 7fbccc958700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:22.534 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.525+0000 7fbccc958700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcc8151c40 con 0x7fbcc80b3340 2023-09-30T02:06:22.535 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.525+0000 7fbccc958700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:22.535 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.525+0000 7fbccc958700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcc8151890 con 0x7fbcc806dd20 2023-09-30T02:06:22.535 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.525+0000 7f41aaab3700 10 monclient(hunting): get_auth_request con 0x7f41a413dc00 auth_method 0 2023-09-30T02:06:22.535 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.525+0000 7f41aaab3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:22.535 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.525+0000 7f41aaab3700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:22.535 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.525+0000 7fbccc958700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:22.535 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.525+0000 7fbccc958700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcc8139f50 con 0x7fbcc80b2f70 2023-09-30T02:06:22.536 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.525+0000 7fbccc958700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:22.536 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.525+0000 7fbccc958700 10 monclient(hunting): authenticate will time out at 1827.827364s 2023-09-30T02:06:22.537 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.525+0000 7f41aaab3700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:22.537 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.526+0000 7f41aaab3700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:22.537 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.526+0000 7f41aaab3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:22.537 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.526+0000 7f41aaab3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:22.537 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.529+0000 7f41aaab3700 10 monclient(hunting): handle_auth_done global_id 4319 payload 824 2023-09-30T02:06:22.537 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.529+0000 7f41aaab3700 1 -- 172.21.15.59:0/2484915535 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f41a406a060 msgr2=0x7f41a4142170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:22.537 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.529+0000 7f41aaab3700 1 --2- 172.21.15.59:0/2484915535 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f41a406a060 0x7f41a4142170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:22.537 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.529+0000 7f41aaab3700 1 -- 172.21.15.59:0/2484915535 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f41a413a750 msgr2=0x7f41a4142c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:22.537 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.529+0000 7f41aaab3700 1 --2- 172.21.15.59:0/2484915535 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f41a413a750 0x7f41a4142c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:22.537 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.529+0000 7f41aaab3700 10 monclient: _finish_hunting 0 2023-09-30T02:06:22.538 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.529+0000 7f41aaab3700 1 monclient: found mon.noname-c 2023-09-30T02:06:22.538 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.529+0000 7f41aaab3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:22.538 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.529+0000 7f41aaab3700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.59:3301/0 2023-09-30T02:06:22.538 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.529+0000 7f41aaab3700 1 -- 172.21.15.59:0/2484915535 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f419c009850 con 0x7f41a413dc00 2023-09-30T02:06:22.538 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.534+0000 7f41aaab3700 1 --2- 172.21.15.59:0/2484915535 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f41a413dc00 0x7f41a414bc30 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f419c015000 tx=0x7f419c018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:22.538 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.534+0000 7f419a7fc700 10 client.?.objecter ms_handle_connect 0x7f41a413dc00 2023-09-30T02:06:22.538 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.534+0000 7f419a7fc700 10 client.?.objecter resend_mon_ops 2023-09-30T02:06:22.604 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.605+0000 7fbcc6d9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbcc80b3340 0x7fbcc813d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:22.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.605+0000 7fbcc6d9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbcc80b3340 0x7fbcc813d590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52036/0 (socket says 172.21.15.59:52036) 2023-09-30T02:06:22.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.605+0000 7fbcc6d9d700 1 -- 172.21.15.59:0/784904923 learned_addr learned my addr 172.21.15.59:0/784904923 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:06:22.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.605+0000 7fbcc6d9d700 10 monclient(hunting): get_auth_request con 0x7fbcc80b3340 auth_method 0 2023-09-30T02:06:22.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.605+0000 7fbcc6d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:22.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.605+0000 7fbcc6d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:22.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.605+0000 7fbcc6d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:22.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.611+0000 7fbcc6d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:22.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.611+0000 7fbcc6d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:22.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.611+0000 7fbcc6d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:22.621 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.621+0000 7fbcc6d9d700 10 monclient(hunting): handle_auth_done global_id 4314 payload 824 2023-09-30T02:06:22.621 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.621+0000 7fbcc6d9d700 1 -- 172.21.15.59:0/784904923 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fbcc80b2f70 msgr2=0x7fbcc8151350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:22.621 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.621+0000 7fbcc6d9d700 1 --2- 172.21.15.59:0/784904923 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fbcc80b2f70 0x7fbcc8151350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:22.621 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.621+0000 7fbcc6d9d700 1 -- 172.21.15.59:0/784904923 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbcc806dd20 msgr2=0x7fbcc8137020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:22.621 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.621+0000 7fbcc6d9d700 1 --2- 172.21.15.59:0/784904923 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbcc806dd20 0x7fbcc8137020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:22.621 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.621+0000 7fbcc6d9d700 10 monclient: _finish_hunting 0 2023-09-30T02:06:22.621 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.621+0000 7fbcc6d9d700 1 monclient: found mon.noname-a 2023-09-30T02:06:22.621 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.621+0000 7fbcc6d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:22.621 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.621+0000 7fbcc6d9d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.59:3300/0 2023-09-30T02:06:22.621 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.621+0000 7fbcc6d9d700 1 -- 172.21.15.59:0/784904923 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcc0009ae0 con 0x7fbcc80b3340 2023-09-30T02:06:22.624 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.625+0000 7fbcc6d9d700 1 --2- 172.21.15.59:0/784904923 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbcc80b3340 0x7fbcc813d590 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fbcc0003000 tx=0x7fbcc0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:22.624 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.625+0000 7fbcb67fc700 10 client.?.objecter ms_handle_connect 0x7fbcc80b3340 2023-09-30T02:06:22.625 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.625+0000 7fbcb67fc700 10 client.?.objecter resend_mon_ops 2023-09-30T02:06:22.630 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.631+0000 7fbcb67fc700 1 -- 172.21.15.59:0/784904923 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc00094e0 con 0x7fbcc80b3340 2023-09-30T02:06:22.630 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.631+0000 7fbcb67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:22.630 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.631+0000 7fbcb67fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T02:06:22.630 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.631+0000 7fbcb67fc700 10 monclient: dump: 2023-09-30T02:06:22.630 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:22.631 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:22.631 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:22.631 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:22.631 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:22.631 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:22.631 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:22.631 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:22.631 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:22.631 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:22.631 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.631+0000 7fbcb67fc700 10 monclient: _finish_auth 0 2023-09-30T02:06:22.631 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.631+0000 7fbcb67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:22.631 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.631+0000 7fbcb67fc700 1 -- 172.21.15.59:0/784904923 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcc0012e60 con 0x7fbcc80b3340 2023-09-30T02:06:22.632 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.631+0000 7fbccc958700 5 monclient: authenticate success, global_id 4314 2023-09-30T02:06:22.632 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.631+0000 7fbccc958700 10 monclient: _renew_subs 2023-09-30T02:06:22.632 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.631+0000 7fbccc958700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:06:22.632 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.631+0000 7fbccc958700 1 -- 172.21.15.59:0/784904923 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcc813a110 con 0x7fbcc80b3340 2023-09-30T02:06:22.632 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.631+0000 7fbcb67fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:22.632 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.631+0000 7fbcb67fc700 1 -- 172.21.15.59:0/784904923 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc001e660 con 0x7fbcc80b3340 2023-09-30T02:06:22.632 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.631+0000 7fbcb67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:22.632 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.631+0000 7fbcb67fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T02:06:22.632 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.631+0000 7fbcb67fc700 10 monclient: dump: 2023-09-30T02:06:22.632 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:22.632 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:22.632 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:22.632 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:22.633 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:22.633 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:22.633 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:22.633 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:22.633 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:22.633 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:22.633 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.631+0000 7fbcc4d99700 4 set_mon_vals no callback set 2023-09-30T02:06:22.633 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.631+0000 7fbccc958700 10 client.4314.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:06:22.633 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.631+0000 7fbccc958700 10 monclient: _renew_subs 2023-09-30T02:06:22.633 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.631+0000 7fbccc958700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:06:22.633 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.631+0000 7fbccc958700 1 -- 172.21.15.59:0/784904923 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcc813dc90 con 0x7fbcc80b3340 2023-09-30T02:06:22.634 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.631+0000 7fbccc958700 1 librados: init done 2023-09-30T02:06:22.634 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.632+0000 7fbccc958700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:22.634 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.632+0000 7fbccc958700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:22.634 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.632+0000 7fbccc958700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:06:22.634 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.632+0000 7fbccc958700 1 -- 172.21.15.59:0/784904923 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcc8151a00 con 0x7fbcc80b3340 2023-09-30T02:06:22.688 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.689+0000 7f1f167fc700 1 -- 172.21.15.59:0/4148507962 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f1f2000d040 con 0x7f1f2813ebc0 2023-09-30T02:06:22.689 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.689+0000 7f1f167fc700 1 --2- 172.21.15.59:0/4148507962 >> v2:172.21.15.110:6800/115870 conn(0x7f1f08068960 0x7f1f0806ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:22.689 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.689+0000 7f1f167fc700 1 -- 172.21.15.59:0/4148507962 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f1f20013070 con 0x7f1f2813ebc0 2023-09-30T02:06:22.689 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.689+0000 7f1f167fc700 10 client.4313.objecter ms_dispatch 0x7f1f28001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:06:22.689 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.690+0000 7f1f167fc700 3 client.4313.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:06:22.690 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.690+0000 7f1f167fc700 3 client.4313.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:06:22.690 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.690+0000 7f1f167fc700 20 client.4313.objecter dump_active .. 0 homeless 2023-09-30T02:06:22.690 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.690+0000 7f1f2659c700 1 --2- 172.21.15.59:0/4148507962 >> v2:172.21.15.110:6800/115870 conn(0x7f1f08068960 0x7f1f0806ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:22.690 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.690+0000 7f1f2659c700 10 monclient: get_auth_request con 0x7f1f08068960 auth_method 0 2023-09-30T02:06:22.690 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.690+0000 7f1f167fc700 10 client.4313.objecter ms_handle_connect 0x7f1f08068960 2023-09-30T02:06:22.690 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.690+0000 7f1f2659c700 1 --2- 172.21.15.59:0/4148507962 >> v2:172.21.15.110:6800/115870 conn(0x7f1f08068960 0x7f1f0806ae10 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1f1800a8c0 tx=0x7f1f18008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:22.759 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.760+0000 7fbcb67fc700 1 -- 172.21.15.59:0/784904923 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fbcc001ea50 con 0x7fbcc80b3340 2023-09-30T02:06:22.760 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.760+0000 7fbcb67fc700 1 --2- 172.21.15.59:0/784904923 >> v2:172.21.15.110:6800/115870 conn(0x7fbca8068960 0x7fbca806ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:22.760 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.760+0000 7fbcb67fc700 1 -- 172.21.15.59:0/784904923 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fbcc0086560 con 0x7fbcc80b3340 2023-09-30T02:06:22.761 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.761+0000 7fbcb67fc700 10 client.4314.objecter ms_dispatch 0x7fbcc8001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:06:22.761 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.761+0000 7fbcb67fc700 3 client.4314.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:06:22.761 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.761+0000 7fbcb67fc700 3 client.4314.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:06:22.761 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.761+0000 7fbcb67fc700 20 client.4314.objecter dump_active .. 0 homeless 2023-09-30T02:06:22.761 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.761+0000 7fbcc659c700 1 --2- 172.21.15.59:0/784904923 >> v2:172.21.15.110:6800/115870 conn(0x7fbca8068960 0x7fbca806ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:22.761 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.761+0000 7fbcc659c700 10 monclient: get_auth_request con 0x7fbca8068960 auth_method 0 2023-09-30T02:06:22.761 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.761+0000 7fbcb67fc700 10 client.4314.objecter ms_handle_connect 0x7fbca8068960 2023-09-30T02:06:22.761 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.761+0000 7fbcc659c700 1 --2- 172.21.15.59:0/784904923 >> v2:172.21.15.110:6800/115870 conn(0x7fbca8068960 0x7fbca806ae10 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fbcb80097c0 tx=0x7fbcb8006ce0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:22.817 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.817+0000 7f13ac6c3700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2023-09-30T02:06:22.817 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.817+0000 7f13ac6c3700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2023-09-30T02:06:22.817 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.817+0000 7f13ac6c3700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:06:22.817 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.817+0000 7f13ac6c3700 1 -- 172.21.15.59:0/3513182853 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f138c005630 con 0x7f13a40ab5e0 2023-09-30T02:06:22.817 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.818+0000 7f22727fc700 1 -- 172.21.15.59:0/1828135727 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f227c053450 con 0x7f2284142290 2023-09-30T02:06:22.818 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.818+0000 7f22727fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:22.818 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.818+0000 7f22727fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:06:22.851 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.851+0000 7f419a7fc700 1 -- 172.21.15.59:0/2484915535 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f419c00dc20 con 0x7f41a413dc00 2023-09-30T02:06:22.851 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.851+0000 7f419a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:22.851 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.851+0000 7f419a7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:06:22.851 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.851+0000 7f419a7fc700 10 monclient: dump: 2023-09-30T02:06:22.851 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:22.851 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:22.851 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:22.851 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:22.851 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:22.852 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:22.852 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:22.852 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:22.852 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:22.852 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:22.852 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.851+0000 7f419a7fc700 10 monclient: _finish_auth 0 2023-09-30T02:06:22.852 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.851+0000 7f419a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:22.852 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.851+0000 7f419a7fc700 1 -- 172.21.15.59:0/2484915535 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f419c01b070 con 0x7f41a413dc00 2023-09-30T02:06:22.852 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.851+0000 7f41ac516700 5 monclient: authenticate success, global_id 4319 2023-09-30T02:06:22.852 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.851+0000 7f419a7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:22.852 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.851+0000 7f419a7fc700 1 -- 172.21.15.59:0/2484915535 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f419c007dc0 con 0x7f41a413dc00 2023-09-30T02:06:22.853 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.851+0000 7f419a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:22.853 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.851+0000 7f419a7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:06:22.853 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.851+0000 7f419a7fc700 10 monclient: dump: 2023-09-30T02:06:22.853 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:22.853 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:22.853 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:22.853 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:22.853 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:22.853 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:22.853 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:22.853 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:22.853 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:22.854 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:22.854 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.851+0000 7f41ac516700 10 monclient: _renew_subs 2023-09-30T02:06:22.854 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.851+0000 7f41ac516700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:06:22.854 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.851+0000 7f41ac516700 1 -- 172.21.15.59:0/2484915535 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41a41434d0 con 0x7f41a413dc00 2023-09-30T02:06:22.854 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.851+0000 7f41ac516700 10 client.4319.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:06:22.854 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.851+0000 7f41ac516700 10 monclient: _renew_subs 2023-09-30T02:06:22.854 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.851+0000 7f41ac516700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:06:22.854 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.851+0000 7f41ac516700 1 -- 172.21.15.59:0/2484915535 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41a414c3b0 con 0x7f41a413dc00 2023-09-30T02:06:22.854 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.851+0000 7f41ac516700 1 librados: init done 2023-09-30T02:06:22.854 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.851+0000 7f41a8aaf700 4 set_mon_vals no callback set 2023-09-30T02:06:22.855 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.853+0000 7f41ac516700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:22.855 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.853+0000 7f41ac516700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:22.855 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.853+0000 7f41ac516700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:06:22.855 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.853+0000 7f41ac516700 1 -- 172.21.15.59:0/2484915535 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f418c0050b0 con 0x7f41a413dc00 2023-09-30T02:06:22.861 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.862+0000 7fb98b07e700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2023-09-30T02:06:22.861 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.862+0000 7fb98b07e700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2023-09-30T02:06:22.861 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.862+0000 7fb98b07e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:06:22.861 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.862+0000 7fb98b07e700 1 -- 172.21.15.59:0/3534270498 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fb984072b30 con 0x7fb984087810 2023-09-30T02:06:22.919 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.919+0000 7fe2e97fa700 1 -- 172.21.15.59:0/1585332057 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe2ec051dd0 con 0x7fe2f40b3340 2023-09-30T02:06:22.919 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.919+0000 7fe2e97fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:22.919 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.919+0000 7fe2e97fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:06:22.935 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.936+0000 7f1f167fc700 1 -- 172.21.15.59:0/4148507962 <== mon.2 v2:172.21.15.59:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1f20052360 con 0x7f1f2813ebc0 2023-09-30T02:06:22.935 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.936+0000 7f1f167fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:22.935 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.936+0000 7f1f167fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:06:22.959 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.959+0000 7f139a7fc700 1 -- 172.21.15.59:0/3513182853 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f13a001f090 con 0x7f13a40ab5e0 2023-09-30T02:06:22.959 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.959+0000 7f139a7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2023-09-30T02:06:22.959 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.959+0000 7f139a7fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:06:22.959 INFO:teuthology.orchestra.run.smithi059.stdout:60129542150 2023-09-30T02:06:22.962 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.963+0000 7f419a7fc700 1 -- 172.21.15.59:0/2484915535 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f419c0173f0 con 0x7f41a413dc00 2023-09-30T02:06:22.963 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.963+0000 7f13ac6c3700 10 librados: watch_flush enter 2023-09-30T02:06:22.963 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.963+0000 7f13ac6c3700 10 librados: watch_flush exit 2023-09-30T02:06:22.963 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.963+0000 7f13ac6c3700 20 client.4302.objecter shutdown clearing up homeless session... 2023-09-30T02:06:22.963 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.963+0000 7f13ac6c3700 10 client.4302.objecter successfully canceled tick 2023-09-30T02:06:22.963 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.964+0000 7f13ac6c3700 1 -- 172.21.15.59:0/3513182853 >> v2:172.21.15.110:6800/115870 conn(0x7f1388068890 msgr2=0x7f138806ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:22.963 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.964+0000 7f13ac6c3700 1 --2- 172.21.15.59:0/3513182853 >> v2:172.21.15.110:6800/115870 conn(0x7f1388068890 0x7f138806ad40 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f139c0097c0 tx=0x7f139c019000 comp rx=0 tx=0).stop 2023-09-30T02:06:22.964 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.964+0000 7f13ac6c3700 10 monclient: shutdown 2023-09-30T02:06:22.964 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.964+0000 7f13ac6c3700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:22.964 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.964+0000 7f13ac6c3700 1 -- 172.21.15.59:0/3513182853 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13a40ab5e0 msgr2=0x7f13a415b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:22.964 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.964+0000 7f13ac6c3700 1 --2- 172.21.15.59:0/3513182853 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13a40ab5e0 0x7f13a415b370 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f13a000e9a0 tx=0x7f13a000a330 comp rx=0 tx=0).stop 2023-09-30T02:06:22.964 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.964+0000 7f13ac6c3700 1 -- 172.21.15.59:0/3513182853 shutdown_connections 2023-09-30T02:06:22.964 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.964+0000 7f13ac6c3700 1 --2- 172.21.15.59:0/3513182853 >> v2:172.21.15.110:6800/115870 conn(0x7f1388068890 0x7f138806ad40 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:22.964 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.964+0000 7f13ac6c3700 1 --2- 172.21.15.59:0/3513182853 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13a40ab5e0 0x7f13a415b370 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:22.964 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.964+0000 7f13ac6c3700 1 --2- 172.21.15.59:0/3513182853 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f13a40a65b0 0x7f13a4136e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:22.964 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.964+0000 7f13ac6c3700 1 --2- 172.21.15.59:0/3513182853 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f13a40a9940 0x7f13a415ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:22.964 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.964+0000 7f13ac6c3700 1 -- 172.21.15.59:0/3513182853 >> 172.21.15.59:0/3513182853 conn(0x7f13a40b0d30 msgr2=0x7f13a40ad250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:22.965 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.964+0000 7f419a7fc700 1 --2- 172.21.15.59:0/2484915535 >> v2:172.21.15.110:6800/115870 conn(0x7f4188068890 0x7f418806ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:22.965 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.964+0000 7f41aa2b2700 1 --2- 172.21.15.59:0/2484915535 >> v2:172.21.15.110:6800/115870 conn(0x7f4188068890 0x7f418806ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:22.965 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.964+0000 7f41aa2b2700 10 monclient: get_auth_request con 0x7f4188068890 auth_method 0 2023-09-30T02:06:22.965 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.965+0000 7f419a7fc700 10 client.4319.objecter ms_handle_connect 0x7f4188068890 2023-09-30T02:06:22.965 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.965+0000 7f41aa2b2700 1 --2- 172.21.15.59:0/2484915535 >> v2:172.21.15.110:6800/115870 conn(0x7f4188068890 0x7f418806ad40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f41a0009b30 tx=0x7f41a0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:22.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.970+0000 7f13ac6c3700 1 -- 172.21.15.59:0/3513182853 shutdown_connections 2023-09-30T02:06:22.976 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.977+0000 7f13ac6c3700 1 -- 172.21.15.59:0/3513182853 wait complete. 2023-09-30T02:06:22.983 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.983+0000 7fbcb67fc700 1 -- 172.21.15.59:0/784904923 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbcc0053e80 con 0x7fbcc80b3340 2023-09-30T02:06:22.983 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.983+0000 7fbcb67fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:22.983 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.983+0000 7fbcb67fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:06:22.983 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:22.983+0000 7f13ac6c3700 1 librados: shutdown 2023-09-30T02:06:23.007 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.007+0000 7f6116ffd700 1 -- 172.21.15.59:0/4256937319 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6120025020 con 0x7f6130069b10 2023-09-30T02:06:23.007 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.007+0000 7f6116ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-09-30T02:06:23.007 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.007+0000 7f6116ffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:06:23.007 INFO:teuthology.orchestra.run.smithi059.stdout:77309411332 2023-09-30T02:06:23.010 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.010+0000 7f6135a5d700 10 librados: watch_flush enter 2023-09-30T02:06:23.010 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.011+0000 7f6135a5d700 10 librados: watch_flush exit 2023-09-30T02:06:23.010 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.011+0000 7f6135a5d700 20 client.4342.objecter shutdown clearing up homeless session... 2023-09-30T02:06:23.010 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.011+0000 7f6135a5d700 10 client.4342.objecter successfully canceled tick 2023-09-30T02:06:23.010 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.011+0000 7f6135a5d700 1 -- 172.21.15.59:0/4256937319 >> v2:172.21.15.110:6800/115870 conn(0x7f611c068960 msgr2=0x7f611c06ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:23.010 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.011+0000 7f6135a5d700 1 --2- 172.21.15.59:0/4256937319 >> v2:172.21.15.110:6800/115870 conn(0x7f611c068960 0x7f611c06ae10 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f61301611a0 tx=0x7f6128009350 comp rx=0 tx=0).stop 2023-09-30T02:06:23.011 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.011+0000 7f6135a5d700 10 monclient: shutdown 2023-09-30T02:06:23.011 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.011+0000 7f6135a5d700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:23.011 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.011+0000 7f6135a5d700 1 -- 172.21.15.59:0/4256937319 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6130069b10 msgr2=0x7f6130081400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:23.011 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.011+0000 7f6135a5d700 1 --2- 172.21.15.59:0/4256937319 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6130069b10 0x7f6130081400 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6120018000 tx=0x7f6120019000 comp rx=0 tx=0).stop 2023-09-30T02:06:23.013 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542150 for osd.7 2023-09-30T02:06:23.013 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.013+0000 7f6135a5d700 1 -- 172.21.15.59:0/4256937319 shutdown_connections 2023-09-30T02:06:23.013 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.013+0000 7f6135a5d700 1 --2- 172.21.15.59:0/4256937319 >> v2:172.21.15.110:6800/115870 conn(0x7f611c068960 0x7f611c06ae10 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:23.013 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.013+0000 7f6135a5d700 1 --2- 172.21.15.59:0/4256937319 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6130069b10 0x7f6130081400 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:23.013 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.013+0000 7f6135a5d700 1 --2- 172.21.15.59:0/4256937319 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6130087810 0x7f61301638f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:23.013 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.013+0000 7f6135a5d700 1 --2- 172.21.15.59:0/4256937319 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f613006bde0 0x7f6130081940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:23.014 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.013+0000 7f6135a5d700 1 -- 172.21.15.59:0/4256937319 >> 172.21.15.59:0/4256937319 conn(0x7f61300b08e0 msgr2=0x7f613014f810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:23.014 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.013+0000 7f6135a5d700 1 -- 172.21.15.59:0/4256937319 shutdown_connections 2023-09-30T02:06:23.014 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.013+0000 7f6135a5d700 1 -- 172.21.15.59:0/4256937319 wait complete. 2023-09-30T02:06:23.014 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.013+0000 7f6135a5d700 1 librados: shutdown 2023-09-30T02:06:23.021 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.022+0000 7fb980ff9700 1 -- 172.21.15.59:0/3534270498 <== mon.2 v2:172.21.15.59:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb97c0542b0 con 0x7fb984087810 2023-09-30T02:06:23.021 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.022+0000 7fb980ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2023-09-30T02:06:23.021 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.022+0000 7fb980ff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:06:23.022 INFO:teuthology.orchestra.run.smithi059.stdout:55834574853 2023-09-30T02:06:23.027 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.026+0000 7fb98b07e700 10 librados: watch_flush enter 2023-09-30T02:06:23.027 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.026+0000 7fb98b07e700 10 librados: watch_flush exit 2023-09-30T02:06:23.027 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.026+0000 7fb98b07e700 20 client.4304.objecter shutdown clearing up homeless session... 2023-09-30T02:06:23.027 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.026+0000 7fb98b07e700 10 client.4304.objecter successfully canceled tick 2023-09-30T02:06:23.027 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.028+0000 7f419a7fc700 1 -- 172.21.15.59:0/2484915535 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f419c057ae0 con 0x7f41a413dc00 2023-09-30T02:06:23.028 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.028+0000 7f419a7fc700 10 client.4319.objecter ms_dispatch 0x7f41a4001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:06:23.028 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.028+0000 7f419a7fc700 3 client.4319.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:06:23.028 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.028+0000 7f419a7fc700 3 client.4319.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:06:23.028 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.028+0000 7f419a7fc700 20 client.4319.objecter dump_active .. 0 homeless 2023-09-30T02:06:23.029 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.030+0000 7fb98b07e700 1 -- 172.21.15.59:0/3534270498 >> v2:172.21.15.110:6800/115870 conn(0x7fb968068890 msgr2=0x7fb96806ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:23.029 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.030+0000 7fb98b07e700 1 --2- 172.21.15.59:0/3534270498 >> v2:172.21.15.110:6800/115870 conn(0x7fb968068890 0x7fb96806ad40 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb970006fd0 tx=0x7fb970008040 comp rx=0 tx=0).stop 2023-09-30T02:06:23.029 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.030+0000 7fb98b07e700 10 monclient: shutdown 2023-09-30T02:06:23.029 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.030+0000 7fb98b07e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:23.029 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.030+0000 7fb98b07e700 1 -- 172.21.15.59:0/3534270498 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb984087810 msgr2=0x7fb9840a90d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:23.029 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.030+0000 7fb98b07e700 1 --2- 172.21.15.59:0/3534270498 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb984087810 0x7fb9840a90d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fb97c00ba70 tx=0x7fb97c01b000 comp rx=0 tx=0).stop 2023-09-30T02:06:23.030 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.030+0000 7fb98b07e700 1 -- 172.21.15.59:0/3534270498 shutdown_connections 2023-09-30T02:06:23.030 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.030+0000 7fb98b07e700 1 --2- 172.21.15.59:0/3534270498 >> v2:172.21.15.110:6800/115870 conn(0x7fb968068890 0x7fb96806ad40 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:23.030 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.030+0000 7fb98b07e700 1 --2- 172.21.15.59:0/3534270498 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb984087810 0x7fb9840a90d0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:23.030 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.030+0000 7fb98b07e700 1 --2- 172.21.15.59:0/3534270498 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb984069b10 0x7fb984163900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:23.030 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.030+0000 7fb98b07e700 1 --2- 172.21.15.59:0/3534270498 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb98406bde0 0x7fb9840a8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:23.030 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.030+0000 7fb98b07e700 1 -- 172.21.15.59:0/3534270498 >> 172.21.15.59:0/3534270498 conn(0x7fb9840b08e0 msgr2=0x7fb9840ada70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:23.034 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.035+0000 7fb98b07e700 1 -- 172.21.15.59:0/3534270498 shutdown_connections 2023-09-30T02:06:23.034 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.035+0000 7fb98b07e700 1 -- 172.21.15.59:0/3534270498 wait complete. 2023-09-30T02:06:23.035 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.036+0000 7fb98b07e700 1 librados: shutdown 2023-09-30T02:06:23.037 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411333 got 77309411332 for osd.2 2023-09-30T02:06:23.058 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.6 2023-09-30T02:06:23.173 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.173+0000 7f1f2cd82700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2023-09-30T02:06:23.173 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.173+0000 7f1f2cd82700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-09-30T02:06:23.173 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.173+0000 7f1f2cd82700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:06:23.173 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.173+0000 7f1f2cd82700 1 -- 172.21.15.59:0/4148507962 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f1f280a7c50 con 0x7f1f2813ebc0 2023-09-30T02:06:23.189 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.189+0000 7f228890e700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2023-09-30T02:06:23.189 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.189+0000 7f228890e700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-09-30T02:06:23.189 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.189+0000 7f228890e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:23.189 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.189+0000 7f228890e700 1 -- 172.21.15.59:0/1828135727 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f2268005630 con 0x7f2284142290 2023-09-30T02:06:23.194 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.194+0000 7f22727fc700 1 -- 172.21.15.59:0/1828135727 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f227c02a090 con 0x7f2284142290 2023-09-30T02:06:23.194 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.194+0000 7f22727fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-09-30T02:06:23.194 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.194+0000 7f22727fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:06:23.194 INFO:teuthology.orchestra.run.smithi059.stdout:64424509445 2023-09-30T02:06:23.198 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.198+0000 7f228890e700 10 librados: watch_flush enter 2023-09-30T02:06:23.198 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.198+0000 7f228890e700 10 librados: watch_flush exit 2023-09-30T02:06:23.198 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.198+0000 7f228890e700 20 client.4354.objecter shutdown clearing up homeless session... 2023-09-30T02:06:23.198 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.198+0000 7f228890e700 10 client.4354.objecter successfully canceled tick 2023-09-30T02:06:23.198 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.198+0000 7f228890e700 1 -- 172.21.15.59:0/1828135727 >> v2:172.21.15.110:6800/115870 conn(0x7f2264068890 msgr2=0x7f226406ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:23.198 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.198+0000 7f228890e700 1 --2- 172.21.15.59:0/1828135727 >> v2:172.21.15.110:6800/115870 conn(0x7f2264068890 0x7f226406ad40 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2274006fd0 tx=0x7f2274006db0 comp rx=0 tx=0).stop 2023-09-30T02:06:23.198 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.198+0000 7f228890e700 10 monclient: shutdown 2023-09-30T02:06:23.198 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.198+0000 7f228890e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:23.199 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.198+0000 7f228890e700 1 -- 172.21.15.59:0/1828135727 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2284142290 msgr2=0x7f2284137880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:23.199 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.198+0000 7f228890e700 1 --2- 172.21.15.59:0/1828135727 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2284142290 0x7f2284137880 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f227c00b6c0 tx=0x7f227c017000 comp rx=0 tx=0).stop 2023-09-30T02:06:23.199 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.198+0000 7f228890e700 1 -- 172.21.15.59:0/1828135727 shutdown_connections 2023-09-30T02:06:23.199 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.198+0000 7f228890e700 1 --2- 172.21.15.59:0/1828135727 >> v2:172.21.15.110:6800/115870 conn(0x7f2264068890 0x7f226406ad40 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:23.199 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.198+0000 7f228890e700 1 --2- 172.21.15.59:0/1828135727 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2284142290 0x7f2284137880 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:23.199 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.198+0000 7f228890e700 1 --2- 172.21.15.59:0/1828135727 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f22840a5b70 0x7f228413ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:23.199 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.198+0000 7f228890e700 1 --2- 172.21.15.59:0/1828135727 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f22840a5f40 0x7f22840a7a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:23.199 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.198+0000 7f228890e700 1 -- 172.21.15.59:0/1828135727 >> 172.21.15.59:0/1828135727 conn(0x7f22840631f0 msgr2=0x7f22840b3220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:23.199 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.198+0000 7f228890e700 1 -- 172.21.15.59:0/1828135727 shutdown_connections 2023-09-30T02:06:23.199 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.198+0000 7f228890e700 1 -- 172.21.15.59:0/1828135727 wait complete. 2023-09-30T02:06:23.199 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.198+0000 7f228890e700 1 librados: shutdown 2023-09-30T02:06:23.216 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509445 for osd.5 2023-09-30T02:06:23.224 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.224+0000 7fbccc958700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2023-09-30T02:06:23.224 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.224+0000 7fbccc958700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-09-30T02:06:23.224 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.224+0000 7fbccc958700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:06:23.224 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.224+0000 7fbccc958700 1 -- 172.21.15.59:0/784904923 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fbcc8064a00 con 0x7fbcc80b3340 2023-09-30T02:06:23.228 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.229+0000 7fbcb67fc700 1 -- 172.21.15.59:0/784904923 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fbcc00574a0 con 0x7fbcc80b3340 2023-09-30T02:06:23.228 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.229+0000 7fbcb67fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-09-30T02:06:23.228 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.229+0000 7fbcb67fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:06:23.229 INFO:teuthology.orchestra.run.smithi059.stdout:81604378629 2023-09-30T02:06:23.232 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.232+0000 7fbccc958700 10 librados: watch_flush enter 2023-09-30T02:06:23.232 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.232+0000 7fbccc958700 10 librados: watch_flush exit 2023-09-30T02:06:23.233 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.232+0000 7fbccc958700 20 client.4314.objecter shutdown clearing up homeless session... 2023-09-30T02:06:23.233 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.232+0000 7fbccc958700 10 client.4314.objecter successfully canceled tick 2023-09-30T02:06:23.233 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.232+0000 7fbccc958700 1 -- 172.21.15.59:0/784904923 >> v2:172.21.15.110:6800/115870 conn(0x7fbca8068960 msgr2=0x7fbca806ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:23.234 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.232+0000 7fbccc958700 1 --2- 172.21.15.59:0/784904923 >> v2:172.21.15.110:6800/115870 conn(0x7fbca8068960 0x7fbca806ae10 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fbcb80097c0 tx=0x7fbcb8006ce0 comp rx=0 tx=0).stop 2023-09-30T02:06:23.234 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.232+0000 7fbccc958700 10 monclient: shutdown 2023-09-30T02:06:23.234 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.232+0000 7fbccc958700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:23.234 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.232+0000 7fbccc958700 1 -- 172.21.15.59:0/784904923 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbcc80b3340 msgr2=0x7fbcc813d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:23.235 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.232+0000 7fbccc958700 1 --2- 172.21.15.59:0/784904923 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbcc80b3340 0x7fbcc813d590 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fbcc0003000 tx=0x7fbcc0017000 comp rx=0 tx=0).stop 2023-09-30T02:06:23.235 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.232+0000 7fbccc958700 1 -- 172.21.15.59:0/784904923 shutdown_connections 2023-09-30T02:06:23.236 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.232+0000 7fbccc958700 1 --2- 172.21.15.59:0/784904923 >> v2:172.21.15.110:6800/115870 conn(0x7fbca8068960 0x7fbca806ae10 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:23.236 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.232+0000 7fbccc958700 1 --2- 172.21.15.59:0/784904923 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbcc806dd20 0x7fbcc8137020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:23.236 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.232+0000 7fbccc958700 1 --2- 172.21.15.59:0/784904923 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbcc80b3340 0x7fbcc813d590 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:23.236 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.232+0000 7fbccc958700 1 --2- 172.21.15.59:0/784904923 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fbcc80b2f70 0x7fbcc8151350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:23.236 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.232+0000 7fbccc958700 1 -- 172.21.15.59:0/784904923 >> 172.21.15.59:0/784904923 conn(0x7fbcc80637c0 msgr2=0x7fbcc80b0660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:23.237 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.233+0000 7fbccc958700 1 -- 172.21.15.59:0/784904923 shutdown_connections 2023-09-30T02:06:23.237 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.233+0000 7fbccc958700 1 -- 172.21.15.59:0/784904923 wait complete. 2023-09-30T02:06:23.237 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.233+0000 7fbccc958700 1 librados: shutdown 2023-09-30T02:06:23.246 INFO:tasks.ceph.ceph_manager.ceph:need seq 81604378630 got 81604378629 for osd.3 2023-09-30T02:06:23.270 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.271+0000 7fe2fb2fc700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2023-09-30T02:06:23.271 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.271+0000 7fe2fb2fc700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-09-30T02:06:23.271 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.271+0000 7fe2fb2fc700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:23.271 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.271+0000 7fe2fb2fc700 1 -- 172.21.15.59:0/1585332057 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fe2f4064a00 con 0x7fe2f40b3340 2023-09-30T02:06:23.275 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.275+0000 7fe2e97fa700 1 -- 172.21.15.59:0/1585332057 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe2ec022020 con 0x7fe2f40b3340 2023-09-30T02:06:23.275 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.276+0000 7fe2e97fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-09-30T02:06:23.275 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.276+0000 7fe2e97fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:06:23.275 INFO:teuthology.orchestra.run.smithi059.stdout:77309411333 2023-09-30T02:06:23.278 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.278+0000 7fe2fb2fc700 10 librados: watch_flush enter 2023-09-30T02:06:23.278 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.278+0000 7fe2fb2fc700 10 librados: watch_flush exit 2023-09-30T02:06:23.278 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.278+0000 7fe2fb2fc700 20 client.4351.objecter shutdown clearing up homeless session... 2023-09-30T02:06:23.278 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.278+0000 7fe2fb2fc700 10 client.4351.objecter successfully canceled tick 2023-09-30T02:06:23.279 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.278+0000 7fe2fb2fc700 1 -- 172.21.15.59:0/1585332057 >> v2:172.21.15.110:6800/115870 conn(0x7fe2d8068960 msgr2=0x7fe2d806ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:23.279 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.278+0000 7fe2fb2fc700 1 --2- 172.21.15.59:0/1585332057 >> v2:172.21.15.110:6800/115870 conn(0x7fe2d8068960 0x7fe2d806ae10 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe2f0006fd0 tx=0x7fe2f0006db0 comp rx=0 tx=0).stop 2023-09-30T02:06:23.279 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.278+0000 7fe2fb2fc700 10 monclient: shutdown 2023-09-30T02:06:23.279 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.278+0000 7fe2fb2fc700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:23.279 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.278+0000 7fe2fb2fc700 1 -- 172.21.15.59:0/1585332057 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe2f40b3340 msgr2=0x7fe2f414bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:23.279 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.278+0000 7fe2fb2fc700 1 --2- 172.21.15.59:0/1585332057 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe2f40b3340 0x7fe2f414bbc0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe2ec00bb00 tx=0x7fe2ec009780 comp rx=0 tx=0).stop 2023-09-30T02:06:23.280 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.278+0000 7fe2fb2fc700 1 -- 172.21.15.59:0/1585332057 shutdown_connections 2023-09-30T02:06:23.280 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.278+0000 7fe2fb2fc700 1 --2- 172.21.15.59:0/1585332057 >> v2:172.21.15.110:6800/115870 conn(0x7fe2d8068960 0x7fe2d806ae10 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:23.280 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.278+0000 7fe2fb2fc700 1 --2- 172.21.15.59:0/1585332057 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe2f40b3340 0x7fe2f414bbc0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:23.280 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.278+0000 7fe2fb2fc700 1 --2- 172.21.15.59:0/1585332057 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2f406dd20 0x7fe2f40a6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:23.280 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.278+0000 7fe2fb2fc700 1 --2- 172.21.15.59:0/1585332057 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe2f40b2f70 0x7fe2f40a64d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:23.280 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.278+0000 7fe2fb2fc700 1 -- 172.21.15.59:0/1585332057 >> 172.21.15.59:0/1585332057 conn(0x7fe2f40637c0 msgr2=0x7fe2f40afa20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:23.281 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.278+0000 7fe2fb2fc700 1 -- 172.21.15.59:0/1585332057 shutdown_connections 2023-09-30T02:06:23.281 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.278+0000 7fe2fb2fc700 1 -- 172.21.15.59:0/1585332057 wait complete. 2023-09-30T02:06:23.281 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.278+0000 7fe2fb2fc700 1 librados: shutdown 2023-09-30T02:06:23.288 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.289+0000 7f419a7fc700 1 -- 172.21.15.59:0/2484915535 <== mon.2 v2:172.21.15.59:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f419c053720 con 0x7f41a413dc00 2023-09-30T02:06:23.288 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.289+0000 7f419a7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:23.288 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.289+0000 7f419a7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:06:23.291 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411334 got 77309411333 for osd.0 2023-09-30T02:06:23.308 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.309+0000 7f1f167fc700 1 -- 172.21.15.59:0/4148507962 <== mon.2 v2:172.21.15.59:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1f280a7c50 con 0x7f1f2813ebc0 2023-09-30T02:06:23.308 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.309+0000 7f1f167fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-09-30T02:06:23.308 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.309+0000 7f1f167fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:06:23.308 INFO:teuthology.orchestra.run.smithi059.stdout:64424509446 2023-09-30T02:06:23.311 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.311+0000 7f1f2cd82700 10 librados: watch_flush enter 2023-09-30T02:06:23.312 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.311+0000 7f1f2cd82700 10 librados: watch_flush exit 2023-09-30T02:06:23.312 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.311+0000 7f1f2cd82700 20 client.4313.objecter shutdown clearing up homeless session... 2023-09-30T02:06:23.312 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.312+0000 7f1f2cd82700 10 client.4313.objecter successfully canceled tick 2023-09-30T02:06:23.312 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.312+0000 7f1f2cd82700 1 -- 172.21.15.59:0/4148507962 >> v2:172.21.15.110:6800/115870 conn(0x7f1f08068960 msgr2=0x7f1f0806ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:23.312 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.312+0000 7f1f2cd82700 1 --2- 172.21.15.59:0/4148507962 >> v2:172.21.15.110:6800/115870 conn(0x7f1f08068960 0x7f1f0806ae10 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1f1800a8c0 tx=0x7f1f18008040 comp rx=0 tx=0).stop 2023-09-30T02:06:23.312 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.312+0000 7f1f2cd82700 10 monclient: shutdown 2023-09-30T02:06:23.312 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.312+0000 7f1f2cd82700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:23.312 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.312+0000 7f1f2cd82700 1 -- 172.21.15.59:0/4148507962 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1f2813ebc0 msgr2=0x7f1f280a7200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:23.312 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.312+0000 7f1f2cd82700 1 --2- 172.21.15.59:0/4148507962 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1f2813ebc0 0x7f1f280a7200 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f1f2000cdb0 tx=0x7f1f200077d0 comp rx=0 tx=0).stop 2023-09-30T02:06:23.312 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.312+0000 7f1f2cd82700 1 -- 172.21.15.59:0/4148507962 shutdown_connections 2023-09-30T02:06:23.312 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.312+0000 7f1f2cd82700 1 --2- 172.21.15.59:0/4148507962 >> v2:172.21.15.110:6800/115870 conn(0x7f1f08068960 0x7f1f0806ae10 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:23.313 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.312+0000 7f1f2cd82700 1 --2- 172.21.15.59:0/4148507962 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1f2813ebc0 0x7f1f280a7200 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:23.313 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.312+0000 7f1f2cd82700 1 --2- 172.21.15.59:0/4148507962 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f280ab210 0x7f1f2813e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:23.313 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.312+0000 7f1f2cd82700 1 --2- 172.21.15.59:0/4148507962 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1f280ae430 0x7f1f28155a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:23.313 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.312+0000 7f1f2cd82700 1 -- 172.21.15.59:0/4148507962 >> 172.21.15.59:0/4148507962 conn(0x7f1f2806a030 msgr2=0x7f1f280b0800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:23.313 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.312+0000 7f1f2cd82700 1 -- 172.21.15.59:0/4148507962 shutdown_connections 2023-09-30T02:06:23.313 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.312+0000 7f1f2cd82700 1 -- 172.21.15.59:0/4148507962 wait complete. 2023-09-30T02:06:23.313 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.312+0000 7f1f2cd82700 1 librados: shutdown 2023-09-30T02:06:23.330 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509446 for osd.4 2023-09-30T02:06:23.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.525+0000 7f41ac516700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2023-09-30T02:06:23.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.525+0000 7f41ac516700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-09-30T02:06:23.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.525+0000 7f41ac516700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:06:23.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.525+0000 7f41ac516700 1 -- 172.21.15.59:0/2484915535 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f418c005630 con 0x7f41a413dc00 2023-09-30T02:06:23.533 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.534+0000 7f419a7fc700 1 -- 172.21.15.59:0/2484915535 <== mon.2 v2:172.21.15.59:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f419c056d40 con 0x7f41a413dc00 2023-09-30T02:06:23.534 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.534+0000 7f419a7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-09-30T02:06:23.534 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.534+0000 7f419a7fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:06:23.534 INFO:teuthology.orchestra.run.smithi059.stdout:77309411332 2023-09-30T02:06:23.537 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.536+0000 7f41ac516700 10 librados: watch_flush enter 2023-09-30T02:06:23.537 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.537+0000 7f41ac516700 10 librados: watch_flush exit 2023-09-30T02:06:23.537 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.537+0000 7f41ac516700 20 client.4319.objecter shutdown clearing up homeless session... 2023-09-30T02:06:23.537 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.537+0000 7f41ac516700 10 client.4319.objecter successfully canceled tick 2023-09-30T02:06:23.537 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.537+0000 7f41ac516700 1 -- 172.21.15.59:0/2484915535 >> v2:172.21.15.110:6800/115870 conn(0x7f4188068890 msgr2=0x7f418806ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:23.537 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.537+0000 7f41ac516700 1 --2- 172.21.15.59:0/2484915535 >> v2:172.21.15.110:6800/115870 conn(0x7f4188068890 0x7f418806ad40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f41a0009b30 tx=0x7f41a0008040 comp rx=0 tx=0).stop 2023-09-30T02:06:23.537 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.537+0000 7f41ac516700 10 monclient: shutdown 2023-09-30T02:06:23.537 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.537+0000 7f41ac516700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:23.537 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.537+0000 7f41ac516700 1 -- 172.21.15.59:0/2484915535 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f41a413dc00 msgr2=0x7f41a414bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:23.537 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.537+0000 7f41ac516700 1 --2- 172.21.15.59:0/2484915535 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f41a413dc00 0x7f41a414bc30 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f419c015000 tx=0x7f419c018000 comp rx=0 tx=0).stop 2023-09-30T02:06:23.537 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.537+0000 7f41ac516700 1 -- 172.21.15.59:0/2484915535 shutdown_connections 2023-09-30T02:06:23.538 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.537+0000 7f41ac516700 1 --2- 172.21.15.59:0/2484915535 >> v2:172.21.15.110:6800/115870 conn(0x7f4188068890 0x7f418806ad40 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:23.538 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.537+0000 7f41ac516700 1 --2- 172.21.15.59:0/2484915535 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f41a406a060 0x7f41a4142170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:23.538 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.537+0000 7f41ac516700 1 --2- 172.21.15.59:0/2484915535 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f41a413dc00 0x7f41a414bc30 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:23.538 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.537+0000 7f41ac516700 1 --2- 172.21.15.59:0/2484915535 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f41a413a750 0x7f41a4142c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:23.538 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.537+0000 7f41ac516700 1 -- 172.21.15.59:0/2484915535 >> 172.21.15.59:0/2484915535 conn(0x7f41a40b08e0 msgr2=0x7f41a413e030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:23.538 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.537+0000 7f41ac516700 1 -- 172.21.15.59:0/2484915535 shutdown_connections 2023-09-30T02:06:23.538 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.537+0000 7f41ac516700 1 -- 172.21.15.59:0/2484915535 wait complete. 2023-09-30T02:06:23.538 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:23.537+0000 7f41ac516700 1 librados: shutdown 2023-09-30T02:06:23.552 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411333 got 77309411332 for osd.1 2023-09-30T02:06:24.013 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-09-30T02:06:24.038 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-09-30T02:06:24.058 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-09-30T02:06:24.162 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.162+0000 7fc52084b700 10 monclient: get_monmap_and_config 2023-09-30T02:06:24.162 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.162+0000 7fc52084b700 10 monclient: build_initial_monmap 2023-09-30T02:06:24.162 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.162+0000 7fc52084b700 10 monclient: monmap: 2023-09-30T02:06:24.162 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:24.162 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:24.162 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:06:24.163042+0000 2023-09-30T02:06:24.163 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:06:24.163042+0000 2023-09-30T02:06:24.163 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:24.163 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:24.163 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:06:24.163 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:06:24.163 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:06:24.163 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:06:24.163 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:06:24.163 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:24.165 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.163+0000 7fc52084b700 1 Processor -- start 2023-09-30T02:06:24.165 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.163+0000 7fc52084b700 1 -- start start 2023-09-30T02:06:24.165 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.163+0000 7fc52084b700 10 monclient: init 2023-09-30T02:06:24.166 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.164+0000 7fc52084b700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:24.166 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.164+0000 7fc52084b700 10 monclient: _add_conns ranks=[4,0,3,2,1] 2023-09-30T02:06:24.166 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.164+0000 7fc52084b700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fc51c0a5b70 0x7fc51c0b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:24.166 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.164+0000 7fc52084b700 10 monclient(hunting): picked mon.noname-c con 0x7fc51c0a5b70 addr v2:172.21.15.110:3300/0 2023-09-30T02:06:24.166 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.164+0000 7fc52084b700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc51c0a5f40 addr v1:172.21.15.59:6789/0 2023-09-30T02:06:24.166 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.164+0000 7fc52084b700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc51c142290 0x7fc51c13af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:24.166 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.164+0000 7fc52084b700 10 monclient(hunting): picked mon.noname-a con 0x7fc51c142290 addr v2:172.21.15.59:3300/0 2023-09-30T02:06:24.166 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.164+0000 7fc52084b700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc51c0a75a0 con 0x7fc51c0a5f40 2023-09-30T02:06:24.167 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.164+0000 7fc52084b700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:24.167 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.164+0000 7fc52084b700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc51c0a63e0 con 0x7fc51c142290 2023-09-30T02:06:24.167 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.164+0000 7fc52084b700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:24.167 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.164+0000 7fc52084b700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc51c1426b0 con 0x7fc51c0a5b70 2023-09-30T02:06:24.167 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.164+0000 7fc52084b700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:24.167 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.164+0000 7fc52084b700 10 monclient(hunting): authenticate will time out at 1829.466378s 2023-09-30T02:06:24.167 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.165+0000 7fc51ad9d700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc51c142290 0x7fc51c13af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:24.167 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.165+0000 7fc51a59c700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fc51c0a5b70 0x7fc51c0b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:24.167 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.165+0000 7fc51a59c700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fc51c0a5b70 0x7fc51c0b1500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:39112/0 (socket says 172.21.15.59:39112) 2023-09-30T02:06:24.167 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.165+0000 7fc51a59c700 1 -- 172.21.15.59:0/73097208 learned_addr learned my addr 172.21.15.59:0/73097208 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:06:24.167 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.165+0000 7fc51a59c700 10 monclient(hunting): get_auth_request con 0x7fc51c0a5b70 auth_method 0 2023-09-30T02:06:24.168 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.165+0000 7fc51a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:24.168 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.165+0000 7fc51a59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:24.168 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.165+0000 7fc51a59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:24.168 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.166+0000 7fc51ad9d700 10 monclient(hunting): get_auth_request con 0x7fc51c142290 auth_method 0 2023-09-30T02:06:24.168 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.166+0000 7fc51ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:24.168 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.166+0000 7fc51ad9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:24.168 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.166+0000 7fc51ad9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:24.168 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.166+0000 7fc51a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:24.168 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.166+0000 7fc51a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:24.168 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.166+0000 7fc51a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:24.169 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.167+0000 7fc51ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:24.169 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.167+0000 7fc51ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:24.170 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.167+0000 7fc51ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:24.170 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.169+0000 7fc51a59c700 10 monclient(hunting): handle_auth_done global_id 4360 payload 274 2023-09-30T02:06:24.170 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.169+0000 7fc51a59c700 1 -- 172.21.15.59:0/73097208 >> v2:172.21.15.59:3300/0 conn(0x7fc51c142290 msgr2=0x7fc51c13af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:06:24.170 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.169+0000 7fc51a59c700 1 --2- 172.21.15.59:0/73097208 >> v2:172.21.15.59:3300/0 conn(0x7fc51c142290 0x7fc51c13af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:24.170 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.169+0000 7fc51a59c700 1 -- 172.21.15.59:0/73097208 >> v1:172.21.15.59:6789/0 conn(0x7fc51c0a5f40 legacy=0x7fc51c141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:24.170 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.169+0000 7fc51a59c700 10 monclient: _finish_hunting 0 2023-09-30T02:06:24.170 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.169+0000 7fc51a59c700 1 monclient: found mon.noname-c 2023-09-30T02:06:24.170 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.169+0000 7fc51a59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:24.170 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.169+0000 7fc51a59c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3300/0 2023-09-30T02:06:24.170 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.169+0000 7fc51a59c700 1 -- 172.21.15.59:0/73097208 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc51c0af3f0 con 0x7fc51c0a5b70 2023-09-30T02:06:24.171 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.171+0000 7fc51a59c700 1 --2- 172.21.15.59:0/73097208 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc51c0a5b70 0x7fc51c0b1500 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc51c0b49e0 tx=0x7fc50c009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=debf67335526a45c server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:24.175 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.175+0000 7fc51959a700 1 -- 172.21.15.59:0/73097208 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc50c009e70 con 0x7fc51c0a5b70 2023-09-30T02:06:24.175 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.175+0000 7fc51959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:24.176 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.175+0000 7fc51959a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:06:24.176 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.175+0000 7fc51959a700 10 monclient: dump: 2023-09-30T02:06:24.176 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:24.176 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:24.176 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:24.176 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:24.176 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:24.176 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:24.177 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:24.177 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:24.177 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:24.177 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:24.177 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.175+0000 7fc51959a700 10 monclient: _finish_auth 0 2023-09-30T02:06:24.177 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.175+0000 7fc51959a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:24.177 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.175+0000 7fc51959a700 1 -- 172.21.15.59:0/73097208 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc50c00b860 con 0x7fc51c0a5b70 2023-09-30T02:06:24.177 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.175+0000 7fc52084b700 5 monclient: authenticate success, global_id 4360 2023-09-30T02:06:24.178 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.175+0000 7fc52084b700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:06:24.178 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.175+0000 7fc51959a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:24.178 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.175+0000 7fc52084b700 10 monclient: get_monmap_and_config success 2023-09-30T02:06:24.178 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.175+0000 7fc52084b700 4 set_mon_vals no callback set 2023-09-30T02:06:24.178 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.175+0000 7fc51959a700 1 -- 172.21.15.59:0/73097208 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc50c00bc00 con 0x7fc51c0a5b70 2023-09-30T02:06:24.178 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.175+0000 7fc51959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:24.178 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.175+0000 7fc51959a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:06:24.178 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.175+0000 7fc51959a700 10 monclient: dump: 2023-09-30T02:06:24.178 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:24.178 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:24.178 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:24.179 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:24.179 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:24.179 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:24.179 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:24.179 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:24.179 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:24.179 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:24.179 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.176+0000 7fc52084b700 10 monclient: shutdown 2023-09-30T02:06:24.179 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.176+0000 7fc52084b700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:24.179 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.176+0000 7fc52084b700 1 -- 172.21.15.59:0/73097208 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc51c0a5b70 msgr2=0x7fc51c0b1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:24.179 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.176+0000 7fc52084b700 1 --2- 172.21.15.59:0/73097208 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc51c0a5b70 0x7fc51c0b1500 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fc51c0b49e0 tx=0x7fc50c009250 comp rx=0 tx=0).stop 2023-09-30T02:06:24.179 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.176+0000 7fc52084b700 1 -- 172.21.15.59:0/73097208 shutdown_connections 2023-09-30T02:06:24.180 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.176+0000 7fc52084b700 1 --2- 172.21.15.59:0/73097208 >> v2:172.21.15.59:3300/0 conn(0x7fc51c142290 0x7fc51c13af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:24.180 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.176+0000 7fc52084b700 1 --2- 172.21.15.59:0/73097208 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc51c0a5b70 0x7fc51c0b1500 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:24.180 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.176+0000 7fc52084b700 1 -- 172.21.15.59:0/73097208 >> 172.21.15.59:0/73097208 conn(0x7fc51c0631f0 msgr2=0x7fc51c0635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:24.180 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.176+0000 7fc52084b700 1 -- 172.21.15.59:0/73097208 shutdown_connections 2023-09-30T02:06:24.180 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.176+0000 7fc52084b700 1 -- 172.21.15.59:0/73097208 wait complete. 2023-09-30T02:06:24.183 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.177+0000 7fc52084b700 10 monclient: build_initial_monmap 2023-09-30T02:06:24.183 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.177+0000 7fc52084b700 10 monclient: monmap: 2023-09-30T02:06:24.183 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:24.183 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:06:24.183 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:06:24.183 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:06:24.183 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:24.183 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:06:24.183 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:06:24.183 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:06:24.183 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:06:24.184 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:24.184 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.177+0000 7fc52084b700 1 librados: starting msgr at 2023-09-30T02:06:24.184 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.177+0000 7fc52084b700 1 librados: starting objecter 2023-09-30T02:06:24.184 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.177+0000 7fc52084b700 1 Processor -- start 2023-09-30T02:06:24.184 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.177+0000 7fc52084b700 1 -- start start 2023-09-30T02:06:24.184 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.177+0000 7fc52084b700 1 librados: setting wanted keys 2023-09-30T02:06:24.184 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.177+0000 7fc52084b700 1 librados: calling monclient init 2023-09-30T02:06:24.184 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.177+0000 7fc52084b700 10 monclient: init 2023-09-30T02:06:24.184 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.178+0000 7fc52084b700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:24.184 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.178+0000 7fc52084b700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T02:06:24.184 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.178+0000 7fc52084b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc51c0a5b70 0x7fc51c13db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:24.185 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.178+0000 7fc52084b700 10 monclient(hunting): picked mon.noname-a con 0x7fc51c0a5b70 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:06:24.185 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.178+0000 7fc52084b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc51c0a5f40 0x7fc51c139bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:24.185 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.178+0000 7fc52084b700 10 monclient(hunting): picked mon.noname-b con 0x7fc51c0a5f40 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:06:24.185 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.178+0000 7fc52084b700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc51c142290 0x7fc51c13e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:24.185 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.178+0000 7fc52084b700 10 monclient(hunting): picked mon.noname-c con 0x7fc51c142290 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:24.185 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.178+0000 7fc52084b700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:24.185 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.178+0000 7fc52084b700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc51c13a310 con 0x7fc51c0a5b70 2023-09-30T02:06:24.185 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.178+0000 7fc52084b700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:24.185 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.178+0000 7fc52084b700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc51c13c030 con 0x7fc51c0a5f40 2023-09-30T02:06:24.185 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.178+0000 7fc52084b700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:24.185 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.178+0000 7fc52084b700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc51c136ff0 con 0x7fc51c142290 2023-09-30T02:06:24.185 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.178+0000 7fc52084b700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:24.185 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.178+0000 7fc52084b700 10 monclient(hunting): authenticate will time out at 1829.480549s 2023-09-30T02:06:24.186 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.179+0000 7fc519d9b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc51c0a5f40 0x7fc51c139bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:24.186 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.179+0000 7fc51ad9d700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc51c142290 0x7fc51c13e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:24.186 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.179+0000 7fc51a59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc51c0a5b70 0x7fc51c13db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:24.186 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.179+0000 7fc519d9b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc51c0a5f40 0x7fc51c139bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:39116/0 (socket says 172.21.15.59:39116) 2023-09-30T02:06:24.186 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.179+0000 7fc519d9b700 1 -- 172.21.15.59:0/3488309627 learned_addr learned my addr 172.21.15.59:0/3488309627 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:06:24.186 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.179+0000 7fc519d9b700 10 monclient(hunting): get_auth_request con 0x7fc51c0a5f40 auth_method 0 2023-09-30T02:06:24.186 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.179+0000 7fc519d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:24.186 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.179+0000 7fc519d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:24.186 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.179+0000 7fc519d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:24.186 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.179+0000 7fc51a59c700 10 monclient(hunting): get_auth_request con 0x7fc51c0a5b70 auth_method 0 2023-09-30T02:06:24.187 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.179+0000 7fc51a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:24.187 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.179+0000 7fc51a59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:24.187 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.179+0000 7fc51a59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:24.187 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.180+0000 7fc519d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:24.187 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.180+0000 7fc519d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:24.187 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.180+0000 7fc519d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:24.187 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.180+0000 7fc51ad9d700 10 monclient(hunting): get_auth_request con 0x7fc51c142290 auth_method 0 2023-09-30T02:06:24.187 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.180+0000 7fc51ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:24.187 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.180+0000 7fc51ad9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:24.187 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.180+0000 7fc51ad9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:24.187 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.180+0000 7fc51a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:24.187 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.180+0000 7fc51a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:24.188 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.180+0000 7fc51a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:24.188 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.180+0000 7fc51ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:24.188 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.180+0000 7fc51ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:24.188 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.180+0000 7fc51ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:24.188 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.183+0000 7fc519d9b700 10 monclient(hunting): handle_auth_done global_id 4363 payload 824 2023-09-30T02:06:24.188 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.183+0000 7fc519d9b700 1 -- 172.21.15.59:0/3488309627 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc51c142290 msgr2=0x7fc51c13e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:24.188 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.183+0000 7fc519d9b700 1 --2- 172.21.15.59:0/3488309627 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc51c142290 0x7fc51c13e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:24.188 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.183+0000 7fc519d9b700 1 -- 172.21.15.59:0/3488309627 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc51c0a5b70 msgr2=0x7fc51c13db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:24.188 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.183+0000 7fc519d9b700 1 --2- 172.21.15.59:0/3488309627 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc51c0a5b70 0x7fc51c13db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:24.188 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.183+0000 7fc519d9b700 10 monclient: _finish_hunting 0 2023-09-30T02:06:24.188 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.183+0000 7fc519d9b700 1 monclient: found mon.noname-b 2023-09-30T02:06:24.189 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.183+0000 7fc519d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:24.189 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.183+0000 7fc519d9b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:06:24.189 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.183+0000 7fc519d9b700 1 -- 172.21.15.59:0/3488309627 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc50c0094f0 con 0x7fc51c0a5f40 2023-09-30T02:06:24.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.185+0000 7fc519d9b700 1 --2- 172.21.15.59:0/3488309627 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc51c0a5f40 0x7fc51c139bf0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc51000c590 tx=0x7fc51000c8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:24.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.185+0000 7fc50a7fc700 10 client.?.objecter ms_handle_connect 0x7fc51c0a5f40 2023-09-30T02:06:24.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.185+0000 7fc50a7fc700 10 client.?.objecter resend_mon_ops 2023-09-30T02:06:24.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.189+0000 7fc50a7fc700 1 -- 172.21.15.59:0/3488309627 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc510011590 con 0x7fc51c0a5f40 2023-09-30T02:06:24.191 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.189+0000 7fc50a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:24.191 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.189+0000 7fc50a7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:06:24.191 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.189+0000 7fc50a7fc700 10 monclient: dump: 2023-09-30T02:06:24.191 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:24.191 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:24.191 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:24.191 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:24.191 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:24.191 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:24.191 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:24.191 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:24.191 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:24.192 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:24.192 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.189+0000 7fc50a7fc700 10 monclient: _finish_auth 0 2023-09-30T02:06:24.192 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.189+0000 7fc50a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:24.192 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.189+0000 7fc50a7fc700 1 -- 172.21.15.59:0/3488309627 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc510011710 con 0x7fc51c0a5f40 2023-09-30T02:06:24.192 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.189+0000 7fc50a7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:24.192 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.189+0000 7fc50a7fc700 1 -- 172.21.15.59:0/3488309627 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc510011ad0 con 0x7fc51c0a5f40 2023-09-30T02:06:24.192 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.189+0000 7fc52084b700 5 monclient: authenticate success, global_id 4363 2023-09-30T02:06:24.192 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.189+0000 7fc52084b700 10 monclient: _renew_subs 2023-09-30T02:06:24.192 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.189+0000 7fc52084b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:24.192 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.189+0000 7fc52084b700 1 -- 172.21.15.59:0/3488309627 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc51c13aa60 con 0x7fc51c0a5f40 2023-09-30T02:06:24.193 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.189+0000 7fc518d99700 4 set_mon_vals no callback set 2023-09-30T02:06:24.193 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.189+0000 7fc50a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:24.193 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.189+0000 7fc50a7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:06:24.193 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.189+0000 7fc50a7fc700 10 monclient: dump: 2023-09-30T02:06:24.193 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:24.193 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:24.193 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:24.193 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:24.193 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:24.193 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:24.194 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:24.194 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:24.194 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:24.194 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:24.194 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.189+0000 7fc52084b700 10 client.4363.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:06:24.194 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.189+0000 7fc52084b700 10 monclient: _renew_subs 2023-09-30T02:06:24.194 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.189+0000 7fc52084b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:24.194 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.189+0000 7fc52084b700 1 -- 172.21.15.59:0/3488309627 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc51c13eb90 con 0x7fc51c0a5f40 2023-09-30T02:06:24.194 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.189+0000 7fc52084b700 1 librados: init done 2023-09-30T02:06:24.194 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.190+0000 7fc52084b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:24.195 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.190+0000 7fc52084b700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:24.195 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.190+0000 7fc52084b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:24.195 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.190+0000 7fc52084b700 1 -- 172.21.15.59:0/3488309627 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc51c13e130 con 0x7fc51c0a5f40 2023-09-30T02:06:24.205 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.206+0000 7f33cac6c700 10 monclient: get_monmap_and_config 2023-09-30T02:06:24.205 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.206+0000 7f33cac6c700 10 monclient: build_initial_monmap 2023-09-30T02:06:24.205 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.206+0000 7f33cac6c700 10 monclient: monmap: 2023-09-30T02:06:24.205 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:24.205 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:24.205 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:06:24.206639+0000 2023-09-30T02:06:24.206 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:06:24.206639+0000 2023-09-30T02:06:24.206 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:24.206 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:24.206 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:06:24.206 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:06:24.206 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:06:24.206 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:06:24.206 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:06:24.206 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:24.207 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.207+0000 7f33cac6c700 1 Processor -- start 2023-09-30T02:06:24.208 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.207+0000 7f33cac6c700 1 -- start start 2023-09-30T02:06:24.208 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.207+0000 7f33cac6c700 10 monclient: init 2023-09-30T02:06:24.208 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.207+0000 7f33cac6c700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:24.208 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.207+0000 7f33cac6c700 10 monclient: _add_conns ranks=[1,3,4,0,2] 2023-09-30T02:06:24.208 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.207+0000 7f33cac6c700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f33c40a5b70 addr v1:172.21.15.110:6789/0 2023-09-30T02:06:24.208 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.207+0000 7f33cac6c700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f33c4142290 0x7f33c40b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:24.208 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.207+0000 7f33cac6c700 10 monclient(hunting): picked mon.noname-a con 0x7f33c4142290 addr v2:172.21.15.59:3300/0 2023-09-30T02:06:24.208 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.207+0000 7f33cac6c700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f33c40a5f40 0x7f33c413af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:24.208 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.207+0000 7f33cac6c700 10 monclient(hunting): picked mon.noname-c con 0x7f33c40a5f40 addr v2:172.21.15.110:3300/0 2023-09-30T02:06:24.208 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.207+0000 7f33cac6c700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33c40a75a0 con 0x7f33c40a5b70 2023-09-30T02:06:24.209 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.207+0000 7f33cac6c700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:24.209 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.207+0000 7f33cac6c700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33c40a6360 con 0x7f33c4142290 2023-09-30T02:06:24.209 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.207+0000 7f33cac6c700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:24.209 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.207+0000 7f33cac6c700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33c40af3f0 con 0x7f33c40a5f40 2023-09-30T02:06:24.209 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.207+0000 7f33cac6c700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:24.209 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.207+0000 7f33cac6c700 10 monclient(hunting): authenticate will time out at 1829.509909s 2023-09-30T02:06:24.209 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.208+0000 7f35c142d700 10 monclient: get_monmap_and_config 2023-09-30T02:06:24.210 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.208+0000 7f35c142d700 10 monclient: build_initial_monmap 2023-09-30T02:06:24.210 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.208+0000 7f35c142d700 10 monclient: monmap: 2023-09-30T02:06:24.210 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:24.210 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:24.210 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:06:24.208631+0000 2023-09-30T02:06:24.210 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:06:24.208631+0000 2023-09-30T02:06:24.210 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:24.210 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:24.210 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:06:24.210 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:06:24.210 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:06:24.211 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:06:24.211 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:06:24.211 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:24.211 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.208+0000 7f33c3fff700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f33c4142290 0x7f33c40b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:24.211 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.209+0000 7f35c142d700 1 Processor -- start 2023-09-30T02:06:24.212 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.209+0000 7f35c142d700 1 -- start start 2023-09-30T02:06:24.212 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.209+0000 7f35c142d700 10 monclient: init 2023-09-30T02:06:24.212 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.209+0000 7f33c3fff700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f33c4142290 0x7f33c40b1500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52052/0 (socket says 172.21.15.59:52052) 2023-09-30T02:06:24.212 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.209+0000 7f33c3fff700 1 -- 172.21.15.59:0/2418441615 learned_addr learned my addr 172.21.15.59:0/2418441615 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:06:24.212 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.209+0000 7f33c3fff700 10 monclient(hunting): get_auth_request con 0x7f33c4142290 auth_method 0 2023-09-30T02:06:24.212 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.209+0000 7f33c3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:24.212 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.209+0000 7f33c3fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:24.212 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.209+0000 7f33c3fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:24.213 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.210+0000 7f35c142d700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:24.213 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.210+0000 7f35c142d700 10 monclient: _add_conns ranks=[0,4,3,2,1] 2023-09-30T02:06:24.214 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.210+0000 7f35c142d700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f35bc0b2f70 addr v1:172.21.15.59:6789/0 2023-09-30T02:06:24.214 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.210+0000 7f35c142d700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f35bc06dd20 0x7f35bc06e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:24.214 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.210+0000 7f35c142d700 10 monclient(hunting): picked mon.noname-c con 0x7f35bc06dd20 addr v2:172.21.15.110:3300/0 2023-09-30T02:06:24.214 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.210+0000 7f35c142d700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f35bc0b3340 0x7f35bc0abd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:24.214 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.210+0000 7f35c142d700 10 monclient(hunting): picked mon.noname-a con 0x7f35bc0b3340 addr v2:172.21.15.59:3300/0 2023-09-30T02:06:24.214 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.210+0000 7f35c142d700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35bc139cc0 con 0x7f35bc0b2f70 2023-09-30T02:06:24.214 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.210+0000 7f35c142d700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:24.214 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.210+0000 7f35c142d700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35bc06f3c0 con 0x7f35bc0b3340 2023-09-30T02:06:24.214 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.210+0000 7f35c142d700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:24.214 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.210+0000 7f35c142d700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35bc06f6a0 con 0x7f35bc06dd20 2023-09-30T02:06:24.214 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.210+0000 7f35c142d700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:24.215 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.210+0000 7f33c3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:24.215 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.210+0000 7f35c142d700 10 monclient(hunting): authenticate will time out at 1829.512195s 2023-09-30T02:06:24.216 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.210+0000 7f33c3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:24.216 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.210+0000 7f33c3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:24.216 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.212+0000 7f33c3fff700 10 monclient(hunting): handle_auth_done global_id 4323 payload 274 2023-09-30T02:06:24.216 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.212+0000 7f33c3fff700 1 -- 172.21.15.59:0/2418441615 >> v2:172.21.15.110:3300/0 conn(0x7f33c40a5f40 msgr2=0x7f33c413af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:06:24.216 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.212+0000 7f33c3fff700 1 --2- 172.21.15.59:0/2418441615 >> v2:172.21.15.110:3300/0 conn(0x7f33c40a5f40 0x7f33c413af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:24.216 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.212+0000 7f33c3fff700 1 -- 172.21.15.59:0/2418441615 >> v1:172.21.15.110:6789/0 conn(0x7f33c40a5b70 legacy=0x7f33c4141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:06:24.216 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.212+0000 7f33c3fff700 10 monclient: _finish_hunting 0 2023-09-30T02:06:24.216 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.212+0000 7f33c3fff700 1 monclient: found mon.noname-a 2023-09-30T02:06:24.216 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.212+0000 7f33c3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:24.216 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.212+0000 7f33c3fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.59:3300/0 2023-09-30T02:06:24.217 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.212+0000 7f33c3fff700 1 -- 172.21.15.59:0/2418441615 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33c413b470 con 0x7f33c4142290 2023-09-30T02:06:24.217 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.212+0000 7f35bb7fe700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f35bc0b3340 0x7f35bc0abd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:24.217 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.215+0000 7f35baffd700 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f35bc0b2f70 0x7f35bc13f600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:35080/0 (socket says 172.21.15.59:35080) 2023-09-30T02:06:24.217 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.215+0000 7f35baffd700 1 -- 172.21.15.59:0/2812428832 learned_addr learned my addr 172.21.15.59:0/2812428832 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:06:24.217 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.215+0000 7f35bb7fe700 10 monclient(hunting): get_auth_request con 0x7f35bc0b3340 auth_method 0 2023-09-30T02:06:24.218 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.215+0000 7f35bb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:24.218 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.215+0000 7f35bb7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:24.218 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.215+0000 7f35bb7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:24.218 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.215+0000 7f33c3fff700 1 --2- 172.21.15.59:0/2418441615 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33c4142290 0x7f33c40b1500 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f33c40b0640 tx=0x7f33b8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33646e8a3f998b8d server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:24.218 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.217+0000 7f35bb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:24.219 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.217+0000 7f35bb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:24.219 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.218+0000 7f35bb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:24.219 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-09-30T02:06:24.222 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.222+0000 7f35bb7fe700 10 monclient(hunting): handle_auth_done global_id 4326 payload 274 2023-09-30T02:06:24.222 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.222+0000 7f35bb7fe700 1 -- 172.21.15.59:0/2812428832 >> v2:172.21.15.110:3300/0 conn(0x7f35bc06dd20 msgr2=0x7f35bc06e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:06:24.222 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.222+0000 7f35bb7fe700 1 --2- 172.21.15.59:0/2812428832 >> v2:172.21.15.110:3300/0 conn(0x7f35bc06dd20 0x7f35bc06e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:24.222 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.222+0000 7f35bb7fe700 1 -- 172.21.15.59:0/2812428832 >> v1:172.21.15.59:6789/0 conn(0x7f35bc0b2f70 legacy=0x7f35bc13f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:24.222 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.222+0000 7f35bb7fe700 10 monclient: _finish_hunting 0 2023-09-30T02:06:24.222 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.222+0000 7f35bb7fe700 1 monclient: found mon.noname-a 2023-09-30T02:06:24.222 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.222+0000 7f35bb7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:24.222 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.222+0000 7f35bb7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.59:3300/0 2023-09-30T02:06:24.222 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.222+0000 7f35bb7fe700 1 -- 172.21.15.59:0/2812428832 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35bc06f980 con 0x7f35bc0b3340 2023-09-30T02:06:24.224 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.224+0000 7f33c37fe700 1 -- 172.21.15.59:0/2418441615 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33b800b820 con 0x7f33c4142290 2023-09-30T02:06:24.224 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.224+0000 7f33c37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:24.224 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.224+0000 7f33c37fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T02:06:24.224 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.224+0000 7f33c37fe700 10 monclient: dump: 2023-09-30T02:06:24.224 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:24.224 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:24.224 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:24.224 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:24.224 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:24.224 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:24.225 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:24.225 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:24.225 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:24.225 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:24.225 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.224+0000 7f33c37fe700 10 monclient: _finish_auth 0 2023-09-30T02:06:24.225 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.224+0000 7f33c37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:24.225 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.224+0000 7f33c37fe700 1 -- 172.21.15.59:0/2418441615 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33b800b980 con 0x7f33c4142290 2023-09-30T02:06:24.225 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.224+0000 7f33cac6c700 5 monclient: authenticate success, global_id 4323 2023-09-30T02:06:24.225 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.224+0000 7f33cac6c700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:06:24.225 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.224+0000 7f33c37fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:24.225 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.224+0000 7f33cac6c700 10 monclient: get_monmap_and_config success 2023-09-30T02:06:24.226 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.224+0000 7f33cac6c700 4 set_mon_vals no callback set 2023-09-30T02:06:24.226 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.224+0000 7f33c37fe700 1 -- 172.21.15.59:0/2418441615 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33b800bd20 con 0x7f33c4142290 2023-09-30T02:06:24.226 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.224+0000 7f33c37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:24.226 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.224+0000 7f33c37fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T02:06:24.226 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.224+0000 7f33c37fe700 10 monclient: dump: 2023-09-30T02:06:24.226 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:24.226 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:24.226 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:24.226 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:24.226 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:24.226 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:24.226 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:24.227 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:24.227 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:24.227 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:24.229 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.225+0000 7f33cac6c700 10 monclient: shutdown 2023-09-30T02:06:24.229 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.225+0000 7f33cac6c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:24.229 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.225+0000 7f33cac6c700 1 -- 172.21.15.59:0/2418441615 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33c4142290 msgr2=0x7f33c40b1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:24.229 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.225+0000 7f33cac6c700 1 --2- 172.21.15.59:0/2418441615 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33c4142290 0x7f33c40b1500 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f33c40b0640 tx=0x7f33b8009250 comp rx=0 tx=0).stop 2023-09-30T02:06:24.229 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.225+0000 7f33cac6c700 1 -- 172.21.15.59:0/2418441615 shutdown_connections 2023-09-30T02:06:24.229 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.225+0000 7f33cac6c700 1 --2- 172.21.15.59:0/2418441615 >> v2:172.21.15.110:3300/0 conn(0x7f33c40a5f40 0x7f33c413af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:24.229 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.225+0000 7f33cac6c700 1 --2- 172.21.15.59:0/2418441615 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33c4142290 0x7f33c40b1500 secure :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f33c40b0640 tx=0x7f33b8009250 comp rx=0 tx=0).stop 2023-09-30T02:06:24.230 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.225+0000 7f33cac6c700 1 -- 172.21.15.59:0/2418441615 >> 172.21.15.59:0/2418441615 conn(0x7f33c40631f0 msgr2=0x7f33c40635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:24.230 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.225+0000 7f33cac6c700 1 -- 172.21.15.59:0/2418441615 shutdown_connections 2023-09-30T02:06:24.230 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.225+0000 7f33cac6c700 1 -- 172.21.15.59:0/2418441615 wait complete. 2023-09-30T02:06:24.230 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.226+0000 7f33cac6c700 10 monclient: build_initial_monmap 2023-09-30T02:06:24.230 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.226+0000 7f33cac6c700 10 monclient: monmap: 2023-09-30T02:06:24.230 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:24.230 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:06:24.230 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:06:24.230 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:06:24.230 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:24.230 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:06:24.230 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:06:24.231 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:06:24.231 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:06:24.231 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:24.231 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.226+0000 7f33cac6c700 1 librados: starting msgr at 2023-09-30T02:06:24.231 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.226+0000 7f33cac6c700 1 librados: starting objecter 2023-09-30T02:06:24.231 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.227+0000 7f33cac6c700 1 Processor -- start 2023-09-30T02:06:24.231 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.227+0000 7f33cac6c700 1 -- start start 2023-09-30T02:06:24.231 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.227+0000 7f33cac6c700 1 librados: setting wanted keys 2023-09-30T02:06:24.231 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.227+0000 7f33cac6c700 1 librados: calling monclient init 2023-09-30T02:06:24.231 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.227+0000 7f33cac6c700 10 monclient: init 2023-09-30T02:06:24.231 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.227+0000 7f33cac6c700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:24.232 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.227+0000 7f33cac6c700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T02:06:24.232 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.227+0000 7f33cac6c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f33c40a5b70 0x7f33c413c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:24.232 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.227+0000 7f33cac6c700 10 monclient(hunting): picked mon.noname-b con 0x7f33c40a5b70 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:06:24.232 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.228+0000 7f33cac6c700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f33c40a5f40 0x7f33c413dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:24.232 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.228+0000 7f33cac6c700 10 monclient(hunting): picked mon.noname-c con 0x7f33c40a5f40 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:24.232 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.228+0000 7f33cac6c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33c413e800 0x7f33c4151170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:24.232 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.228+0000 7f33cac6c700 10 monclient(hunting): picked mon.noname-a con 0x7f33c413e800 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:06:24.232 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.228+0000 7f33cac6c700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:24.232 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.228+0000 7f33cac6c700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33c413e370 con 0x7f33c413e800 2023-09-30T02:06:24.232 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.228+0000 7f33cac6c700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:24.232 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.228+0000 7f33cac6c700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33c4141450 con 0x7f33c40a5b70 2023-09-30T02:06:24.232 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.228+0000 7f33cac6c700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:24.232 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.228+0000 7f33cac6c700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33c41415c0 con 0x7f33c40a5f40 2023-09-30T02:06:24.233 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.228+0000 7f33cac6c700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:24.233 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.228+0000 7f33cac6c700 10 monclient(hunting): authenticate will time out at 1829.530074s 2023-09-30T02:06:24.233 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.229+0000 7f33c3fff700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f33c40a5f40 0x7f33c413dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:24.233 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.229+0000 7f33c3fff700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f33c40a5f40 0x7f33c413dbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:47752/0 (socket says 172.21.15.59:47752) 2023-09-30T02:06:24.233 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.229+0000 7f33c3fff700 1 -- 172.21.15.59:0/2578571364 learned_addr learned my addr 172.21.15.59:0/2578571364 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:06:24.233 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.229+0000 7f33c3fff700 10 monclient(hunting): get_auth_request con 0x7f33c40a5f40 auth_method 0 2023-09-30T02:06:24.233 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.229+0000 7f33c3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:24.233 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.229+0000 7f33c3fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:24.233 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.229+0000 7f33c3fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:24.234 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.229+0000 7f35bb7fe700 1 --2- 172.21.15.59:0/2812428832 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35bc0b3340 0x7f35bc0abd10 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f35bc0af160 tx=0x7f35b4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=615f272fd6904e5a server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:24.234 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.230+0000 7f33c3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:24.234 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.230+0000 7f33c3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:24.234 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.230+0000 7f33c3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:24.235 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.234+0000 7f33c3fff700 10 monclient(hunting): handle_auth_done global_id 4325 payload 824 2023-09-30T02:06:24.235 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.234+0000 7f33c3fff700 1 -- 172.21.15.59:0/2578571364 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f33c40a5b70 msgr2=0x7f33c413c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:24.235 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.234+0000 7f33c3fff700 1 --2- 172.21.15.59:0/2578571364 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f33c40a5b70 0x7f33c413c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:24.235 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.234+0000 7f33c3fff700 1 -- 172.21.15.59:0/2578571364 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33c413e800 msgr2=0x7f33c4151170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:24.235 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.234+0000 7f33c3fff700 1 --2- 172.21.15.59:0/2578571364 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33c413e800 0x7f33c4151170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:24.235 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.234+0000 7f33c3fff700 10 monclient: _finish_hunting 0 2023-09-30T02:06:24.235 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.234+0000 7f33c3fff700 1 monclient: found mon.noname-c 2023-09-30T02:06:24.235 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.234+0000 7f33c3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:24.235 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.234+0000 7f33c3fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.59:3301/0 2023-09-30T02:06:24.235 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.234+0000 7f33c3fff700 1 -- 172.21.15.59:0/2578571364 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33b80094f0 con 0x7f33c40a5f40 2023-09-30T02:06:24.236 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.237+0000 7f33c3fff700 1 --2- 172.21.15.59:0/2578571364 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f33c40a5f40 0x7f33c413dbe0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f33b8009290 tx=0x7f33b8016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:24.236 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.237+0000 7f33c0ff9700 10 client.?.objecter ms_handle_connect 0x7f33c40a5f40 2023-09-30T02:06:24.236 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.237+0000 7f33c0ff9700 10 client.?.objecter resend_mon_ops 2023-09-30T02:06:24.239 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.239+0000 7f35b9ffb700 1 -- 172.21.15.59:0/2812428832 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35b400e070 con 0x7f35bc0b3340 2023-09-30T02:06:24.239 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.239+0000 7f35b9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:24.239 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.239+0000 7f35b9ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T02:06:24.239 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.239+0000 7f35b9ffb700 10 monclient: dump: 2023-09-30T02:06:24.239 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:24.239 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:24.239 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:24.239 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:24.240 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:24.240 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:24.240 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:24.240 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:24.240 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:24.240 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:24.240 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.239+0000 7f35b9ffb700 10 monclient: _finish_auth 0 2023-09-30T02:06:24.240 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.239+0000 7f35b9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:24.240 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.239+0000 7f35b9ffb700 1 -- 172.21.15.59:0/2812428832 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35b400bc30 con 0x7f35bc0b3340 2023-09-30T02:06:24.240 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.239+0000 7f35c142d700 5 monclient: authenticate success, global_id 4326 2023-09-30T02:06:24.240 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.239+0000 7f35c142d700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:06:24.240 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.239+0000 7f35b9ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:24.240 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.239+0000 7f35c142d700 10 monclient: get_monmap_and_config success 2023-09-30T02:06:24.241 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.239+0000 7f35c142d700 4 set_mon_vals no callback set 2023-09-30T02:06:24.241 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.239+0000 7f35b9ffb700 1 -- 172.21.15.59:0/2812428832 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35b4012430 con 0x7f35bc0b3340 2023-09-30T02:06:24.241 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.239+0000 7f35b9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:24.241 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.239+0000 7f35b9ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T02:06:24.241 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.239+0000 7f35b9ffb700 10 monclient: dump: 2023-09-30T02:06:24.241 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:24.241 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:24.241 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:24.241 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:24.241 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:24.241 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:24.241 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:24.242 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:24.242 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:24.242 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:24.243 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.240+0000 7f35c142d700 10 monclient: shutdown 2023-09-30T02:06:24.243 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.240+0000 7f35c142d700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:24.243 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.240+0000 7f35c142d700 1 -- 172.21.15.59:0/2812428832 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35bc0b3340 msgr2=0x7f35bc0abd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:24.243 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.240+0000 7f35c142d700 1 --2- 172.21.15.59:0/2812428832 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35bc0b3340 0x7f35bc0abd10 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f35bc0af160 tx=0x7f35b4009780 comp rx=0 tx=0).stop 2023-09-30T02:06:24.243 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.240+0000 7f35c142d700 1 -- 172.21.15.59:0/2812428832 shutdown_connections 2023-09-30T02:06:24.243 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.240+0000 7f35c142d700 1 --2- 172.21.15.59:0/2812428832 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35bc0b3340 0x7f35bc0abd10 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:24.243 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.240+0000 7f35c142d700 1 --2- 172.21.15.59:0/2812428832 >> v2:172.21.15.110:3300/0 conn(0x7f35bc06dd20 0x7f35bc06e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:24.243 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.240+0000 7f35c142d700 1 -- 172.21.15.59:0/2812428832 >> 172.21.15.59:0/2812428832 conn(0x7f35bc0637c0 msgr2=0x7f35bc0b2860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:24.244 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.241+0000 7f35c142d700 1 -- 172.21.15.59:0/2812428832 shutdown_connections 2023-09-30T02:06:24.244 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.241+0000 7f35c142d700 1 -- 172.21.15.59:0/2812428832 wait complete. 2023-09-30T02:06:24.244 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.242+0000 7f35c142d700 10 monclient: build_initial_monmap 2023-09-30T02:06:24.244 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.242+0000 7f35c142d700 10 monclient: monmap: 2023-09-30T02:06:24.244 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:24.244 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:06:24.244 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:06:24.244 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:06:24.244 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:24.244 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:06:24.244 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:06:24.244 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:06:24.245 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:06:24.245 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:24.245 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.242+0000 7f35c142d700 1 librados: starting msgr at 2023-09-30T02:06:24.245 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.242+0000 7f35c142d700 1 librados: starting objecter 2023-09-30T02:06:24.246 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.242+0000 7f33c0ff9700 1 -- 172.21.15.59:0/2578571364 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33b8026480 con 0x7f33c40a5f40 2023-09-30T02:06:24.246 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.242+0000 7f33c0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:24.246 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.242+0000 7f33c0ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:06:24.246 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.242+0000 7f33c0ff9700 10 monclient: dump: 2023-09-30T02:06:24.246 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:24.247 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:24.247 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:24.247 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:24.247 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:24.247 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:24.247 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:24.247 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:24.247 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:24.247 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:24.247 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.242+0000 7f33c0ff9700 10 monclient: _finish_auth 0 2023-09-30T02:06:24.247 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.242+0000 7f33c0ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:24.247 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.242+0000 7f33c0ff9700 1 -- 172.21.15.59:0/2578571364 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33b801f070 con 0x7f33c40a5f40 2023-09-30T02:06:24.248 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.242+0000 7f33c0ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:24.248 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.242+0000 7f33c0ff9700 1 -- 172.21.15.59:0/2578571364 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33b8026830 con 0x7f33c40a5f40 2023-09-30T02:06:24.248 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.242+0000 7f33c0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:24.248 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.242+0000 7f33c0ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:06:24.248 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.242+0000 7f33c0ff9700 10 monclient: dump: 2023-09-30T02:06:24.248 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:24.248 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:24.248 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:24.248 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:24.248 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:24.248 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:24.248 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:24.249 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:24.249 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:24.249 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:24.249 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.242+0000 7f33cac6c700 5 monclient: authenticate success, global_id 4325 2023-09-30T02:06:24.249 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.242+0000 7f33cac6c700 10 monclient: _renew_subs 2023-09-30T02:06:24.249 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.242+0000 7f33cac6c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:06:24.249 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.242+0000 7f33cac6c700 1 -- 172.21.15.59:0/2578571364 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33c40ae320 con 0x7f33c40a5f40 2023-09-30T02:06:24.249 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.242+0000 7f33c2ffd700 4 set_mon_vals no callback set 2023-09-30T02:06:24.249 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.242+0000 7f33cac6c700 10 client.4325.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:06:24.249 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.242+0000 7f33cac6c700 10 monclient: _renew_subs 2023-09-30T02:06:24.249 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.242+0000 7f33cac6c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:06:24.249 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.242+0000 7f33cac6c700 1 -- 172.21.15.59:0/2578571364 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33c4140a50 con 0x7f33c40a5f40 2023-09-30T02:06:24.250 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.242+0000 7f33cac6c700 1 librados: init done 2023-09-30T02:06:24.250 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-09-30T02:06:24.251 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.242+0000 7f35c142d700 1 Processor -- start 2023-09-30T02:06:24.251 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.243+0000 7f35c142d700 1 -- start start 2023-09-30T02:06:24.251 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.243+0000 7f35c142d700 1 librados: setting wanted keys 2023-09-30T02:06:24.251 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.243+0000 7f35c142d700 1 librados: calling monclient init 2023-09-30T02:06:24.251 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.243+0000 7f35c142d700 10 monclient: init 2023-09-30T02:06:24.252 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.244+0000 7f35c142d700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:24.252 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.244+0000 7f35c142d700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T02:06:24.252 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.244+0000 7f35c142d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f35bc06dd20 0x7f35bc13cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:24.252 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.244+0000 7f35c142d700 10 monclient(hunting): picked mon.noname-b con 0x7f35bc06dd20 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:06:24.252 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.244+0000 7f35c142d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35bc0b2f70 0x7f35bc14bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:24.252 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.244+0000 7f35c142d700 10 monclient(hunting): picked mon.noname-a con 0x7f35bc0b2f70 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:06:24.252 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.244+0000 7f35c142d700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f35bc0b3340 0x7f35bc14c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:24.252 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.244+0000 7f35c142d700 10 monclient(hunting): picked mon.noname-c con 0x7f35bc0b3340 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:24.252 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.244+0000 7f35c142d700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:24.252 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.244+0000 7f35c142d700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35bc14c7e0 con 0x7f35bc0b2f70 2023-09-30T02:06:24.252 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.244+0000 7f35c142d700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:24.252 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.244+0000 7f35c142d700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35bc139e70 con 0x7f35bc06dd20 2023-09-30T02:06:24.253 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.244+0000 7f35c142d700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:24.253 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.244+0000 7f35c142d700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35bc13a150 con 0x7f35bc0b3340 2023-09-30T02:06:24.253 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.244+0000 7f35c142d700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:24.253 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.244+0000 7f35c142d700 10 monclient(hunting): authenticate will time out at 1829.546206s 2023-09-30T02:06:24.253 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.244+0000 7f35ba7fc700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35bc0b2f70 0x7f35bc14bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:24.253 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.244+0000 7f33cac6c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:24.253 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.244+0000 7f33cac6c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:24.253 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.244+0000 7f33cac6c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:06:24.253 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.244+0000 7f33cac6c700 1 -- 172.21.15.59:0/2578571364 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33c4064a00 con 0x7f33c40a5f40 2023-09-30T02:06:24.255 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.245+0000 7f35ba7fc700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35bc0b2f70 0x7f35bc14bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52078/0 (socket says 172.21.15.59:52078) 2023-09-30T02:06:24.255 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.245+0000 7f35ba7fc700 1 -- 172.21.15.59:0/2524337962 learned_addr learned my addr 172.21.15.59:0/2524337962 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:06:24.255 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.245+0000 7f35ba7fc700 10 monclient(hunting): get_auth_request con 0x7f35bc0b2f70 auth_method 0 2023-09-30T02:06:24.255 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.245+0000 7f35ba7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:24.255 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.245+0000 7f35ba7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:24.255 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.245+0000 7f35ba7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:24.255 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.245+0000 7f35ba7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:24.255 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.245+0000 7f35ba7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:24.256 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.245+0000 7f35ba7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:24.256 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.249+0000 7f35ba7fc700 10 monclient(hunting): handle_auth_done global_id 4332 payload 824 2023-09-30T02:06:24.256 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.249+0000 7f35ba7fc700 1 -- 172.21.15.59:0/2524337962 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f35bc0b3340 msgr2=0x7f35bc14c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:24.256 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.249+0000 7f35ba7fc700 1 --2- 172.21.15.59:0/2524337962 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f35bc0b3340 0x7f35bc14c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:24.256 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.250+0000 7f35ba7fc700 1 -- 172.21.15.59:0/2524337962 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f35bc06dd20 msgr2=0x7f35bc13cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:24.256 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.250+0000 7f35ba7fc700 1 --2- 172.21.15.59:0/2524337962 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f35bc06dd20 0x7f35bc13cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:24.256 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.250+0000 7f35ba7fc700 10 monclient: _finish_hunting 0 2023-09-30T02:06:24.256 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.250+0000 7f35ba7fc700 1 monclient: found mon.noname-a 2023-09-30T02:06:24.256 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.250+0000 7f35ba7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:24.256 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.250+0000 7f35ba7fc700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.59:3300/0 2023-09-30T02:06:24.256 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.250+0000 7f35ba7fc700 1 -- 172.21.15.59:0/2524337962 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35b4009a20 con 0x7f35bc0b2f70 2023-09-30T02:06:24.256 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.253+0000 7f35ba7fc700 1 --2- 172.21.15.59:0/2524337962 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35bc0b2f70 0x7f35bc14bd60 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f35b000b810 tx=0x7f35b0009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:24.257 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.253+0000 7f35a2ffd700 10 client.?.objecter ms_handle_connect 0x7f35bc0b2f70 2023-09-30T02:06:24.257 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.253+0000 7f35a2ffd700 10 client.?.objecter resend_mon_ops 2023-09-30T02:06:24.258 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.258+0000 7f35a2ffd700 1 -- 172.21.15.59:0/2524337962 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35b000f5f0 con 0x7f35bc0b2f70 2023-09-30T02:06:24.258 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.258+0000 7f35a2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:24.258 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.258+0000 7f35a2ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T02:06:24.258 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.258+0000 7f35a2ffd700 10 monclient: dump: 2023-09-30T02:06:24.258 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:24.258 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:24.258 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:24.258 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:24.258 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:24.258 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:24.258 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:24.259 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:24.259 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:24.259 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:24.259 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.258+0000 7f35a2ffd700 10 monclient: _finish_auth 0 2023-09-30T02:06:24.259 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.258+0000 7f35a2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:24.262 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.258+0000 7f35a2ffd700 1 -- 172.21.15.59:0/2524337962 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35b000f770 con 0x7f35bc0b2f70 2023-09-30T02:06:24.262 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.258+0000 7f35a2ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:24.262 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.258+0000 7f35a2ffd700 1 -- 172.21.15.59:0/2524337962 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35b000fb30 con 0x7f35bc0b2f70 2023-09-30T02:06:24.262 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.258+0000 7f35a2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:24.262 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.258+0000 7f35a2ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T02:06:24.262 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.258+0000 7f35a2ffd700 10 monclient: dump: 2023-09-30T02:06:24.262 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:24.262 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:24.263 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:24.263 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:24.263 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:24.263 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:24.263 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:24.263 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:24.263 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:24.263 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:24.263 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.258+0000 7f35b97fa700 4 set_mon_vals no callback set 2023-09-30T02:06:24.263 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.258+0000 7f35c142d700 5 monclient: authenticate success, global_id 4332 2023-09-30T02:06:24.263 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.258+0000 7f35c142d700 10 monclient: _renew_subs 2023-09-30T02:06:24.263 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.258+0000 7f35c142d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:06:24.263 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.258+0000 7f35c142d700 1 -- 172.21.15.59:0/2524337962 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35bc13a640 con 0x7f35bc0b2f70 2023-09-30T02:06:24.264 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.259+0000 7f35c142d700 10 client.4332.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:06:24.264 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.259+0000 7f35c142d700 10 monclient: _renew_subs 2023-09-30T02:06:24.264 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.259+0000 7f35c142d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:06:24.264 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.259+0000 7f35c142d700 1 -- 172.21.15.59:0/2524337962 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35bc13a890 con 0x7f35bc0b2f70 2023-09-30T02:06:24.264 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.259+0000 7f35c142d700 1 librados: init done 2023-09-30T02:06:24.264 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.260+0000 7f35c142d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:24.264 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.260+0000 7f35c142d700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:24.264 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.260+0000 7f35c142d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:06:24.264 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.260+0000 7f35c142d700 1 -- 172.21.15.59:0/2524337962 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35bc064a00 con 0x7f35bc0b2f70 2023-09-30T02:06:24.276 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.277+0000 7f33c0ff9700 1 -- 172.21.15.59:0/2578571364 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f33b8026c20 con 0x7f33c40a5f40 2023-09-30T02:06:24.277 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.277+0000 7f33c0ff9700 1 --2- 172.21.15.59:0/2578571364 >> v2:172.21.15.110:6800/115870 conn(0x7f33b0068960 0x7f33b006ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:24.277 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.277+0000 7f33c0ff9700 1 -- 172.21.15.59:0/2578571364 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f33b8021070 con 0x7f33c40a5f40 2023-09-30T02:06:24.277 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.277+0000 7f33c0ff9700 10 client.4325.objecter ms_dispatch 0x7f33c4001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:06:24.277 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.277+0000 7f33c0ff9700 3 client.4325.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:06:24.277 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.277+0000 7f33c0ff9700 3 client.4325.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:06:24.277 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.277+0000 7f33c0ff9700 20 client.4325.objecter dump_active .. 0 homeless 2023-09-30T02:06:24.277 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.278+0000 7f33c8a08700 1 --2- 172.21.15.59:0/2578571364 >> v2:172.21.15.110:6800/115870 conn(0x7f33b0068960 0x7f33b006ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:24.277 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.278+0000 7f33c8a08700 10 monclient: get_auth_request con 0x7f33b0068960 auth_method 0 2023-09-30T02:06:24.278 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.278+0000 7f33c0ff9700 10 client.4325.objecter ms_handle_connect 0x7f33b0068960 2023-09-30T02:06:24.278 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.278+0000 7f33c8a08700 1 --2- 172.21.15.59:0/2578571364 >> v2:172.21.15.110:6800/115870 conn(0x7f33b0068960 0x7f33b006ae10 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f33b40097c0 tx=0x7f33b4006ce0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:24.292 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-09-30T02:06:24.331 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-09-30T02:06:24.355 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.356+0000 7f35a2ffd700 1 -- 172.21.15.59:0/2524337962 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f35b0010480 con 0x7f35bc0b2f70 2023-09-30T02:06:24.356 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.356+0000 7f35a2ffd700 1 --2- 172.21.15.59:0/2524337962 >> v2:172.21.15.110:6800/115870 conn(0x7f35a8068910 0x7f35a806adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:24.356 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.356+0000 7f35a2ffd700 1 -- 172.21.15.59:0/2524337962 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f35b00848d0 con 0x7f35bc0b2f70 2023-09-30T02:06:24.356 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.356+0000 7f35a2ffd700 10 client.4332.objecter ms_dispatch 0x7f35bc001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:06:24.356 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.356+0000 7f35a2ffd700 3 client.4332.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:06:24.356 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.356+0000 7f35a2ffd700 3 client.4332.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:06:24.356 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.356+0000 7f35baffd700 1 --2- 172.21.15.59:0/2524337962 >> v2:172.21.15.110:6800/115870 conn(0x7f35a8068910 0x7f35a806adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:24.356 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.356+0000 7f35a2ffd700 20 client.4332.objecter dump_active .. 0 homeless 2023-09-30T02:06:24.356 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.357+0000 7f35baffd700 10 monclient: get_auth_request con 0x7f35a8068910 auth_method 0 2023-09-30T02:06:24.357 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.357+0000 7f35a2ffd700 10 client.4332.objecter ms_handle_connect 0x7f35a8068910 2023-09-30T02:06:24.357 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.357+0000 7f35baffd700 1 --2- 172.21.15.59:0/2524337962 >> v2:172.21.15.110:6800/115870 conn(0x7f35a8068910 0x7f35a806adc0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f35ac0016a0 tx=0x7f35ac007b30 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:24.393 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.393+0000 7f079f90c700 10 monclient: get_monmap_and_config 2023-09-30T02:06:24.393 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.393+0000 7f079f90c700 10 monclient: build_initial_monmap 2023-09-30T02:06:24.393 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.393+0000 7f079f90c700 10 monclient: monmap: 2023-09-30T02:06:24.393 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:24.393 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:24.393 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:06:24.394141+0000 2023-09-30T02:06:24.393 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:06:24.394141+0000 2023-09-30T02:06:24.393 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:24.393 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:24.394 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:06:24.394 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:06:24.394 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:06:24.394 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:06:24.394 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:06:24.394 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:24.403 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.403+0000 7f079f90c700 1 Processor -- start 2023-09-30T02:06:24.403 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.404+0000 7f079f90c700 1 -- start start 2023-09-30T02:06:24.403 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.404+0000 7f079f90c700 10 monclient: init 2023-09-30T02:06:24.404 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.404+0000 7f079f90c700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:24.404 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.404+0000 7f079f90c700 10 monclient: _add_conns ranks=[1,2,0,3,4] 2023-09-30T02:06:24.404 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.405+0000 7f079f90c700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f07980a5b70 addr v1:172.21.15.110:6789/0 2023-09-30T02:06:24.404 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.405+0000 7f079f90c700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0798142290 0x7f07980b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:24.404 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.405+0000 7f079f90c700 10 monclient(hunting): picked mon.noname-b con 0x7f0798142290 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:24.404 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.405+0000 7f079f90c700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f07980a5f40 addr v1:172.21.15.59:6789/0 2023-09-30T02:06:24.404 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.405+0000 7f079f90c700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07980a6380 con 0x7f07980a5f40 2023-09-30T02:06:24.404 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.405+0000 7f079f90c700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07980a75a0 con 0x7f07980a5b70 2023-09-30T02:06:24.405 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.405+0000 7f079f90c700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:24.405 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.405+0000 7f079f90c700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07980af3f0 con 0x7f0798142290 2023-09-30T02:06:24.405 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.405+0000 7f079f90c700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:24.405 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.405+0000 7f079f90c700 10 monclient(hunting): authenticate will time out at 1829.707129s 2023-09-30T02:06:24.412 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.412+0000 7f27ea4d1700 10 monclient: get_monmap_and_config 2023-09-30T02:06:24.412 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.413+0000 7f27ea4d1700 10 monclient: build_initial_monmap 2023-09-30T02:06:24.412 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.413+0000 7f27ea4d1700 10 monclient: monmap: 2023-09-30T02:06:24.412 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:24.412 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:24.412 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:06:24.413418+0000 2023-09-30T02:06:24.412 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:06:24.413418+0000 2023-09-30T02:06:24.412 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:24.412 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:24.413 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:06:24.413 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:06:24.413 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:06:24.413 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:06:24.413 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:06:24.413 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:24.414 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.414+0000 7f27ea4d1700 1 Processor -- start 2023-09-30T02:06:24.414 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.414+0000 7f27ea4d1700 1 -- start start 2023-09-30T02:06:24.414 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.414+0000 7f27ea4d1700 10 monclient: init 2023-09-30T02:06:24.414 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.414+0000 7f27ea4d1700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:24.414 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.414+0000 7f27ea4d1700 10 monclient: _add_conns ranks=[4,0,2,1,3] 2023-09-30T02:06:24.414 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.414+0000 7f27ea4d1700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f27e40a5b70 0x7f27e40b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:24.414 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.414+0000 7f27ea4d1700 10 monclient(hunting): picked mon.noname-c con 0x7f27e40a5b70 addr v2:172.21.15.110:3300/0 2023-09-30T02:06:24.414 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.414+0000 7f27ea4d1700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f27e40a5f40 addr v1:172.21.15.59:6789/0 2023-09-30T02:06:24.415 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.414+0000 7f27ea4d1700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f27e4142290 0x7f27e413af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:24.415 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.414+0000 7f27ea4d1700 10 monclient(hunting): picked mon.noname-b con 0x7f27e4142290 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:24.415 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.415+0000 7f27ea4d1700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27e40a75a0 con 0x7f27e40a5f40 2023-09-30T02:06:24.415 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.415+0000 7f27ea4d1700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:24.415 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.415+0000 7f27ea4d1700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27e41426d0 con 0x7f27e40a5b70 2023-09-30T02:06:24.415 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.415+0000 7f27ea4d1700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:24.415 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.415+0000 7f27ea4d1700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27e40a63e0 con 0x7f27e4142290 2023-09-30T02:06:24.415 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.415+0000 7f27ea4d1700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:24.415 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.415+0000 7f27ea4d1700 10 monclient(hunting): authenticate will time out at 1829.717035s 2023-09-30T02:06:24.439 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.439+0000 7fc50a7fc700 1 -- 172.21.15.59:0/3488309627 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fc51001b3b0 con 0x7fc51c0a5f40 2023-09-30T02:06:24.440 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.440+0000 7fc50a7fc700 1 --2- 172.21.15.59:0/3488309627 >> v2:172.21.15.110:6800/115870 conn(0x7fc4fc068890 0x7fc4fc06ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:24.440 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.440+0000 7fc50a7fc700 1 -- 172.21.15.59:0/3488309627 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fc510083550 con 0x7fc51c0a5f40 2023-09-30T02:06:24.440 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.440+0000 7fc50a7fc700 10 client.4363.objecter ms_dispatch 0x7fc51c001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:06:24.440 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.440+0000 7fc50a7fc700 3 client.4363.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:06:24.440 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.440+0000 7fc50a7fc700 3 client.4363.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:06:24.440 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.440+0000 7fc50a7fc700 20 client.4363.objecter dump_active .. 0 homeless 2023-09-30T02:06:24.440 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.440+0000 7fc51a59c700 1 --2- 172.21.15.59:0/3488309627 >> v2:172.21.15.110:6800/115870 conn(0x7fc4fc068890 0x7fc4fc06ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:24.440 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.440+0000 7fc51a59c700 10 monclient: get_auth_request con 0x7fc4fc068890 auth_method 0 2023-09-30T02:06:24.441 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.440+0000 7fc50a7fc700 10 client.4363.objecter ms_handle_connect 0x7fc4fc068890 2023-09-30T02:06:24.441 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.440+0000 7fc51a59c700 1 --2- 172.21.15.59:0/3488309627 >> v2:172.21.15.110:6800/115870 conn(0x7fc4fc068890 0x7fc4fc06ad40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc50c0094c0 tx=0x7fc50c017000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:24.447 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.448+0000 7fc50a7fc700 1 -- 172.21.15.59:0/3488309627 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc510050e70 con 0x7fc51c0a5f40 2023-09-30T02:06:24.447 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.448+0000 7fc50a7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:24.448 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.448+0000 7fc50a7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:06:24.455 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.456+0000 7f27e3fff700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f27e40a5b70 0x7f27e40b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:24.456 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.456+0000 7f27e3fff700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f27e40a5b70 0x7f27e40b1500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:39156/0 (socket says 172.21.15.59:39156) 2023-09-30T02:06:24.456 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.457+0000 7f27e3fff700 1 -- 172.21.15.59:0/3214889340 learned_addr learned my addr 172.21.15.59:0/3214889340 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:06:24.456 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.457+0000 7f27e3fff700 10 monclient(hunting): get_auth_request con 0x7f27e40a5b70 auth_method 0 2023-09-30T02:06:24.456 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.457+0000 7f27e3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:24.456 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.457+0000 7f27e3fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:24.456 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.457+0000 7f27e3fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:24.457 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.458+0000 7f27e3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:24.457 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.458+0000 7f27e3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:24.458 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.458+0000 7f27e3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:24.460 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.461+0000 7f27e3fff700 10 monclient(hunting): handle_auth_done global_id 4366 payload 274 2023-09-30T02:06:24.460 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.461+0000 7f27e3fff700 1 -- 172.21.15.59:0/3214889340 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f27e4142290 msgr2=0x7f27e413af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:06:24.460 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.461+0000 7f27e3fff700 1 --2- 172.21.15.59:0/3214889340 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f27e4142290 0x7f27e413af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:24.460 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.461+0000 7f27e3fff700 1 -- 172.21.15.59:0/3214889340 >> v1:172.21.15.59:6789/0 conn(0x7f27e40a5f40 legacy=0x7f27e4141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:06:24.460 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.461+0000 7f27e3fff700 10 monclient: _finish_hunting 0 2023-09-30T02:06:24.460 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.461+0000 7f27e3fff700 1 monclient: found mon.noname-c 2023-09-30T02:06:24.460 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.461+0000 7f27e3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:24.461 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.461+0000 7f27e3fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3300/0 2023-09-30T02:06:24.461 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.461+0000 7f27e3fff700 1 -- 172.21.15.59:0/3214889340 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27e40af3f0 con 0x7f27e40a5b70 2023-09-30T02:06:24.461 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.461+0000 7fd0c1b86700 10 monclient: get_monmap_and_config 2023-09-30T02:06:24.461 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.461+0000 7fd0c1b86700 10 monclient: build_initial_monmap 2023-09-30T02:06:24.461 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.461+0000 7fd0c1b86700 10 monclient: monmap: 2023-09-30T02:06:24.462 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:24.462 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:24.462 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:06:24.461566+0000 2023-09-30T02:06:24.462 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:06:24.461566+0000 2023-09-30T02:06:24.462 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:24.462 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:24.462 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:06:24.462 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:06:24.462 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:06:24.462 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:06:24.462 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:06:24.463 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:24.463 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.462+0000 7fd0c1b86700 1 Processor -- start 2023-09-30T02:06:24.463 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.462+0000 7fd0c1b86700 1 -- start start 2023-09-30T02:06:24.463 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.462+0000 7fd0c1b86700 10 monclient: init 2023-09-30T02:06:24.463 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.463+0000 7fd0c1b86700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:24.463 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.463+0000 7fd0c1b86700 10 monclient: _add_conns ranks=[1,2,4,0,3] 2023-09-30T02:06:24.464 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.463+0000 7fd0c1b86700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd0bc06b810 addr v1:172.21.15.110:6789/0 2023-09-30T02:06:24.464 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.463+0000 7fd0c1b86700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd0bc06e710 0x7fd0bc139740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:24.464 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.463+0000 7fd0c1b86700 10 monclient(hunting): picked mon.noname-b con 0x7fd0bc06e710 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:24.464 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.463+0000 7fd0c1b86700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd0bc139c80 0x7fd0bc13a100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:24.464 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.463+0000 7fd0c1b86700 10 monclient(hunting): picked mon.noname-c con 0x7fd0bc139c80 addr v2:172.21.15.110:3300/0 2023-09-30T02:06:24.464 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.463+0000 7fd0c1b86700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0bc06bbe0 con 0x7fd0bc06b810 2023-09-30T02:06:24.465 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.463+0000 7fd0c1b86700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:24.465 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.463+0000 7fd0c1b86700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0bc13a640 con 0x7fd0bc139c80 2023-09-30T02:06:24.465 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.463+0000 7fd0c1b86700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:24.465 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.463+0000 7fd0c1b86700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0bc06eb20 con 0x7fd0bc06e710 2023-09-30T02:06:24.465 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.463+0000 7fd0c1b86700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:24.465 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.463+0000 7fd0c1b86700 10 monclient(hunting): authenticate will time out at 1829.765521s 2023-09-30T02:06:24.466 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.466+0000 7f27e3fff700 1 --2- 172.21.15.59:0/3214889340 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f27e40a5b70 0x7f27e40b1500 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f27d40099e0 tx=0x7f27d4009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8bed426c5b9513f0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:24.466 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.467+0000 7fd0bb7fe700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fd0bc06b810 0x7fd0bc06e000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.59:54888/0 (socket says 172.21.15.59:54888) 2023-09-30T02:06:24.466 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.467+0000 7fd0bb7fe700 1 -- 172.21.15.59:0/3170156779 learned_addr learned my addr 172.21.15.59:0/3170156779 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:06:24.466 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.467+0000 7fd0bbfff700 1 --2- 172.21.15.59:0/3170156779 >> v2:172.21.15.110:3300/0 conn(0x7fd0bc139c80 0x7fd0bc13a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:24.467 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.467+0000 7fd0bbfff700 10 monclient(hunting): get_auth_request con 0x7fd0bc139c80 auth_method 0 2023-09-30T02:06:24.467 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.467+0000 7fd0bbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:24.467 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.467+0000 7fd0bbfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:24.467 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.467+0000 7fd0bbfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:24.468 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.469+0000 7fd0bbfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:24.468 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.469+0000 7fd0bbfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:24.468 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.469+0000 7fd0bbfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:24.469 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.469+0000 7f079d6a8700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f07980a5b70 0x7f0798141b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.59:54878/0 (socket says 172.21.15.59:54878) 2023-09-30T02:06:24.469 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.470+0000 7f079d6a8700 1 -- 172.21.15.59:0/1929635689 learned_addr learned my addr 172.21.15.59:0/1929635689 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:06:24.473 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.474+0000 7fd0bbfff700 10 monclient(hunting): handle_auth_done global_id 4369 payload 274 2023-09-30T02:06:24.474 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.474+0000 7fd0bbfff700 1 -- 172.21.15.59:0/3170156779 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd0bc06e710 msgr2=0x7fd0bc139740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:06:24.474 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.474+0000 7fd0bbfff700 1 --2- 172.21.15.59:0/3170156779 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd0bc06e710 0x7fd0bc139740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:24.474 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.474+0000 7fd0bbfff700 1 -- 172.21.15.59:0/3170156779 >> v1:172.21.15.110:6789/0 conn(0x7fd0bc06b810 legacy=0x7fd0bc06e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:24.474 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.474+0000 7fd0bbfff700 10 monclient: _finish_hunting 0 2023-09-30T02:06:24.474 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.474+0000 7fd0bbfff700 1 monclient: found mon.noname-c 2023-09-30T02:06:24.474 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.474+0000 7fd0bbfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:24.474 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.474+0000 7fd0bbfff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3300/0 2023-09-30T02:06:24.474 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.474+0000 7fd0bbfff700 1 -- 172.21.15.59:0/3170156779 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0bc06c4e0 con 0x7fd0bc139c80 2023-09-30T02:06:24.477 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.478+0000 7fd0bbfff700 1 --2- 172.21.15.59:0/3170156779 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd0bc139c80 0x7fd0bc13a100 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd0b400afa0 tx=0x7fd0b400a890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=294bfc55d21823e server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:24.483 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.483+0000 7f27e2ffd700 1 -- 172.21.15.59:0/3214889340 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27d400b7e0 con 0x7f27e40a5b70 2023-09-30T02:06:24.483 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.483+0000 7f27e2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:24.483 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.483+0000 7f27e2ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:06:24.483 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.483+0000 7f27e2ffd700 10 monclient: dump: 2023-09-30T02:06:24.483 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:24.483 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:24.483 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:24.484 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:24.484 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:24.484 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:24.484 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:24.484 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:24.484 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:24.484 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:24.484 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.483+0000 7f27e2ffd700 10 monclient: _finish_auth 0 2023-09-30T02:06:24.484 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.483+0000 7f27e2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:24.484 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.483+0000 7f27e2ffd700 1 -- 172.21.15.59:0/3214889340 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27d400b940 con 0x7f27e40a5b70 2023-09-30T02:06:24.484 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.483+0000 7f27ea4d1700 5 monclient: authenticate success, global_id 4366 2023-09-30T02:06:24.484 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.483+0000 7f27ea4d1700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:06:24.485 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.483+0000 7f27e2ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:24.485 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.483+0000 7f27ea4d1700 10 monclient: get_monmap_and_config success 2023-09-30T02:06:24.485 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.483+0000 7f27e2ffd700 1 -- 172.21.15.59:0/3214889340 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27d400bce0 con 0x7f27e40a5b70 2023-09-30T02:06:24.485 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.483+0000 7f27ea4d1700 4 set_mon_vals no callback set 2023-09-30T02:06:24.485 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.483+0000 7f27e2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:24.485 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.483+0000 7f27e2ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:06:24.485 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.483+0000 7f27e2ffd700 10 monclient: dump: 2023-09-30T02:06:24.485 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:24.485 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:24.486 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:24.486 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:24.486 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:24.486 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:24.486 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:24.486 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:24.486 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:24.486 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:24.486 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.483+0000 7f27ea4d1700 10 monclient: shutdown 2023-09-30T02:06:24.486 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.483+0000 7f27ea4d1700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:24.487 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.483+0000 7f27ea4d1700 1 -- 172.21.15.59:0/3214889340 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f27e40a5b70 msgr2=0x7f27e40b1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:24.487 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.483+0000 7f27ea4d1700 1 --2- 172.21.15.59:0/3214889340 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f27e40a5b70 0x7f27e40b1500 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f27d40099e0 tx=0x7f27d4009250 comp rx=0 tx=0).stop 2023-09-30T02:06:24.487 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.484+0000 7f27ea4d1700 1 -- 172.21.15.59:0/3214889340 shutdown_connections 2023-09-30T02:06:24.487 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.484+0000 7f27ea4d1700 1 --2- 172.21.15.59:0/3214889340 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f27e40a5b70 0x7f27e40b1500 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:24.487 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.484+0000 7f27ea4d1700 1 --2- 172.21.15.59:0/3214889340 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f27e4142290 0x7f27e413af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:24.487 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.484+0000 7f27ea4d1700 1 -- 172.21.15.59:0/3214889340 >> 172.21.15.59:0/3214889340 conn(0x7f27e40631f0 msgr2=0x7f27e40635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:24.487 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.484+0000 7f27ea4d1700 1 -- 172.21.15.59:0/3214889340 shutdown_connections 2023-09-30T02:06:24.487 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.484+0000 7f27ea4d1700 1 -- 172.21.15.59:0/3214889340 wait complete. 2023-09-30T02:06:24.488 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.484+0000 7f27ea4d1700 10 monclient: build_initial_monmap 2023-09-30T02:06:24.488 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.484+0000 7f27ea4d1700 10 monclient: monmap: 2023-09-30T02:06:24.488 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:24.488 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:06:24.488 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:06:24.488 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:06:24.488 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:24.489 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:06:24.489 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:06:24.489 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:06:24.489 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:06:24.489 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:24.489 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.485+0000 7f27ea4d1700 1 librados: starting msgr at 2023-09-30T02:06:24.489 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.485+0000 7f27ea4d1700 1 librados: starting objecter 2023-09-30T02:06:24.489 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.485+0000 7f27ea4d1700 1 Processor -- start 2023-09-30T02:06:24.489 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.485+0000 7f078ffff700 1 -- 172.21.15.59:0/1929635689 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 674646144 0 0) 0x7f07980a75a0 con 0x7f07980a5b70 2023-09-30T02:06:24.490 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.485+0000 7f078ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:24.490 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.485+0000 7f078ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:24.490 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.485+0000 7f078ffff700 10 monclient(hunting): my global_id is 4375 2023-09-30T02:06:24.490 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.485+0000 7f078ffff700 1 -- 172.21.15.59:0/1929635689 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0784003570 con 0x7f07980a5b70 2023-09-30T02:06:24.491 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.485+0000 7f27ea4d1700 1 -- start start 2023-09-30T02:06:24.491 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.485+0000 7f27ea4d1700 1 librados: setting wanted keys 2023-09-30T02:06:24.491 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.485+0000 7f27ea4d1700 1 librados: calling monclient init 2023-09-30T02:06:24.491 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.485+0000 7f27ea4d1700 10 monclient: init 2023-09-30T02:06:24.491 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.486+0000 7f27ea4d1700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:24.491 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.486+0000 7f27ea4d1700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T02:06:24.492 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.486+0000 7f27ea4d1700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f27e40a5b70 0x7f27e41400f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:24.492 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.486+0000 7f27ea4d1700 10 monclient(hunting): picked mon.noname-c con 0x7f27e40a5b70 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:24.492 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.486+0000 7f27ea4d1700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27e40a5f40 0x7f27e4154970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:24.492 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.486+0000 7f27ea4d1700 10 monclient(hunting): picked mon.noname-a con 0x7f27e40a5f40 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:06:24.492 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.486+0000 7f27ea4d1700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f27e4142290 0x7f27e413e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:24.492 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.486+0000 7f27ea4d1700 10 monclient(hunting): picked mon.noname-b con 0x7f27e4142290 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:06:24.492 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.486+0000 7f27ea4d1700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:24.492 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.486+0000 7f27ea4d1700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27e413c030 con 0x7f27e40a5f40 2023-09-30T02:06:24.492 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.486+0000 7f27ea4d1700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:24.492 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.486+0000 7f27ea4d1700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27e4140630 con 0x7f27e4142290 2023-09-30T02:06:24.492 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.486+0000 7f27ea4d1700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:24.493 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.486+0000 7f27ea4d1700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27e4137620 con 0x7f27e40a5b70 2023-09-30T02:06:24.493 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.486+0000 7f27ea4d1700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:24.493 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.486+0000 7f27ea4d1700 10 monclient(hunting): authenticate will time out at 1829.788345s 2023-09-30T02:06:24.495 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.485+0000 7fd0ba7fc700 1 -- 172.21.15.59:0/3170156779 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0b400f070 con 0x7fd0bc139c80 2023-09-30T02:06:24.495 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.486+0000 7fd0ba7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:24.495 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.486+0000 7fd0ba7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:06:24.495 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.486+0000 7fd0ba7fc700 10 monclient: dump: 2023-09-30T02:06:24.495 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:24.495 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:24.495 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:24.495 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:24.495 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:24.496 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:24.496 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:24.496 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:24.496 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:24.496 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:24.496 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.486+0000 7fd0ba7fc700 10 monclient: _finish_auth 0 2023-09-30T02:06:24.496 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.486+0000 7fd0ba7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:24.497 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.486+0000 7fd0ba7fc700 1 -- 172.21.15.59:0/3170156779 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0b400a2e0 con 0x7fd0bc139c80 2023-09-30T02:06:24.497 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.486+0000 7fd0ba7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:24.497 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.486+0000 7fd0ba7fc700 1 -- 172.21.15.59:0/3170156779 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0b400cd10 con 0x7fd0bc139c80 2023-09-30T02:06:24.497 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.486+0000 7fd0ba7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:24.497 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.486+0000 7fd0ba7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:06:24.497 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.486+0000 7fd0ba7fc700 10 monclient: dump: 2023-09-30T02:06:24.497 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:24.497 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:24.498 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:24.498 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:24.498 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:24.498 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:24.498 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:24.498 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:24.498 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:24.498 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:24.499 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.486+0000 7fd0c1b86700 5 monclient: authenticate success, global_id 4369 2023-09-30T02:06:24.499 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.486+0000 7fd0c1b86700 10 monclient: get_monmap_and_config success 2023-09-30T02:06:24.499 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.486+0000 7fd0c1b86700 4 set_mon_vals no callback set 2023-09-30T02:06:24.499 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.487+0000 7fd0c1b86700 10 monclient: shutdown 2023-09-30T02:06:24.499 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.487+0000 7fd0c1b86700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:24.499 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.487+0000 7fd0c1b86700 1 -- 172.21.15.59:0/3170156779 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd0bc139c80 msgr2=0x7fd0bc13a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:24.499 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.487+0000 7fd0c1b86700 1 --2- 172.21.15.59:0/3170156779 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd0bc139c80 0x7fd0bc13a100 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd0b400afa0 tx=0x7fd0b400a890 comp rx=0 tx=0).stop 2023-09-30T02:06:24.500 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.487+0000 7fd0c1b86700 1 -- 172.21.15.59:0/3170156779 shutdown_connections 2023-09-30T02:06:24.500 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.487+0000 7fd0c1b86700 1 --2- 172.21.15.59:0/3170156779 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd0bc139c80 0x7fd0bc13a100 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:24.500 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.487+0000 7fd0c1b86700 1 --2- 172.21.15.59:0/3170156779 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd0bc06e710 0x7fd0bc139740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:24.500 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.487+0000 7fd0c1b86700 1 -- 172.21.15.59:0/3170156779 >> 172.21.15.59:0/3170156779 conn(0x7fd0bc067620 msgr2=0x7fd0bc067a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:24.500 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.487+0000 7fd0c1b86700 1 -- 172.21.15.59:0/3170156779 shutdown_connections 2023-09-30T02:06:24.500 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.487+0000 7fd0c1b86700 1 -- 172.21.15.59:0/3170156779 wait complete. 2023-09-30T02:06:24.500 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.487+0000 7fd0c1b86700 10 monclient: build_initial_monmap 2023-09-30T02:06:24.501 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.487+0000 7fd0c1b86700 10 monclient: monmap: 2023-09-30T02:06:24.501 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:24.501 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:06:24.501 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:06:24.501 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:06:24.501 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:24.501 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:06:24.501 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:06:24.502 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:06:24.502 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:06:24.502 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:24.502 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.488+0000 7fd0c1b86700 1 librados: starting msgr at 2023-09-30T02:06:24.502 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.488+0000 7fd0c1b86700 1 librados: starting objecter 2023-09-30T02:06:24.502 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.488+0000 7fd0c1b86700 1 Processor -- start 2023-09-30T02:06:24.503 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.488+0000 7fd0c1b86700 1 -- start start 2023-09-30T02:06:24.503 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.488+0000 7fd0c1b86700 1 librados: setting wanted keys 2023-09-30T02:06:24.503 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.488+0000 7fd0c1b86700 1 librados: calling monclient init 2023-09-30T02:06:24.503 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.488+0000 7fd0c1b86700 10 monclient: init 2023-09-30T02:06:24.503 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.489+0000 7fd0c1b86700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:24.503 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.489+0000 7fd0c1b86700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T02:06:24.503 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.489+0000 7fd0c1b86700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd0bc06b810 0x7fd0bc142960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:24.504 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.489+0000 7fd0c1b86700 10 monclient(hunting): picked mon.noname-b con 0x7fd0bc06b810 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:06:24.504 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.489+0000 7fd0c1b86700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd0bc06e710 0x7fd0bc118a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:24.504 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.489+0000 7fd0c1b86700 10 monclient(hunting): picked mon.noname-c con 0x7fd0bc06e710 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:24.504 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.489+0000 7fd0c1b86700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd0bc139c80 0x7fd0bc0a6970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:24.504 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.489+0000 7fd0c1b86700 10 monclient(hunting): picked mon.noname-a con 0x7fd0bc139c80 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:06:24.504 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.489+0000 7fd0c1b86700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:24.504 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.489+0000 7fd0c1b86700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0bc0a71f0 con 0x7fd0bc139c80 2023-09-30T02:06:24.504 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.489+0000 7fd0c1b86700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:24.505 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.489+0000 7fd0c1b86700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0bc0a6eb0 con 0x7fd0bc06b810 2023-09-30T02:06:24.505 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.489+0000 7fd0c1b86700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:24.505 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.489+0000 7fd0c1b86700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0bc14e130 con 0x7fd0bc06e710 2023-09-30T02:06:24.505 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.489+0000 7fd0c1b86700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:24.505 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.489+0000 7fd0c1b86700 10 monclient(hunting): authenticate will time out at 1829.791703s 2023-09-30T02:06:24.505 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.490+0000 7f078ffff700 1 -- 172.21.15.59:0/1929635689 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3511655744 0 0) 0x7f0784003570 con 0x7f07980a5b70 2023-09-30T02:06:24.506 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.490+0000 7f078ffff700 1 -- 172.21.15.59:0/1929635689 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0798142290 msgr2=0x7f07980b1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:06:24.506 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.490+0000 7f078ffff700 1 --2- 172.21.15.59:0/1929635689 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0798142290 0x7f07980b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:24.506 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.490+0000 7f078ffff700 1 -- 172.21.15.59:0/1929635689 >> v1:172.21.15.59:6789/0 conn(0x7f07980a5f40 legacy=0x7f079813af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:06:24.506 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.490+0000 7f078ffff700 10 monclient: _finish_hunting 0 2023-09-30T02:06:24.506 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.490+0000 7f078ffff700 1 monclient: found mon.noname-c-legacy 2023-09-30T02:06:24.506 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.490+0000 7f078ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:24.506 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.490+0000 7f078ffff700 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.110:6789/0 2023-09-30T02:06:24.507 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.490+0000 7f078ffff700 1 -- 172.21.15.59:0/1929635689 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f079813b640 con 0x7f07980a5b70 2023-09-30T02:06:24.507 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.490+0000 7f078ffff700 10 monclient: _finish_auth 0 2023-09-30T02:06:24.507 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.490+0000 7f078ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:24.507 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.490+0000 7f079f90c700 5 monclient: authenticate success, global_id 4375 2023-09-30T02:06:24.507 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.490+0000 7f079f90c700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:06:24.507 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.490+0000 7fd0bb7fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd0bc06b810 0x7fd0bc142960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:24.508 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.491+0000 7fd0bb7fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd0bc06b810 0x7fd0bc142960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:39186/0 (socket says 172.21.15.59:39186) 2023-09-30T02:06:24.508 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.491+0000 7fd0bb7fe700 1 -- 172.21.15.59:0/725419722 learned_addr learned my addr 172.21.15.59:0/725419722 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:06:24.508 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.491+0000 7fd0bb7fe700 10 monclient(hunting): get_auth_request con 0x7fd0bc06b810 auth_method 0 2023-09-30T02:06:24.508 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.491+0000 7fd0bb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:24.508 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.491+0000 7fd0bb7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:24.508 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.491+0000 7fd0bb7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:24.508 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.491+0000 7f27e8a6e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f27e4142290 0x7f27e413e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:24.510 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.491+0000 7f078ffff700 1 -- 172.21.15.59:0/1929635689 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2174283827 0 0) 0x7f0794001470 con 0x7f07980a5b70 2023-09-30T02:06:24.510 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.491+0000 7f078ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:24.510 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.491+0000 7f078ffff700 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2023-09-30T02:06:24.510 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.491+0000 7f078ffff700 10 monclient: dump: 2023-09-30T02:06:24.511 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:24.511 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:24.511 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:24.511 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:24.511 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:24.511 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:24.511 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:24.512 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:24.512 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:24.512 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:24.512 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.491+0000 7f078ffff700 1 monclient: mon.b has (v2) addrs [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] but i'm connected to v1:172.21.15.110:6789/0, reconnecting 2023-09-30T02:06:24.512 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.491+0000 7f078ffff700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:24.513 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.491+0000 7f078ffff700 1 -- 172.21.15.59:0/1929635689 >> v1:172.21.15.110:6789/0 conn(0x7f07980a5b70 legacy=0x7f0798141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:24.513 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.491+0000 7f078ffff700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T02:06:24.513 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.491+0000 7f078ffff700 1 --2- 172.21.15.59:0/1929635689 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0784003a60 0x7f0784003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:24.513 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.491+0000 7f078ffff700 10 monclient(hunting): picked mon.b con 0x7f0784003a60 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:06:24.513 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.491+0000 7f078ffff700 1 --2- 172.21.15.59:0/1929635689 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0798142290 0x7f0784004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:24.513 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.491+0000 7f078ffff700 10 monclient(hunting): picked mon.c con 0x7f0798142290 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:24.513 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.491+0000 7f078ffff700 1 --2- 172.21.15.59:0/1929635689 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0784004e70 0x7f07840052e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:24.513 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.491+0000 7f078ffff700 10 monclient(hunting): picked mon.a con 0x7f0784004e70 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:06:24.514 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.491+0000 7f078ffff700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:24.514 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.491+0000 7f078ffff700 1 -- 172.21.15.59:0/1929635689 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07980af3f0 con 0x7f0784004e70 2023-09-30T02:06:24.514 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.491+0000 7f078ffff700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:24.514 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.491+0000 7f078ffff700 1 -- 172.21.15.59:0/1929635689 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f078400bcf0 con 0x7f0784003a60 2023-09-30T02:06:24.514 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.491+0000 7f078ffff700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:24.514 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.491+0000 7f078ffff700 1 -- 172.21.15.59:0/1929635689 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f078400beb0 con 0x7f0798142290 2023-09-30T02:06:24.514 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.491+0000 7f078ffff700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:24.515 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.491+0000 7f078ffff700 10 monclient(hunting): _finish_auth 0 2023-09-30T02:06:24.515 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.491+0000 7f079f90c700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-09-30T02:06:24.515 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.493+0000 7f27e8a6e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f27e4142290 0x7f27e413e100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:39170/0 (socket says 172.21.15.59:39170) 2023-09-30T02:06:24.515 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.493+0000 7f27e8a6e700 1 -- 172.21.15.59:0/1035930162 learned_addr learned my addr 172.21.15.59:0/1035930162 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:06:24.515 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.493+0000 7f27e8a6e700 10 monclient(hunting): get_auth_request con 0x7f27e4142290 auth_method 0 2023-09-30T02:06:24.516 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.493+0000 7f27e8a6e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:24.516 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.493+0000 7f27e8a6e700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:24.516 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.493+0000 7f27e8a6e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:24.516 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.494+0000 7fd0bb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:24.516 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.494+0000 7fd0bb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:24.517 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.494+0000 7fd0bb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:24.517 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.495+0000 7f27e8a6e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:24.517 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.495+0000 7f27e8a6e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:24.517 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.495+0000 7f27e8a6e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:24.519 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.497+0000 7fd0bb7fe700 10 monclient(hunting): handle_auth_done global_id 4378 payload 824 2023-09-30T02:06:24.519 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.497+0000 7fd0bb7fe700 1 -- 172.21.15.59:0/725419722 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd0bc06e710 msgr2=0x7fd0bc118a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:24.519 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.497+0000 7fd0bb7fe700 1 --2- 172.21.15.59:0/725419722 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd0bc06e710 0x7fd0bc118a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:24.519 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.497+0000 7fd0bb7fe700 1 -- 172.21.15.59:0/725419722 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd0bc139c80 msgr2=0x7fd0bc0a6970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:24.519 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.497+0000 7fd0bb7fe700 1 --2- 172.21.15.59:0/725419722 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd0bc139c80 0x7fd0bc0a6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:24.519 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.497+0000 7fd0bb7fe700 10 monclient: _finish_hunting 0 2023-09-30T02:06:24.519 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.497+0000 7fd0bb7fe700 1 monclient: found mon.noname-b 2023-09-30T02:06:24.520 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.497+0000 7fd0bb7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:24.520 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.497+0000 7fd0bb7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:06:24.520 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.497+0000 7fd0bb7fe700 1 -- 172.21.15.59:0/725419722 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0b400ab30 con 0x7fd0bc06b810 2023-09-30T02:06:24.520 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.501+0000 7fd0bb7fe700 1 --2- 172.21.15.59:0/725419722 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd0bc06b810 0x7fd0bc142960 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd0ac00c7c0 tx=0x7fd0ac007f70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:24.520 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.501+0000 7fd0a37fe700 10 client.?.objecter ms_handle_connect 0x7fd0bc06b810 2023-09-30T02:06:24.521 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.501+0000 7fd0a37fe700 10 client.?.objecter resend_mon_ops 2023-09-30T02:06:24.521 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.504+0000 7f27e8a6e700 10 monclient(hunting): handle_auth_done global_id 4381 payload 824 2023-09-30T02:06:24.521 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.504+0000 7f27e8a6e700 1 -- 172.21.15.59:0/1035930162 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f27e40a5b70 msgr2=0x7f27e41400f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:24.522 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.505+0000 7f27e8a6e700 1 --2- 172.21.15.59:0/1035930162 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f27e40a5b70 0x7f27e41400f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:24.522 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.505+0000 7f27e8a6e700 1 -- 172.21.15.59:0/1035930162 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27e40a5f40 msgr2=0x7f27e4154970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:24.522 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.505+0000 7f27e8a6e700 1 --2- 172.21.15.59:0/1035930162 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27e40a5f40 0x7f27e4154970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:24.522 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.505+0000 7f27e8a6e700 10 monclient: _finish_hunting 0 2023-09-30T02:06:24.522 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.505+0000 7f27e8a6e700 1 monclient: found mon.noname-b 2023-09-30T02:06:24.522 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.505+0000 7f27e8a6e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:24.522 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.505+0000 7f27e8a6e700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:06:24.523 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.505+0000 7f27e8a6e700 1 -- 172.21.15.59:0/1035930162 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27d40094f0 con 0x7f27e4142290 2023-09-30T02:06:24.523 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.511+0000 7f079d6a8700 1 --2- 172.21.15.59:0/1929635689 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0784003a60 0x7f0784003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:24.523 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.512+0000 7f079d6a8700 10 monclient(hunting): get_auth_request con 0x7f0784003a60 auth_method 0 2023-09-30T02:06:24.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.512+0000 7f079d6a8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:24.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.512+0000 7f079d6a8700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:24.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.512+0000 7f079d6a8700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T02:06:24.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.512+0000 7fd0a37fe700 1 -- 172.21.15.59:0/725419722 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0ac005b30 con 0x7fd0bc06b810 2023-09-30T02:06:24.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.512+0000 7fd0a37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:24.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.512+0000 7fd0a37fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:06:24.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.512+0000 7fd0a37fe700 10 monclient: dump: 2023-09-30T02:06:24.525 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:24.525 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:24.526 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:24.526 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:24.526 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:24.526 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:24.526 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:24.526 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:24.526 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:24.527 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:24.527 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.512+0000 7fd0a37fe700 10 monclient: _finish_auth 0 2023-09-30T02:06:24.527 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.512+0000 7fd0a37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:24.527 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.512+0000 7fd0a37fe700 1 -- 172.21.15.59:0/725419722 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0ac005cb0 con 0x7fd0bc06b810 2023-09-30T02:06:24.527 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.512+0000 7fd0c1b86700 5 monclient: authenticate success, global_id 4378 2023-09-30T02:06:24.527 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.512+0000 7fd0c1b86700 10 monclient: _renew_subs 2023-09-30T02:06:24.528 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.512+0000 7fd0c1b86700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:24.528 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.512+0000 7fd0c1b86700 1 -- 172.21.15.59:0/725419722 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0bc14e300 con 0x7fd0bc06b810 2023-09-30T02:06:24.529 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.513+0000 7f079d6a8700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:24.529 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.513+0000 7f079d6a8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:24.529 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.513+0000 7f079d6a8700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T02:06:24.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.512+0000 7fd0a37fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:24.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.512+0000 7fd0a37fe700 1 -- 172.21.15.59:0/725419722 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0ac004a80 con 0x7fd0bc06b810 2023-09-30T02:06:24.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.512+0000 7fd0a37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:24.531 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.512+0000 7fd0c1b86700 10 client.4378.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:06:24.531 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.512+0000 7fd0a37fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:06:24.531 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.512+0000 7fd0a37fe700 10 monclient: dump: 2023-09-30T02:06:24.531 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:24.531 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:24.531 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:24.532 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:24.532 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:24.532 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:24.532 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:24.532 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:24.532 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:24.533 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:24.533 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.512+0000 7fd0b9ffb700 4 set_mon_vals no callback set 2023-09-30T02:06:24.533 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.512+0000 7fd0c1b86700 10 monclient: _renew_subs 2023-09-30T02:06:24.533 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.512+0000 7fd0c1b86700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:24.533 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.512+0000 7fd0c1b86700 1 -- 172.21.15.59:0/725419722 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0bc14e890 con 0x7fd0bc06b810 2023-09-30T02:06:24.533 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.512+0000 7fd0c1b86700 1 librados: init done 2023-09-30T02:06:24.534 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.513+0000 7fd0c1b86700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:24.534 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.513+0000 7fd0c1b86700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:24.534 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.513+0000 7fd0c1b86700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:24.534 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.513+0000 7fd0c1b86700 1 -- 172.21.15.59:0/725419722 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0bc072b30 con 0x7fd0bc06b810 2023-09-30T02:06:24.535 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.513+0000 7f27e8a6e700 1 --2- 172.21.15.59:0/1035930162 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f27e4142290 0x7f27e413e100 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f27dc00bc30 tx=0x7f27dc009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:24.535 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.513+0000 7f27cbfff700 10 client.?.objecter ms_handle_connect 0x7f27e4142290 2023-09-30T02:06:24.535 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.513+0000 7f27cbfff700 10 client.?.objecter resend_mon_ops 2023-09-30T02:06:24.535 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.523+0000 7f079d6a8700 10 monclient(hunting): handle_auth_done global_id 4375 payload 293 2023-09-30T02:06:24.536 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.523+0000 7f079d6a8700 1 -- 172.21.15.59:0/1929635689 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0798142290 msgr2=0x7f0784004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:06:24.536 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.523+0000 7f079d6a8700 1 --2- 172.21.15.59:0/1929635689 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0798142290 0x7f0784004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:24.536 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.523+0000 7f079d6a8700 1 -- 172.21.15.59:0/1929635689 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0784004e70 msgr2=0x7f07840052e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:06:24.536 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.523+0000 7f079d6a8700 1 --2- 172.21.15.59:0/1929635689 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0784004e70 0x7f07840052e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:24.536 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.523+0000 7f079d6a8700 10 monclient: _finish_hunting 0 2023-09-30T02:06:24.536 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.523+0000 7f079d6a8700 1 monclient: found mon.b 2023-09-30T02:06:24.536 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.523+0000 7f079d6a8700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:24.537 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.523+0000 7f079d6a8700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:24.537 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.523+0000 7f079d6a8700 1 -- 172.21.15.59:0/1929635689 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0784003570 con 0x7f0784003a60 2023-09-30T02:06:24.537 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.523+0000 7f079d6a8700 10 monclient: _finish_auth 0 2023-09-30T02:06:24.537 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.523+0000 7f079d6a8700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:24.539 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.536+0000 7fa8bb274700 10 monclient: get_monmap_and_config 2023-09-30T02:06:24.539 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.536+0000 7fa8bb274700 10 monclient: build_initial_monmap 2023-09-30T02:06:24.539 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.536+0000 7fa8bb274700 10 monclient: monmap: 2023-09-30T02:06:24.539 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:24.539 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:24.539 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:06:24.536522+0000 2023-09-30T02:06:24.540 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:06:24.536522+0000 2023-09-30T02:06:24.540 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:24.540 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:24.540 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:06:24.540 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:06:24.540 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:06:24.540 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:06:24.540 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:06:24.541 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:24.541 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.536+0000 7fa8bb274700 1 Processor -- start 2023-09-30T02:06:24.541 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.537+0000 7fa8bb274700 1 -- start start 2023-09-30T02:06:24.541 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.537+0000 7fa8bb274700 10 monclient: init 2023-09-30T02:06:24.541 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.537+0000 7fa8bb274700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:24.541 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.537+0000 7fa8bb274700 10 monclient: _add_conns ranks=[1,3,0,4,2] 2023-09-30T02:06:24.542 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.537+0000 7fa8bb274700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa8b40a5b70 addr v1:172.21.15.110:6789/0 2023-09-30T02:06:24.542 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.537+0000 7fa8bb274700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa8b4142290 0x7fa8b40b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:24.542 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.537+0000 7fa8bb274700 10 monclient(hunting): picked mon.noname-a con 0x7fa8b4142290 addr v2:172.21.15.59:3300/0 2023-09-30T02:06:24.542 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.537+0000 7fa8bb274700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa8b40a5f40 addr v1:172.21.15.59:6789/0 2023-09-30T02:06:24.542 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.537+0000 7fa8bb274700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8b40a75a0 con 0x7fa8b40a5f40 2023-09-30T02:06:24.543 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.537+0000 7fa8bb274700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8b40a6360 con 0x7fa8b40a5b70 2023-09-30T02:06:24.543 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.537+0000 7fa8bb274700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:24.543 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.537+0000 7fa8bb274700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8b40af3f0 con 0x7fa8b4142290 2023-09-30T02:06:24.543 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.537+0000 7fa8bb274700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:24.543 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.537+0000 7fa8bb274700 10 monclient(hunting): authenticate will time out at 1829.839988s 2023-09-30T02:06:24.545 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.541+0000 7fd0a37fe700 1 -- 172.21.15.59:0/725419722 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fd0ac0070f0 con 0x7fd0bc06b810 2023-09-30T02:06:24.545 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.541+0000 7fd0a37fe700 1 --2- 172.21.15.59:0/725419722 >> v2:172.21.15.110:6800/115870 conn(0x7fd0a8068890 0x7fd0a806ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:24.545 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.541+0000 7fd0a37fe700 1 -- 172.21.15.59:0/725419722 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fd0ac083760 con 0x7fd0bc06b810 2023-09-30T02:06:24.545 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.541+0000 7fd0a37fe700 10 client.4378.objecter ms_dispatch 0x7fd0bc001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:06:24.545 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.541+0000 7fd0a37fe700 3 client.4378.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:06:24.546 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.541+0000 7fd0a37fe700 3 client.4378.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:06:24.546 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.541+0000 7fd0a37fe700 20 client.4378.objecter dump_active .. 0 homeless 2023-09-30T02:06:24.546 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.542+0000 7fd0baffd700 1 --2- 172.21.15.59:0/725419722 >> v2:172.21.15.110:6800/115870 conn(0x7fd0a8068890 0x7fd0a806ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:24.546 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.542+0000 7fd0baffd700 10 monclient: get_auth_request con 0x7fd0a8068890 auth_method 0 2023-09-30T02:06:24.546 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.542+0000 7fd0a37fe700 10 client.4378.objecter ms_handle_connect 0x7fd0a8068890 2023-09-30T02:06:24.546 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.542+0000 7fd0baffd700 1 --2- 172.21.15.59:0/725419722 >> v2:172.21.15.110:6800/115870 conn(0x7fd0a8068890 0x7fd0a806ad40 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd0b0006fd0 tx=0x7fd0b0006db0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:24.553 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-09-30T02:06:24.664 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.665+0000 7f33c0ff9700 1 -- 172.21.15.59:0/2578571364 <== mon.2 v2:172.21.15.59:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f33b8053c30 con 0x7f33c40a5f40 2023-09-30T02:06:24.665 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.665+0000 7f33c0ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:24.665 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.665+0000 7f33c0ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:06:24.675 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.676+0000 7f35a2ffd700 1 -- 172.21.15.59:0/2524337962 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f35b00521f0 con 0x7f35bc0b2f70 2023-09-30T02:06:24.675 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.676+0000 7f35a2ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:24.675 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.676+0000 7f35a2ffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:06:24.691 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.691+0000 7fa8b880f700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa8b4142290 0x7fa8b40b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:24.695 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.696+0000 7fa8b880f700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa8b4142290 0x7fa8b40b1500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52120/0 (socket says 172.21.15.59:52120) 2023-09-30T02:06:24.695 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.696+0000 7fa8b880f700 1 -- 172.21.15.59:0/2591879088 learned_addr learned my addr 172.21.15.59:0/2591879088 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:06:24.695 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.696+0000 7fa8b880f700 10 monclient(hunting): get_auth_request con 0x7fa8b4142290 auth_method 0 2023-09-30T02:06:24.695 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.696+0000 7fa8b880f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:24.695 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.696+0000 7fa8b880f700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:24.696 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.696+0000 7fa8b880f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:24.701 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.702+0000 7f1d54ac5700 10 monclient: get_monmap_and_config 2023-09-30T02:06:24.701 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.702+0000 7f1d54ac5700 10 monclient: build_initial_monmap 2023-09-30T02:06:24.701 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.702+0000 7f1d54ac5700 10 monclient: monmap: 2023-09-30T02:06:24.701 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:24.701 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:24.701 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:06:24.702528+0000 2023-09-30T02:06:24.702 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:06:24.702528+0000 2023-09-30T02:06:24.702 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:24.702 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:24.702 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:06:24.702 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:06:24.702 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:06:24.702 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:06:24.702 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:06:24.702 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:24.703 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.703+0000 7f1d54ac5700 1 Processor -- start 2023-09-30T02:06:24.703 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.703+0000 7f1d54ac5700 1 -- start start 2023-09-30T02:06:24.703 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.703+0000 7f1d54ac5700 10 monclient: init 2023-09-30T02:06:24.703 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.704+0000 7f1d54ac5700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:24.703 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.704+0000 7f1d54ac5700 10 monclient: _add_conns ranks=[1,0,3,2,4] 2023-09-30T02:06:24.704 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.704+0000 7f1d54ac5700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1d500a5b70 addr v1:172.21.15.110:6789/0 2023-09-30T02:06:24.704 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.704+0000 7f1d54ac5700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1d50142290 addr v1:172.21.15.59:6789/0 2023-09-30T02:06:24.704 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.704+0000 7f1d54ac5700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f1d500b1500 0x7f1d500a5f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:24.704 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.704+0000 7f1d54ac5700 10 monclient(hunting): picked mon.noname-a con 0x7f1d500b1500 addr v2:172.21.15.59:3300/0 2023-09-30T02:06:24.704 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.704+0000 7f1d54ac5700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d500b1920 con 0x7f1d50142290 2023-09-30T02:06:24.704 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.704+0000 7f1d54ac5700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d500a75a0 con 0x7f1d500a5b70 2023-09-30T02:06:24.704 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.704+0000 7f1d54ac5700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:24.704 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.704+0000 7f1d54ac5700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d500af3f0 con 0x7f1d500b1500 2023-09-30T02:06:24.704 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.704+0000 7f1d54ac5700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:24.705 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.704+0000 7f1d54ac5700 10 monclient(hunting): authenticate will time out at 1830.006531s 2023-09-30T02:06:24.706 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.706+0000 7fa8b880f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:24.706 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.706+0000 7fa8b880f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:24.706 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.706+0000 7fa8b880f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:24.706 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.707+0000 7fc52084b700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2023-09-30T02:06:24.706 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.707+0000 7fc52084b700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2023-09-30T02:06:24.706 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.707+0000 7fc52084b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:24.707 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.707+0000 7fc52084b700 1 -- 172.21.15.59:0/3488309627 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fc51c13e130 con 0x7fc51c0a5f40 2023-09-30T02:06:24.708 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.709+0000 7fa8b880f700 10 monclient(hunting): handle_auth_done global_id 4335 payload 274 2023-09-30T02:06:24.708 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.709+0000 7fa8b880f700 1 -- 172.21.15.59:0/2591879088 >> v1:172.21.15.110:6789/0 conn(0x7fa8b40a5b70 legacy=0x7fa8b4141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:06:24.708 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.709+0000 7fa8b880f700 1 -- 172.21.15.59:0/2591879088 >> v1:172.21.15.59:6789/0 conn(0x7fa8b40a5f40 legacy=0x7fa8b413af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:06:24.708 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.709+0000 7fa8b880f700 10 monclient: _finish_hunting 0 2023-09-30T02:06:24.708 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.709+0000 7fa8b880f700 1 monclient: found mon.noname-a 2023-09-30T02:06:24.709 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.709+0000 7fa8b880f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:24.709 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.709+0000 7fa8b880f700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.59:3300/0 2023-09-30T02:06:24.709 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.709+0000 7fa8b880f700 1 -- 172.21.15.59:0/2591879088 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8b413b640 con 0x7fa8b4142290 2023-09-30T02:06:24.713 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.714+0000 7f1d47fff700 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f1d50142290 0x7f1d5013af30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:35114/0 (socket says 172.21.15.59:35114) 2023-09-30T02:06:24.714 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.714+0000 7f1d47fff700 1 -- 172.21.15.59:0/1423282691 learned_addr learned my addr 172.21.15.59:0/1423282691 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:06:24.714 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.715+0000 7fa8b880f700 1 --2- 172.21.15.59:0/2591879088 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8b4142290 0x7fa8b40b1500 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa8a40099b0 tx=0x7fa8a4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1b5b8799bf0e738 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:24.714 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.715+0000 7f1d4ed9d700 1 --2- 172.21.15.59:0/1423282691 >> v2:172.21.15.59:3300/0 conn(0x7f1d500b1500 0x7f1d500a5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:24.715 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.716+0000 7f1d4ed9d700 10 monclient(hunting): get_auth_request con 0x7f1d500b1500 auth_method 0 2023-09-30T02:06:24.715 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.716+0000 7f1d4ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:24.715 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.716+0000 7f1d4ed9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:24.715 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.716+0000 7f1d4ed9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:24.716 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.717+0000 7f1d4ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:24.716 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.717+0000 7f1d4ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:24.716 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.717+0000 7f1d4ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:24.719 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.719+0000 7f1d4ed9d700 10 monclient(hunting): handle_auth_done global_id 4338 payload 274 2023-09-30T02:06:24.719 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.719+0000 7f1d4ed9d700 1 -- 172.21.15.59:0/1423282691 >> v1:172.21.15.110:6789/0 conn(0x7f1d500a5b70 legacy=0x7f1d50141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:06:24.719 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.719+0000 7f1d4ed9d700 1 -- 172.21.15.59:0/1423282691 >> v1:172.21.15.59:6789/0 conn(0x7f1d50142290 legacy=0x7f1d5013af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:24.719 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.719+0000 7f1d4ed9d700 10 monclient: _finish_hunting 0 2023-09-30T02:06:24.719 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.719+0000 7f1d4ed9d700 1 monclient: found mon.noname-a 2023-09-30T02:06:24.719 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.719+0000 7f1d4ed9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:24.719 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.719+0000 7f1d4ed9d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.59:3300/0 2023-09-30T02:06:24.719 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.719+0000 7f1d4ed9d700 1 -- 172.21.15.59:0/1423282691 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d5013b640 con 0x7f1d500b1500 2023-09-30T02:06:24.726 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.727+0000 7fa8abfff700 1 -- 172.21.15.59:0/2591879088 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8a400b7e0 con 0x7fa8b4142290 2023-09-30T02:06:24.727 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.727+0000 7fa8abfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:24.727 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.727+0000 7fa8abfff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T02:06:24.727 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.727+0000 7fa8abfff700 10 monclient: dump: 2023-09-30T02:06:24.727 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:24.727 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:24.727 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:24.727 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:24.727 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:24.727 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:24.727 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:24.727 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:24.728 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:24.728 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:24.728 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.727+0000 7fa8abfff700 10 monclient: _finish_auth 0 2023-09-30T02:06:24.728 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.727+0000 7fa8abfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:24.728 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.727+0000 7fa8abfff700 1 -- 172.21.15.59:0/2591879088 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8a400b940 con 0x7fa8b4142290 2023-09-30T02:06:24.728 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.727+0000 7fa8bb274700 5 monclient: authenticate success, global_id 4335 2023-09-30T02:06:24.728 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.727+0000 7fa8bb274700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:06:24.728 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.727+0000 7fa8abfff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:24.728 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.727+0000 7fa8bb274700 10 monclient: get_monmap_and_config success 2023-09-30T02:06:24.728 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.727+0000 7fa8abfff700 1 -- 172.21.15.59:0/2591879088 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8a400bce0 con 0x7fa8b4142290 2023-09-30T02:06:24.728 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.727+0000 7fa8abfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:24.729 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.727+0000 7fa8bb274700 4 set_mon_vals no callback set 2023-09-30T02:06:24.729 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.727+0000 7fa8abfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T02:06:24.729 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.727+0000 7fa8abfff700 10 monclient: dump: 2023-09-30T02:06:24.729 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:24.729 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:24.729 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:24.729 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:24.729 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:24.729 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:24.729 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:24.729 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:24.729 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:24.730 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:24.730 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.727+0000 7f1d4ed9d700 1 --2- 172.21.15.59:0/1423282691 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d500b1500 0x7f1d500a5f40 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f1d48009e60 tx=0x7f1d48009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f54b69107e939e04 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:24.732 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.728+0000 7fa8bb274700 10 monclient: shutdown 2023-09-30T02:06:24.732 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.728+0000 7fa8bb274700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:24.732 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.728+0000 7fa8bb274700 1 -- 172.21.15.59:0/2591879088 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8b4142290 msgr2=0x7fa8b40b1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:24.732 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.728+0000 7fa8bb274700 1 --2- 172.21.15.59:0/2591879088 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8b4142290 0x7fa8b40b1500 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa8a40099b0 tx=0x7fa8a4009250 comp rx=0 tx=0).stop 2023-09-30T02:06:24.732 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.728+0000 7fa8bb274700 1 -- 172.21.15.59:0/2591879088 shutdown_connections 2023-09-30T02:06:24.732 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.728+0000 7fa8bb274700 1 --2- 172.21.15.59:0/2591879088 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8b4142290 0x7fa8b40b1500 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:24.733 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.728+0000 7fa8bb274700 1 -- 172.21.15.59:0/2591879088 >> 172.21.15.59:0/2591879088 conn(0x7fa8b40631f0 msgr2=0x7fa8b40635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:24.733 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.728+0000 7fa8bb274700 1 -- 172.21.15.59:0/2591879088 shutdown_connections 2023-09-30T02:06:24.733 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.728+0000 7fa8bb274700 1 -- 172.21.15.59:0/2591879088 wait complete. 2023-09-30T02:06:24.733 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.729+0000 7fa8bb274700 10 monclient: build_initial_monmap 2023-09-30T02:06:24.733 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.729+0000 7fa8bb274700 10 monclient: monmap: 2023-09-30T02:06:24.733 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:24.733 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:06:24.733 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:06:24.733 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:06:24.733 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:24.733 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:06:24.733 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:06:24.734 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:06:24.734 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:06:24.734 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:24.734 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.729+0000 7fa8bb274700 1 librados: starting msgr at 2023-09-30T02:06:24.734 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.729+0000 7fa8bb274700 1 librados: starting objecter 2023-09-30T02:06:24.734 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.729+0000 7fa8bb274700 1 Processor -- start 2023-09-30T02:06:24.734 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.729+0000 7fa8bb274700 1 -- start start 2023-09-30T02:06:24.734 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.729+0000 7fa8bb274700 1 librados: setting wanted keys 2023-09-30T02:06:24.734 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.729+0000 7fa8bb274700 1 librados: calling monclient init 2023-09-30T02:06:24.734 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.729+0000 7fa8bb274700 10 monclient: init 2023-09-30T02:06:24.734 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.730+0000 7fa8bb274700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:24.734 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.730+0000 7fa8bb274700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T02:06:24.734 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.730+0000 7fa8bb274700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8b40a5b70 0x7fa8b4140280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:24.735 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.730+0000 7fa8bb274700 10 monclient(hunting): picked mon.noname-a con 0x7fa8b40a5b70 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:06:24.735 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.730+0000 7fa8bb274700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa8b40a5f40 0x7fa8b41407c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:24.735 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.730+0000 7fa8bb274700 10 monclient(hunting): picked mon.noname-c con 0x7fa8b40a5f40 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:24.735 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.730+0000 7fa8bb274700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa8b4142290 0x7fa8b4151340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:24.735 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.730+0000 7fa8bb274700 10 monclient(hunting): picked mon.noname-b con 0x7fa8b4142290 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:06:24.735 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.730+0000 7fa8bb274700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:24.735 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.730+0000 7fa8bb274700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b413fdf0 con 0x7fa8b40a5b70 2023-09-30T02:06:24.735 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.730+0000 7fa8bb274700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:24.735 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.730+0000 7fa8bb274700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b413efd0 con 0x7fa8b4142290 2023-09-30T02:06:24.735 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.730+0000 7fa8bb274700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:24.735 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.730+0000 7fa8bb274700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b4140d00 con 0x7fa8b40a5f40 2023-09-30T02:06:24.735 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.730+0000 7fa8bb274700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:24.736 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.730+0000 7fa8bb274700 10 monclient(hunting): authenticate will time out at 1830.032688s 2023-09-30T02:06:24.736 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.731+0000 7fa8b880f700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa8b40a5f40 0x7fa8b41407c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:24.736 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.732+0000 7fa8b880f700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa8b40a5f40 0x7fa8b41407c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:47816/0 (socket says 172.21.15.59:47816) 2023-09-30T02:06:24.736 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.732+0000 7fa8b880f700 1 -- 172.21.15.59:0/144204252 learned_addr learned my addr 172.21.15.59:0/144204252 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:06:24.736 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.732+0000 7fa8b880f700 10 monclient(hunting): get_auth_request con 0x7fa8b40a5f40 auth_method 0 2023-09-30T02:06:24.736 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.732+0000 7fa8b880f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:24.736 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.732+0000 7fa8b880f700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:24.736 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.732+0000 7fa8b880f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:24.736 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.733+0000 7fa8b880f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:24.736 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.733+0000 7fa8b880f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:24.736 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.733+0000 7fa8b880f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:24.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.737+0000 7fa8b880f700 10 monclient(hunting): handle_auth_done global_id 4328 payload 824 2023-09-30T02:06:24.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.737+0000 7fa8b880f700 1 -- 172.21.15.59:0/144204252 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa8b4142290 msgr2=0x7fa8b4151340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:24.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.737+0000 7fa8b880f700 1 --2- 172.21.15.59:0/144204252 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa8b4142290 0x7fa8b4151340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:24.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.737+0000 7fa8b880f700 1 -- 172.21.15.59:0/144204252 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8b40a5b70 msgr2=0x7fa8b4140280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:24.738 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.737+0000 7fa8b880f700 1 --2- 172.21.15.59:0/144204252 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8b40a5b70 0x7fa8b4140280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:24.738 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.737+0000 7fa8b880f700 10 monclient: _finish_hunting 0 2023-09-30T02:06:24.738 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.737+0000 7fa8b880f700 1 monclient: found mon.noname-c 2023-09-30T02:06:24.738 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.737+0000 7fa8b880f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:24.738 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.737+0000 7fa8b880f700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.59:3301/0 2023-09-30T02:06:24.738 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.737+0000 7fa8b880f700 1 -- 172.21.15.59:0/144204252 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8a40094f0 con 0x7fa8b40a5f40 2023-09-30T02:06:24.739 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.740+0000 7fa8b880f700 1 --2- 172.21.15.59:0/144204252 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa8b40a5f40 0x7fa8b41407c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa8b40b1ff0 tx=0x7fa8a4016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:24.739 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.740+0000 7fa8a97fa700 10 client.?.objecter ms_handle_connect 0x7fa8b40a5f40 2023-09-30T02:06:24.739 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.740+0000 7fa8a97fa700 10 client.?.objecter resend_mon_ops 2023-09-30T02:06:24.742 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.743+0000 7fa8a97fa700 1 -- 172.21.15.59:0/144204252 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8a401b680 con 0x7fa8b40a5f40 2023-09-30T02:06:24.743 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.743+0000 7fa8a97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:24.743 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.743+0000 7fa8a97fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:06:24.743 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.743+0000 7fa8a97fa700 10 monclient: dump: 2023-09-30T02:06:24.743 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:24.743 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:24.743 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:24.743 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:24.743 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:24.743 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:24.743 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:24.743 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:24.744 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:24.744 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:24.744 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.743+0000 7fa8a97fa700 10 monclient: _finish_auth 0 2023-09-30T02:06:24.744 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.743+0000 7fa8a97fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:24.744 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.743+0000 7fa8bb274700 5 monclient: authenticate success, global_id 4328 2023-09-30T02:06:24.744 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.743+0000 7fa8bb274700 10 monclient: _renew_subs 2023-09-30T02:06:24.744 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.743+0000 7fa8bb274700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:06:24.744 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.743+0000 7fa8bb274700 1 -- 172.21.15.59:0/144204252 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8b4140f20 con 0x7fa8b40a5f40 2023-09-30T02:06:24.744 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.743+0000 7fa8bb274700 10 client.4328.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:06:24.744 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.743+0000 7fa8bb274700 10 monclient: _renew_subs 2023-09-30T02:06:24.744 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.743+0000 7fa8bb274700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:06:24.744 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.743+0000 7fa8bb274700 1 -- 172.21.15.59:0/144204252 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8b4151ac0 con 0x7fa8b40a5f40 2023-09-30T02:06:24.744 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.743+0000 7fa8bb274700 1 librados: init done 2023-09-30T02:06:24.745 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.744+0000 7fa8bb274700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:24.745 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.744+0000 7fa8bb274700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:24.745 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.744+0000 7fa8bb274700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:06:24.745 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.744+0000 7fa8bb274700 1 -- 172.21.15.59:0/144204252 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8b41400d0 con 0x7fa8b40a5f40 2023-09-30T02:06:24.746 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.744+0000 7f1d4dd9b700 1 -- 172.21.15.59:0/1423282691 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d4800e070 con 0x7f1d500b1500 2023-09-30T02:06:24.746 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.744+0000 7f1d4dd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:24.746 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.744+0000 7f1d4dd9b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T02:06:24.746 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.744+0000 7f1d4dd9b700 10 monclient: dump: 2023-09-30T02:06:24.746 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:24.746 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:24.746 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:24.746 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:24.746 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:24.746 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:24.746 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:24.747 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:24.747 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:24.747 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:24.747 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.744+0000 7f1d4dd9b700 10 monclient: _finish_auth 0 2023-09-30T02:06:24.747 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.744+0000 7f1d4dd9b700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:24.747 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.744+0000 7f1d4dd9b700 1 -- 172.21.15.59:0/1423282691 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d4800bbf0 con 0x7f1d500b1500 2023-09-30T02:06:24.747 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.744+0000 7f1d4dd9b700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:24.747 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.744+0000 7f1d4dd9b700 1 -- 172.21.15.59:0/1423282691 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d48012430 con 0x7f1d500b1500 2023-09-30T02:06:24.747 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.744+0000 7f1d4dd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:24.747 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.744+0000 7f1d4dd9b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T02:06:24.747 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.744+0000 7f1d4dd9b700 10 monclient: dump: 2023-09-30T02:06:24.747 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:24.747 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:24.748 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:24.748 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:24.748 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:24.748 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:24.748 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:24.748 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:24.748 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:24.748 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:24.748 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.744+0000 7f1d54ac5700 5 monclient: authenticate success, global_id 4338 2023-09-30T02:06:24.748 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.744+0000 7f1d54ac5700 10 monclient: get_monmap_and_config success 2023-09-30T02:06:24.748 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.744+0000 7f1d54ac5700 4 set_mon_vals no callback set 2023-09-30T02:06:24.749 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.744+0000 7fa8a97fa700 1 -- 172.21.15.59:0/144204252 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8a4006d70 con 0x7fa8b40a5f40 2023-09-30T02:06:24.749 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.744+0000 7fa8a97fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:24.749 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.744+0000 7fa8ab7fe700 4 set_mon_vals no callback set 2023-09-30T02:06:24.749 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.745+0000 7fa8a97fa700 1 -- 172.21.15.59:0/144204252 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8a401b9f0 con 0x7fa8b40a5f40 2023-09-30T02:06:24.749 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.745+0000 7fa8a97fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:24.749 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.745+0000 7fa8a97fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:06:24.749 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.745+0000 7fa8a97fa700 10 monclient: dump: 2023-09-30T02:06:24.749 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:24.750 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:24.750 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:24.750 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:24.750 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:24.750 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:24.750 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:24.750 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:24.750 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:24.750 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:24.752 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.745+0000 7f1d54ac5700 10 monclient: shutdown 2023-09-30T02:06:24.752 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.745+0000 7f1d54ac5700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:24.752 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.745+0000 7f1d54ac5700 1 -- 172.21.15.59:0/1423282691 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d500b1500 msgr2=0x7f1d500a5f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:24.752 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.745+0000 7f1d54ac5700 1 --2- 172.21.15.59:0/1423282691 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d500b1500 0x7f1d500a5f40 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f1d48009e60 tx=0x7f1d48009780 comp rx=0 tx=0).stop 2023-09-30T02:06:24.752 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.746+0000 7f1d54ac5700 1 -- 172.21.15.59:0/1423282691 shutdown_connections 2023-09-30T02:06:24.752 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.746+0000 7f1d54ac5700 1 --2- 172.21.15.59:0/1423282691 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d500b1500 0x7f1d500a5f40 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:24.752 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.746+0000 7f1d54ac5700 1 -- 172.21.15.59:0/1423282691 >> 172.21.15.59:0/1423282691 conn(0x7f1d500631f0 msgr2=0x7f1d500635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:24.752 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.746+0000 7f1d54ac5700 1 -- 172.21.15.59:0/1423282691 shutdown_connections 2023-09-30T02:06:24.753 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.746+0000 7f1d54ac5700 1 -- 172.21.15.59:0/1423282691 wait complete. 2023-09-30T02:06:24.753 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.747+0000 7f1d54ac5700 10 monclient: build_initial_monmap 2023-09-30T02:06:24.753 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.747+0000 7f1d54ac5700 10 monclient: monmap: 2023-09-30T02:06:24.753 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:24.753 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:06:24.753 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:06:24.753 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:06:24.753 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:24.753 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:06:24.753 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:06:24.754 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:06:24.754 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:06:24.754 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:24.754 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.747+0000 7f1d54ac5700 1 librados: starting msgr at 2023-09-30T02:06:24.754 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.747+0000 7f1d54ac5700 1 librados: starting objecter 2023-09-30T02:06:24.754 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.747+0000 7f1d54ac5700 1 Processor -- start 2023-09-30T02:06:24.754 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.748+0000 7f1d54ac5700 1 -- start start 2023-09-30T02:06:24.754 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.748+0000 7f1d54ac5700 1 librados: setting wanted keys 2023-09-30T02:06:24.754 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.748+0000 7f1d54ac5700 1 librados: calling monclient init 2023-09-30T02:06:24.754 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.748+0000 7f1d54ac5700 10 monclient: init 2023-09-30T02:06:24.754 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.748+0000 7f1d54ac5700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:24.755 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.748+0000 7f1d54ac5700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T02:06:24.755 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.748+0000 7f1d54ac5700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1d500a5b70 0x7f1d50140c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:24.755 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.748+0000 7f1d54ac5700 10 monclient(hunting): picked mon.noname-b con 0x7f1d500a5b70 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:06:24.755 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.748+0000 7f1d54ac5700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d500b1500 0x7f1d5013ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:24.755 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.748+0000 7f1d54ac5700 10 monclient(hunting): picked mon.noname-a con 0x7f1d500b1500 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:06:24.755 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.748+0000 7f1d54ac5700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1d50142290 0x7f1d5014e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:24.755 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.748+0000 7f1d54ac5700 10 monclient(hunting): picked mon.noname-c con 0x7f1d50142290 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:24.755 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.748+0000 7f1d54ac5700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:24.755 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.748+0000 7f1d54ac5700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d5013e850 con 0x7f1d500b1500 2023-09-30T02:06:24.755 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.749+0000 7f1d54ac5700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:24.756 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.749+0000 7f1d54ac5700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d500ae720 con 0x7f1d500a5b70 2023-09-30T02:06:24.756 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.749+0000 7f1d54ac5700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:24.756 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.749+0000 7f1d54ac5700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d5014e6c0 con 0x7f1d50142290 2023-09-30T02:06:24.756 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.749+0000 7f1d54ac5700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:24.756 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.749+0000 7f1d54ac5700 10 monclient(hunting): authenticate will time out at 1830.051020s 2023-09-30T02:06:24.756 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.749+0000 7f1d47fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d500b1500 0x7f1d5013ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:24.756 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.750+0000 7f1d47fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d500b1500 0x7f1d5013ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52142/0 (socket says 172.21.15.59:52142) 2023-09-30T02:06:24.756 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.750+0000 7f1d47fff700 1 -- 172.21.15.59:0/3857864768 learned_addr learned my addr 172.21.15.59:0/3857864768 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:06:24.756 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.750+0000 7f1d47fff700 10 monclient(hunting): get_auth_request con 0x7f1d500b1500 auth_method 0 2023-09-30T02:06:24.756 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.750+0000 7f1d47fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:24.756 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.750+0000 7f1d47fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:24.757 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.750+0000 7f1d47fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:24.757 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.751+0000 7f1d47fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:24.757 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.751+0000 7f1d47fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:24.757 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.751+0000 7f1d47fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:24.758 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.756+0000 7f1d47fff700 10 monclient(hunting): handle_auth_done global_id 4341 payload 824 2023-09-30T02:06:24.758 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.756+0000 7f1d47fff700 1 -- 172.21.15.59:0/3857864768 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1d50142290 msgr2=0x7f1d5014e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:24.758 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.756+0000 7f1d47fff700 1 --2- 172.21.15.59:0/3857864768 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1d50142290 0x7f1d5014e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:24.758 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.756+0000 7f1d47fff700 1 -- 172.21.15.59:0/3857864768 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1d500a5b70 msgr2=0x7f1d50140c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:24.758 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.756+0000 7f1d47fff700 1 --2- 172.21.15.59:0/3857864768 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1d500a5b70 0x7f1d50140c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:24.758 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.756+0000 7f1d47fff700 10 monclient: _finish_hunting 0 2023-09-30T02:06:24.758 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.756+0000 7f1d47fff700 1 monclient: found mon.noname-a 2023-09-30T02:06:24.758 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.757+0000 7f1d47fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:24.758 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.757+0000 7f1d47fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.59:3300/0 2023-09-30T02:06:24.758 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.757+0000 7f1d47fff700 1 -- 172.21.15.59:0/3857864768 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d48009a20 con 0x7f1d500b1500 2023-09-30T02:06:24.759 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.759+0000 7f1d47fff700 1 --2- 172.21.15.59:0/3857864768 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d500b1500 0x7f1d5013ece0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1d3800c560 tx=0x7f1d38001020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:24.759 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.759+0000 7f1d467fc700 10 client.?.objecter ms_handle_connect 0x7f1d500b1500 2023-09-30T02:06:24.759 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.759+0000 7f1d467fc700 10 client.?.objecter resend_mon_ops 2023-09-30T02:06:24.764 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.765+0000 7f079d6a8700 1 --2- 172.21.15.59:0/1929635689 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0784003a60 0x7f0784003e50 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f079400c3e0 tx=0x7f0794001e20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c30687e73771d64f server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:24.765 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.766+0000 7f1d467fc700 1 -- 172.21.15.59:0/3857864768 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d38005b30 con 0x7f1d500b1500 2023-09-30T02:06:24.765 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.766+0000 7f1d467fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:24.765 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.766+0000 7f1d467fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T02:06:24.766 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.766+0000 7f1d467fc700 10 monclient: dump: 2023-09-30T02:06:24.766 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:24.766 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:24.766 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:24.766 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:24.766 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:24.766 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:24.766 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:24.766 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:24.766 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:24.766 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:24.767 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.766+0000 7f1d467fc700 10 monclient: _finish_auth 0 2023-09-30T02:06:24.767 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.766+0000 7f1d467fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:24.767 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.766+0000 7f1d467fc700 1 -- 172.21.15.59:0/3857864768 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d38005cb0 con 0x7f1d500b1500 2023-09-30T02:06:24.767 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.766+0000 7f1d467fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:24.767 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.766+0000 7f1d467fc700 1 -- 172.21.15.59:0/3857864768 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d38007320 con 0x7f1d500b1500 2023-09-30T02:06:24.767 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.766+0000 7f1d467fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:24.767 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.766+0000 7f1d467fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T02:06:24.767 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.766+0000 7f1d467fc700 10 monclient: dump: 2023-09-30T02:06:24.767 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:24.767 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:24.767 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:24.767 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:24.768 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:24.768 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:24.768 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:24.768 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:24.768 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:24.768 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:24.768 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.766+0000 7f1d54ac5700 5 monclient: authenticate success, global_id 4341 2023-09-30T02:06:24.768 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.766+0000 7f1d4d59a700 4 set_mon_vals no callback set 2023-09-30T02:06:24.768 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.766+0000 7f1d54ac5700 10 monclient: _renew_subs 2023-09-30T02:06:24.768 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.766+0000 7f1d54ac5700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:06:24.768 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.766+0000 7f1d54ac5700 1 -- 172.21.15.59:0/3857864768 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d5013dd60 con 0x7f1d500b1500 2023-09-30T02:06:24.769 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.766+0000 7f1d54ac5700 10 client.4341.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:06:24.769 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.766+0000 7f1d54ac5700 10 monclient: _renew_subs 2023-09-30T02:06:24.769 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.766+0000 7f1d54ac5700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:06:24.769 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.766+0000 7f1d54ac5700 1 -- 172.21.15.59:0/3857864768 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d5013e200 con 0x7f1d500b1500 2023-09-30T02:06:24.769 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.766+0000 7f1d54ac5700 1 librados: init done 2023-09-30T02:06:24.769 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.767+0000 7f1d54ac5700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:24.770 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.767+0000 7f1d54ac5700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:24.770 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.767+0000 7f1d54ac5700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:06:24.770 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.767+0000 7f1d54ac5700 1 -- 172.21.15.59:0/3857864768 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d5014e880 con 0x7f1d500b1500 2023-09-30T02:06:24.771 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.769+0000 7f27cbfff700 1 -- 172.21.15.59:0/1035930162 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27dc0092e0 con 0x7f27e4142290 2023-09-30T02:06:24.771 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.769+0000 7f27cbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:24.771 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.769+0000 7f27cbfff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:06:24.771 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.769+0000 7f27cbfff700 10 monclient: dump: 2023-09-30T02:06:24.771 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:24.771 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:24.771 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:24.771 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:24.771 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:24.771 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:24.772 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:24.772 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:24.772 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:24.772 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:24.772 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.769+0000 7f27cbfff700 10 monclient: _finish_auth 0 2023-09-30T02:06:24.772 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.769+0000 7f27cbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:24.772 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.769+0000 7f27cbfff700 1 -- 172.21.15.59:0/1035930162 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27dc00c7a0 con 0x7f27e4142290 2023-09-30T02:06:24.772 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.769+0000 7f27ea4d1700 5 monclient: authenticate success, global_id 4381 2023-09-30T02:06:24.772 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.769+0000 7f27ea4d1700 10 monclient: _renew_subs 2023-09-30T02:06:24.772 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.769+0000 7f27ea4d1700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:24.773 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.769+0000 7f27ea4d1700 1 -- 172.21.15.59:0/1035930162 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27e4154eb0 con 0x7f27e4142290 2023-09-30T02:06:24.773 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.769+0000 7f27ea4d1700 10 client.4381.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:06:24.773 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.769+0000 7f27ea4d1700 10 monclient: _renew_subs 2023-09-30T02:06:24.773 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.769+0000 7f27ea4d1700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:24.773 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.769+0000 7f27ea4d1700 1 -- 172.21.15.59:0/1035930162 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27e413e880 con 0x7f27e4142290 2023-09-30T02:06:24.773 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.769+0000 7f27ea4d1700 1 librados: init done 2023-09-30T02:06:24.773 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.769+0000 7f27cbfff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:24.773 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.769+0000 7f27cbfff700 1 -- 172.21.15.59:0/1035930162 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27dc00cb80 con 0x7f27e4142290 2023-09-30T02:06:24.773 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.769+0000 7f27cbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:24.773 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.769+0000 7f27cbfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:06:24.773 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.769+0000 7f27e27fc700 4 set_mon_vals no callback set 2023-09-30T02:06:24.774 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.769+0000 7f27cbfff700 10 monclient: dump: 2023-09-30T02:06:24.774 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:24.774 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:24.774 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:24.774 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:24.774 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:24.774 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:24.774 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:24.774 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:24.774 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:24.774 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:24.774 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.770+0000 7f27ea4d1700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:24.775 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.770+0000 7f27ea4d1700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:24.775 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.770+0000 7f27ea4d1700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:24.775 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.770+0000 7f27ea4d1700 1 -- 172.21.15.59:0/1035930162 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27c40050b0 con 0x7f27e4142290 2023-09-30T02:06:24.779 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.779+0000 7fd0a37fe700 1 -- 172.21.15.59:0/725419722 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd0ac051080 con 0x7fd0bc06b810 2023-09-30T02:06:24.779 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.780+0000 7fd0a37fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:24.779 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.780+0000 7fd0a37fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:06:24.793 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.794+0000 7f078ffff700 1 -- 172.21.15.59:0/1929635689 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f079400ccb0 con 0x7f0784003a60 2023-09-30T02:06:24.793 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.794+0000 7f078ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:24.794 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.794+0000 7f078ffff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:06:24.794 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.794+0000 7f078ffff700 10 monclient: dump: 2023-09-30T02:06:24.794 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:24.794 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:24.794 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:24.794 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:24.794 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:24.794 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:24.794 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:24.794 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:24.794 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:24.794 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:24.795 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.794+0000 7f078ffff700 1 -- 172.21.15.59:0/1929635689 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f079400ce10 con 0x7f0784003a60 2023-09-30T02:06:24.795 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.794+0000 7f078ffff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:24.795 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.794+0000 7f078ffff700 1 -- 172.21.15.59:0/1929635689 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0794005b70 con 0x7f0784003a60 2023-09-30T02:06:24.795 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.794+0000 7f078ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:24.795 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.794+0000 7f078ffff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:06:24.795 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.794+0000 7f078ffff700 10 monclient: dump: 2023-09-30T02:06:24.795 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:24.795 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:24.795 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:24.795 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:24.795 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:24.795 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:24.796 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:24.796 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:24.796 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:24.796 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:24.796 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.795+0000 7f079f90c700 10 monclient: get_monmap_and_config success 2023-09-30T02:06:24.796 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.795+0000 7f079f90c700 4 set_mon_vals no callback set 2023-09-30T02:06:24.797 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.795+0000 7f079f90c700 10 monclient: shutdown 2023-09-30T02:06:24.797 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.795+0000 7f079f90c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:24.797 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.795+0000 7f079f90c700 1 -- 172.21.15.59:0/1929635689 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0784003a60 msgr2=0x7f0784003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:24.797 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.795+0000 7f079f90c700 1 --2- 172.21.15.59:0/1929635689 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0784003a60 0x7f0784003e50 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f079400c3e0 tx=0x7f0794001e20 comp rx=0 tx=0).stop 2023-09-30T02:06:24.797 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.796+0000 7f079dea9700 1 -- 172.21.15.59:0/1929635689 reap_dead start 2023-09-30T02:06:24.797 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.796+0000 7f079f90c700 1 -- 172.21.15.59:0/1929635689 shutdown_connections 2023-09-30T02:06:24.797 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.796+0000 7f079f90c700 1 -- 172.21.15.59:0/1929635689 >> 172.21.15.59:0/1929635689 conn(0x7f07980631f0 msgr2=0x7f07980635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:24.806 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.805+0000 7f079f90c700 1 -- 172.21.15.59:0/1929635689 shutdown_connections 2023-09-30T02:06:24.808 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.807+0000 7f079f90c700 1 -- 172.21.15.59:0/1929635689 wait complete. 2023-09-30T02:06:24.808 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.807+0000 7f079f90c700 10 monclient: build_initial_monmap 2023-09-30T02:06:24.808 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.808+0000 7f079f90c700 10 monclient: monmap: 2023-09-30T02:06:24.808 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:24.808 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:06:24.808 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:06:24.808 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:06:24.808 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:24.808 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:06:24.808 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:06:24.808 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:06:24.809 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:06:24.809 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:24.809 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.808+0000 7f079f90c700 1 librados: starting msgr at 2023-09-30T02:06:24.809 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.808+0000 7f079f90c700 1 librados: starting objecter 2023-09-30T02:06:24.809 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.808+0000 7f079f90c700 1 Processor -- start 2023-09-30T02:06:24.810 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.808+0000 7f079f90c700 1 -- start start 2023-09-30T02:06:24.810 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.808+0000 7f079f90c700 1 librados: setting wanted keys 2023-09-30T02:06:24.810 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.808+0000 7f079f90c700 1 librados: calling monclient init 2023-09-30T02:06:24.810 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.808+0000 7f079f90c700 10 monclient: init 2023-09-30T02:06:24.810 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.809+0000 7f079f90c700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:24.810 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.809+0000 7f079f90c700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T02:06:24.810 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.809+0000 7f079f90c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0784003a60 0x7f079806af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:24.810 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.809+0000 7f079f90c700 10 monclient(hunting): picked mon.noname-b con 0x7f0784003a60 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:06:24.810 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.809+0000 7f079f90c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f079806b690 0x7f079806d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:24.810 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.809+0000 7f079f90c700 10 monclient(hunting): picked mon.noname-a con 0x7f079806b690 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:06:24.810 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.809+0000 7f079f90c700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f079806d6b0 0x7f079815da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:24.811 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.809+0000 7f079f90c700 10 monclient(hunting): picked mon.noname-c con 0x7f079806d6b0 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:24.811 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.809+0000 7f079f90c700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:24.811 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.809+0000 7f079f90c700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f079813ecd0 con 0x7f079806b690 2023-09-30T02:06:24.811 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.809+0000 7f079f90c700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:24.811 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.809+0000 7f079f90c700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f079806ca00 con 0x7f0784003a60 2023-09-30T02:06:24.811 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.809+0000 7f079f90c700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:24.811 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.809+0000 7f079f90c700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f079813f120 con 0x7f079806d6b0 2023-09-30T02:06:24.811 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.809+0000 7f079f90c700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:24.811 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.809+0000 7f079f90c700 10 monclient(hunting): authenticate will time out at 1830.111996s 2023-09-30T02:06:24.820 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.820+0000 7f1d467fc700 1 -- 172.21.15.59:0/3857864768 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f1d3801b3b0 con 0x7f1d500b1500 2023-09-30T02:06:24.820 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.821+0000 7f1d467fc700 1 --2- 172.21.15.59:0/3857864768 >> v2:172.21.15.110:6800/115870 conn(0x7f1d30068910 0x7f1d3006adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:24.820 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.821+0000 7f1d467fc700 1 -- 172.21.15.59:0/3857864768 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f1d38083740 con 0x7f1d500b1500 2023-09-30T02:06:24.820 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.821+0000 7f1d467fc700 10 client.4341.objecter ms_dispatch 0x7f1d50001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:06:24.820 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.821+0000 7f1d467fc700 3 client.4341.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:06:24.821 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.821+0000 7f1d467fc700 3 client.4341.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:06:24.821 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.821+0000 7f1d467fc700 20 client.4341.objecter dump_active .. 0 homeless 2023-09-30T02:06:24.821 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.822+0000 7f1d4e59c700 1 --2- 172.21.15.59:0/3857864768 >> v2:172.21.15.110:6800/115870 conn(0x7f1d30068910 0x7f1d3006adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:24.821 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.822+0000 7f1d4e59c700 10 monclient: get_auth_request con 0x7f1d30068910 auth_method 0 2023-09-30T02:06:24.821 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.822+0000 7f1d4e59c700 1 --2- 172.21.15.59:0/3857864768 >> v2:172.21.15.110:6800/115870 conn(0x7f1d30068910 0x7f1d3006adc0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1d400097c0 tx=0x7f1d40006ce0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:24.822 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.822+0000 7f1d467fc700 10 client.4341.objecter ms_handle_connect 0x7f1d30068910 2023-09-30T02:06:24.868 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.868+0000 7f27cbfff700 1 -- 172.21.15.59:0/1035930162 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f27dc012070 con 0x7f27e4142290 2023-09-30T02:06:24.868 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.868+0000 7f27cbfff700 1 --2- 172.21.15.59:0/1035930162 >> v2:172.21.15.110:6800/115870 conn(0x7f27d0068990 0x7f27d006ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:24.868 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.868+0000 7f27e3fff700 1 --2- 172.21.15.59:0/1035930162 >> v2:172.21.15.110:6800/115870 conn(0x7f27d0068990 0x7f27d006ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:24.868 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.868+0000 7f27e3fff700 10 monclient: get_auth_request con 0x7f27d0068990 auth_method 0 2023-09-30T02:06:24.869 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.869+0000 7f27cbfff700 10 client.4381.objecter ms_handle_connect 0x7f27d0068990 2023-09-30T02:06:24.869 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.869+0000 7f27e3fff700 1 --2- 172.21.15.59:0/1035930162 >> v2:172.21.15.110:6800/115870 conn(0x7f27d0068990 0x7f27d006ae40 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f27d40099e0 tx=0x7f27d4016000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:24.871 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.872+0000 7f27cbfff700 1 -- 172.21.15.59:0/1035930162 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f27dc055900 con 0x7f27e4142290 2023-09-30T02:06:24.871 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.872+0000 7f27cbfff700 10 client.4381.objecter ms_dispatch 0x7f27e4001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:06:24.871 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.872+0000 7f27cbfff700 3 client.4381.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:06:24.871 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.872+0000 7f27cbfff700 3 client.4381.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:06:24.872 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.872+0000 7f27cbfff700 20 client.4381.objecter dump_active .. 0 homeless 2023-09-30T02:06:24.942 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.942+0000 7f35c142d700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2023-09-30T02:06:24.942 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.942+0000 7f35c142d700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-09-30T02:06:24.942 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.942+0000 7f35c142d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:06:24.942 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.942+0000 7f35c142d700 1 -- 172.21.15.59:0/2524337962 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f35bc13a2c0 con 0x7f35bc0b2f70 2023-09-30T02:06:24.981 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.982+0000 7fc50a7fc700 1 -- 172.21.15.59:0/3488309627 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc51c13e130 con 0x7fc51c0a5f40 2023-09-30T02:06:24.981 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.982+0000 7fc50a7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2023-09-30T02:06:24.981 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.982+0000 7fc50a7fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:06:24.982 INFO:teuthology.orchestra.run.smithi059.stdout:60129542150 2023-09-30T02:06:24.985 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.985+0000 7fc52084b700 10 librados: watch_flush enter 2023-09-30T02:06:24.985 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.985+0000 7fc52084b700 10 librados: watch_flush exit 2023-09-30T02:06:24.985 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.985+0000 7fc52084b700 20 client.4363.objecter shutdown clearing up homeless session... 2023-09-30T02:06:24.985 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.985+0000 7fc52084b700 10 client.4363.objecter successfully canceled tick 2023-09-30T02:06:24.986 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.985+0000 7fc52084b700 1 -- 172.21.15.59:0/3488309627 >> v2:172.21.15.110:6800/115870 conn(0x7fc4fc068890 msgr2=0x7fc4fc06ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:24.986 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.985+0000 7fc52084b700 1 --2- 172.21.15.59:0/3488309627 >> v2:172.21.15.110:6800/115870 conn(0x7fc4fc068890 0x7fc4fc06ad40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc50c0094c0 tx=0x7fc50c017000 comp rx=0 tx=0).stop 2023-09-30T02:06:24.986 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.985+0000 7fc52084b700 10 monclient: shutdown 2023-09-30T02:06:24.986 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.985+0000 7fc52084b700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:24.986 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.985+0000 7fc52084b700 1 -- 172.21.15.59:0/3488309627 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc51c0a5f40 msgr2=0x7fc51c139bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:24.986 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.985+0000 7fc52084b700 1 --2- 172.21.15.59:0/3488309627 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc51c0a5f40 0x7fc51c139bf0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc51000c590 tx=0x7fc51000c8a0 comp rx=0 tx=0).stop 2023-09-30T02:06:24.986 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.986+0000 7fc52084b700 1 -- 172.21.15.59:0/3488309627 shutdown_connections 2023-09-30T02:06:24.986 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.986+0000 7fc52084b700 1 --2- 172.21.15.59:0/3488309627 >> v2:172.21.15.110:6800/115870 conn(0x7fc4fc068890 0x7fc4fc06ad40 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:24.986 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.986+0000 7fc52084b700 1 --2- 172.21.15.59:0/3488309627 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc51c142290 0x7fc51c13e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:24.986 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.986+0000 7fc52084b700 1 --2- 172.21.15.59:0/3488309627 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc51c0a5b70 0x7fc51c13db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:24.986 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.986+0000 7fc52084b700 1 --2- 172.21.15.59:0/3488309627 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc51c0a5f40 0x7fc51c139bf0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:24.987 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.986+0000 7fc52084b700 1 -- 172.21.15.59:0/3488309627 >> 172.21.15.59:0/3488309627 conn(0x7fc51c0631f0 msgr2=0x7fc51c0635c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:24.987 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.986+0000 7fc52084b700 1 -- 172.21.15.59:0/3488309627 shutdown_connections 2023-09-30T02:06:24.987 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.986+0000 7fc52084b700 1 -- 172.21.15.59:0/3488309627 wait complete. 2023-09-30T02:06:24.987 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.986+0000 7fc52084b700 1 librados: shutdown 2023-09-30T02:06:24.996 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.996+0000 7f33cac6c700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2023-09-30T02:06:24.996 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.996+0000 7f33cac6c700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2023-09-30T02:06:24.996 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.996+0000 7f33cac6c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:06:24.996 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:24.996+0000 7f33cac6c700 1 -- 172.21.15.59:0/2578571364 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f33c406a720 con 0x7f33c40a5f40 2023-09-30T02:06:25.004 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542150 for osd.7 2023-09-30T02:06:25.014 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.015+0000 7fa8a97fa700 1 -- 172.21.15.59:0/144204252 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fa8a401bb50 con 0x7fa8b40a5f40 2023-09-30T02:06:25.015 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.015+0000 7fa8a97fa700 1 --2- 172.21.15.59:0/144204252 >> v2:172.21.15.110:6800/115870 conn(0x7fa898068890 0x7fa89806ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:25.015 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.015+0000 7fa8a97fa700 1 -- 172.21.15.59:0/144204252 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fa8a4085f40 con 0x7fa8b40a5f40 2023-09-30T02:06:25.015 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.015+0000 7fa8a97fa700 10 client.4328.objecter ms_dispatch 0x7fa8b4001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:06:25.015 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.015+0000 7fa8a97fa700 3 client.4328.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:06:25.015 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.015+0000 7fa8a97fa700 3 client.4328.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:06:25.015 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.015+0000 7fa8b9010700 1 --2- 172.21.15.59:0/144204252 >> v2:172.21.15.110:6800/115870 conn(0x7fa898068890 0x7fa89806ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:25.015 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.015+0000 7fa8b9010700 10 monclient: get_auth_request con 0x7fa898068890 auth_method 0 2023-09-30T02:06:25.015 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.015+0000 7fa8a97fa700 20 client.4328.objecter dump_active .. 0 homeless 2023-09-30T02:06:25.015 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.016+0000 7fa8a97fa700 10 client.4328.objecter ms_handle_connect 0x7fa898068890 2023-09-30T02:06:25.016 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.016+0000 7fa8b9010700 1 --2- 172.21.15.59:0/144204252 >> v2:172.21.15.110:6800/115870 conn(0x7fa898068890 0x7fa89806ad40 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa8b00097c0 tx=0x7fa8b0006ce0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:25.038 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.039+0000 7fd0c1b86700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2023-09-30T02:06:25.039 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.039+0000 7fd0c1b86700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-09-30T02:06:25.039 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.039+0000 7fd0c1b86700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:25.039 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.039+0000 7fd0c1b86700 1 -- 172.21.15.59:0/725419722 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fd0bc06d900 con 0x7fd0bc06b810 2023-09-30T02:06:25.066 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.067+0000 7f27cbfff700 1 -- 172.21.15.59:0/1035930162 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f27dc04e5a0 con 0x7f27e4142290 2023-09-30T02:06:25.067 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.067+0000 7f27cbfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:25.067 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.067+0000 7f27cbfff700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:06:25.069 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.069+0000 7f079d6a8700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0784003a60 0x7f079806af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:25.069 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.070+0000 7f079d6a8700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0784003a60 0x7f079806af00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:39230/0 (socket says 172.21.15.59:39230) 2023-09-30T02:06:25.069 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.070+0000 7f079d6a8700 1 -- 172.21.15.59:0/2516421565 learned_addr learned my addr 172.21.15.59:0/2516421565 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:06:25.069 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.070+0000 7f079d6a8700 10 monclient(hunting): get_auth_request con 0x7f0784003a60 auth_method 0 2023-09-30T02:06:25.070 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.070+0000 7f079d6a8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:25.070 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.070+0000 7f079d6a8700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:25.070 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.070+0000 7f079d6a8700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:25.072 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.073+0000 7f079d6a8700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:25.072 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.073+0000 7f079d6a8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:25.072 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.073+0000 7f079d6a8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:25.077 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.078+0000 7f079d6a8700 10 monclient(hunting): handle_auth_done global_id 4384 payload 824 2023-09-30T02:06:25.078 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.078+0000 7f079d6a8700 1 -- 172.21.15.59:0/2516421565 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f079806d6b0 msgr2=0x7f079815da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:25.078 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.078+0000 7f079d6a8700 1 --2- 172.21.15.59:0/2516421565 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f079806d6b0 0x7f079815da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:25.078 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.078+0000 7f079d6a8700 1 -- 172.21.15.59:0/2516421565 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f079806b690 msgr2=0x7f079806d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:25.078 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.078+0000 7f079d6a8700 1 --2- 172.21.15.59:0/2516421565 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f079806b690 0x7f079806d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:25.078 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.078+0000 7f079d6a8700 10 monclient: _finish_hunting 0 2023-09-30T02:06:25.078 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.078+0000 7f079d6a8700 1 monclient: found mon.noname-b 2023-09-30T02:06:25.078 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.078+0000 7f079d6a8700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:25.078 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.078+0000 7f079d6a8700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:06:25.078 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.078+0000 7f079d6a8700 1 -- 172.21.15.59:0/2516421565 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f079806b440 con 0x7f0784003a60 2023-09-30T02:06:25.083 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.083+0000 7f079d6a8700 1 --2- 172.21.15.59:0/2516421565 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0784003a60 0x7f079806af00 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f0794005fa0 tx=0x7f0794015000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:25.083 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.083+0000 7f078d7fa700 10 client.?.objecter ms_handle_connect 0x7f0784003a60 2023-09-30T02:06:25.083 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.083+0000 7f078d7fa700 10 client.?.objecter resend_mon_ops 2023-09-30T02:06:25.084 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.085+0000 7fd0a37fe700 1 -- 172.21.15.59:0/725419722 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd0ac018020 con 0x7fd0bc06b810 2023-09-30T02:06:25.084 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.085+0000 7fd0a37fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-09-30T02:06:25.084 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.085+0000 7fd0a37fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:06:25.085 INFO:teuthology.orchestra.run.smithi059.stdout:77309411333 2023-09-30T02:06:25.087 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.087+0000 7fd0c1b86700 10 librados: watch_flush enter 2023-09-30T02:06:25.087 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.087+0000 7fd0c1b86700 10 librados: watch_flush exit 2023-09-30T02:06:25.088 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.087+0000 7fd0c1b86700 20 client.4378.objecter shutdown clearing up homeless session... 2023-09-30T02:06:25.088 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.087+0000 7fd0c1b86700 10 client.4378.objecter successfully canceled tick 2023-09-30T02:06:25.088 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.087+0000 7fd0c1b86700 1 -- 172.21.15.59:0/725419722 >> v2:172.21.15.110:6800/115870 conn(0x7fd0a8068890 msgr2=0x7fd0a806ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:25.088 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.087+0000 7fd0c1b86700 1 --2- 172.21.15.59:0/725419722 >> v2:172.21.15.110:6800/115870 conn(0x7fd0a8068890 0x7fd0a806ad40 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd0b0006fd0 tx=0x7fd0b0006db0 comp rx=0 tx=0).stop 2023-09-30T02:06:25.088 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.087+0000 7fd0c1b86700 10 monclient: shutdown 2023-09-30T02:06:25.088 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.087+0000 7fd0c1b86700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:25.088 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.087+0000 7fd0c1b86700 1 -- 172.21.15.59:0/725419722 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd0bc06b810 msgr2=0x7fd0bc142960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:25.088 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.087+0000 7fd0c1b86700 1 --2- 172.21.15.59:0/725419722 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd0bc06b810 0x7fd0bc142960 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd0ac00c7c0 tx=0x7fd0ac007f70 comp rx=0 tx=0).stop 2023-09-30T02:06:25.088 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.088+0000 7fd0c1b86700 1 -- 172.21.15.59:0/725419722 shutdown_connections 2023-09-30T02:06:25.088 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.088+0000 7fd0c1b86700 1 --2- 172.21.15.59:0/725419722 >> v2:172.21.15.110:6800/115870 conn(0x7fd0a8068890 0x7fd0a806ad40 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:25.088 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.088+0000 7fd0c1b86700 1 --2- 172.21.15.59:0/725419722 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd0bc06b810 0x7fd0bc142960 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:25.088 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.088+0000 7fd0c1b86700 1 --2- 172.21.15.59:0/725419722 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd0bc139c80 0x7fd0bc0a6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:25.089 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.088+0000 7fd0c1b86700 1 --2- 172.21.15.59:0/725419722 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd0bc06e710 0x7fd0bc118a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:25.089 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.088+0000 7fd0c1b86700 1 -- 172.21.15.59:0/725419722 >> 172.21.15.59:0/725419722 conn(0x7fd0bc067620 msgr2=0x7fd0bc13a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:25.089 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.088+0000 7fd0c1b86700 1 -- 172.21.15.59:0/725419722 shutdown_connections 2023-09-30T02:06:25.089 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.088+0000 7fd0c1b86700 1 -- 172.21.15.59:0/725419722 wait complete. 2023-09-30T02:06:25.089 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.088+0000 7fd0c1b86700 1 librados: shutdown 2023-09-30T02:06:25.090 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.089+0000 7f078d7fa700 1 -- 172.21.15.59:0/2516421565 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07940072e0 con 0x7f0784003a60 2023-09-30T02:06:25.090 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.089+0000 7f078d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:25.090 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.089+0000 7f078d7fa700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:06:25.091 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.089+0000 7f078d7fa700 10 monclient: dump: 2023-09-30T02:06:25.091 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:25.091 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:25.091 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:25.091 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:25.091 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:25.091 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:25.091 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:25.091 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:25.091 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:25.091 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:25.091 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.089+0000 7f078d7fa700 10 monclient: _finish_auth 0 2023-09-30T02:06:25.092 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.089+0000 7f078d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:25.092 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.089+0000 7f078d7fa700 1 -- 172.21.15.59:0/2516421565 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0794007460 con 0x7f0784003a60 2023-09-30T02:06:25.092 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.089+0000 7f079f90c700 5 monclient: authenticate success, global_id 4384 2023-09-30T02:06:25.092 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.089+0000 7f079f90c700 10 monclient: _renew_subs 2023-09-30T02:06:25.092 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.089+0000 7f079f90c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:25.092 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.089+0000 7f079f90c700 1 -- 172.21.15.59:0/2516421565 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f079813e090 con 0x7f0784003a60 2023-09-30T02:06:25.092 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.089+0000 7f079f90c700 10 client.4384.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:06:25.092 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.089+0000 7f079f90c700 10 monclient: _renew_subs 2023-09-30T02:06:25.092 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.089+0000 7f079f90c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:25.092 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.089+0000 7f079f90c700 1 -- 172.21.15.59:0/2516421565 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f079815df40 con 0x7f0784003a60 2023-09-30T02:06:25.092 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.089+0000 7f079f90c700 1 librados: init done 2023-09-30T02:06:25.092 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.089+0000 7f078d7fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:25.093 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.089+0000 7f078d7fa700 1 -- 172.21.15.59:0/2516421565 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0794007820 con 0x7f0784003a60 2023-09-30T02:06:25.093 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.089+0000 7f078d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:25.093 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.089+0000 7f078d7fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:06:25.093 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.089+0000 7f078f7fe700 4 set_mon_vals no callback set 2023-09-30T02:06:25.093 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.089+0000 7f078d7fa700 10 monclient: dump: 2023-09-30T02:06:25.093 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:25.093 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:25.093 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:25.093 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:25.093 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:25.093 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:25.093 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:25.094 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:25.094 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:25.094 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:25.094 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.090+0000 7f079f90c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:25.094 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.090+0000 7f079f90c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:25.094 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.090+0000 7f079f90c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:25.094 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.090+0000 7f079f90c700 1 -- 172.21.15.59:0/2516421565 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07780050b0 con 0x7f0784003a60 2023-09-30T02:06:25.102 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411334 got 77309411333 for osd.0 2023-09-30T02:06:25.112 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.112+0000 7fa8a97fa700 1 -- 172.21.15.59:0/144204252 <== mon.2 v2:172.21.15.59:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa8a4053760 con 0x7fa8b40a5f40 2023-09-30T02:06:25.112 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.112+0000 7fa8a97fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:25.112 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.112+0000 7fa8a97fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:06:25.122 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.123+0000 7f33c0ff9700 1 -- 172.21.15.59:0/2578571364 <== mon.2 v2:172.21.15.59:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f33b8057250 con 0x7f33c40a5f40 2023-09-30T02:06:25.123 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.123+0000 7f33c0ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2023-09-30T02:06:25.123 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.123+0000 7f33c0ff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:06:25.123 INFO:teuthology.orchestra.run.smithi059.stdout:55834574853 2023-09-30T02:06:25.126 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.126+0000 7f33cac6c700 10 librados: watch_flush enter 2023-09-30T02:06:25.126 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.126+0000 7f33cac6c700 10 librados: watch_flush exit 2023-09-30T02:06:25.126 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.126+0000 7f33cac6c700 20 client.4325.objecter shutdown clearing up homeless session... 2023-09-30T02:06:25.126 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.126+0000 7f33cac6c700 10 client.4325.objecter successfully canceled tick 2023-09-30T02:06:25.126 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.126+0000 7f33cac6c700 1 -- 172.21.15.59:0/2578571364 >> v2:172.21.15.110:6800/115870 conn(0x7f33b0068960 msgr2=0x7f33b006ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:25.126 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.126+0000 7f33cac6c700 1 --2- 172.21.15.59:0/2578571364 >> v2:172.21.15.110:6800/115870 conn(0x7f33b0068960 0x7f33b006ae10 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f33b40097c0 tx=0x7f33b4006ce0 comp rx=0 tx=0).stop 2023-09-30T02:06:25.126 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.127+0000 7f33cac6c700 10 monclient: shutdown 2023-09-30T02:06:25.126 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.127+0000 7f33cac6c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:25.127 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.127+0000 7f33cac6c700 1 -- 172.21.15.59:0/2578571364 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f33c40a5f40 msgr2=0x7f33c413dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:25.127 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.127+0000 7f33cac6c700 1 --2- 172.21.15.59:0/2578571364 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f33c40a5f40 0x7f33c413dbe0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f33b8009290 tx=0x7f33b8016000 comp rx=0 tx=0).stop 2023-09-30T02:06:25.127 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.128+0000 7f33cac6c700 1 -- 172.21.15.59:0/2578571364 shutdown_connections 2023-09-30T02:06:25.127 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.128+0000 7f33cac6c700 1 --2- 172.21.15.59:0/2578571364 >> v2:172.21.15.110:6800/115870 conn(0x7f33b0068960 0x7f33b006ae10 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:25.128 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.128+0000 7f33cac6c700 1 --2- 172.21.15.59:0/2578571364 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f33c40a5b70 0x7f33c413c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:25.128 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.128+0000 7f33cac6c700 1 --2- 172.21.15.59:0/2578571364 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33c413e800 0x7f33c4151170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:25.128 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.128+0000 7f33cac6c700 1 --2- 172.21.15.59:0/2578571364 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f33c40a5f40 0x7f33c413dbe0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:25.128 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.128+0000 7f33cac6c700 1 -- 172.21.15.59:0/2578571364 >> 172.21.15.59:0/2578571364 conn(0x7f33c40631f0 msgr2=0x7f33c4141b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:25.128 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.128+0000 7f33cac6c700 1 -- 172.21.15.59:0/2578571364 shutdown_connections 2023-09-30T02:06:25.128 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.128+0000 7f33cac6c700 1 -- 172.21.15.59:0/2578571364 wait complete. 2023-09-30T02:06:25.128 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.128+0000 7f33cac6c700 1 librados: shutdown 2023-09-30T02:06:25.147 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.6 2023-09-30T02:06:25.201 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.201+0000 7f1d467fc700 1 -- 172.21.15.59:0/3857864768 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1d38050fe0 con 0x7f1d500b1500 2023-09-30T02:06:25.201 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.201+0000 7f1d467fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:25.201 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.201+0000 7f1d467fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:06:25.202 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.203+0000 7f35a2ffd700 1 -- 172.21.15.59:0/2524337962 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f35bc13a2c0 con 0x7f35bc0b2f70 2023-09-30T02:06:25.202 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.203+0000 7f35a2ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-09-30T02:06:25.202 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.203+0000 7f35a2ffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:06:25.202 INFO:teuthology.orchestra.run.smithi059.stdout:77309411332 2023-09-30T02:06:25.205 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.205+0000 7f35c142d700 10 librados: watch_flush enter 2023-09-30T02:06:25.205 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.205+0000 7f35c142d700 10 librados: watch_flush exit 2023-09-30T02:06:25.205 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.205+0000 7f35c142d700 20 client.4332.objecter shutdown clearing up homeless session... 2023-09-30T02:06:25.205 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.205+0000 7f35c142d700 10 client.4332.objecter successfully canceled tick 2023-09-30T02:06:25.205 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.206+0000 7f35c142d700 1 -- 172.21.15.59:0/2524337962 >> v2:172.21.15.110:6800/115870 conn(0x7f35a8068910 msgr2=0x7f35a806adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:25.206 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.206+0000 7f35c142d700 1 --2- 172.21.15.59:0/2524337962 >> v2:172.21.15.110:6800/115870 conn(0x7f35a8068910 0x7f35a806adc0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f35ac0016a0 tx=0x7f35ac007b30 comp rx=0 tx=0).stop 2023-09-30T02:06:25.206 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.206+0000 7f35c142d700 10 monclient: shutdown 2023-09-30T02:06:25.206 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.206+0000 7f35c142d700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:25.206 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.206+0000 7f35c142d700 1 -- 172.21.15.59:0/2524337962 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35bc0b2f70 msgr2=0x7f35bc14bd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:25.206 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.206+0000 7f35c142d700 1 --2- 172.21.15.59:0/2524337962 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35bc0b2f70 0x7f35bc14bd60 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f35b000b810 tx=0x7f35b0009220 comp rx=0 tx=0).stop 2023-09-30T02:06:25.207 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.208+0000 7f35c142d700 1 -- 172.21.15.59:0/2524337962 shutdown_connections 2023-09-30T02:06:25.207 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.208+0000 7f35c142d700 1 --2- 172.21.15.59:0/2524337962 >> v2:172.21.15.110:6800/115870 conn(0x7f35a8068910 0x7f35a806adc0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:25.207 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.208+0000 7f35c142d700 1 --2- 172.21.15.59:0/2524337962 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f35bc06dd20 0x7f35bc13cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:25.207 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.208+0000 7f35c142d700 1 --2- 172.21.15.59:0/2524337962 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f35bc0b3340 0x7f35bc14c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:25.207 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.208+0000 7f35c142d700 1 --2- 172.21.15.59:0/2524337962 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f35bc0b2f70 0x7f35bc14bd60 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:25.208 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.208+0000 7f35c142d700 1 -- 172.21.15.59:0/2524337962 >> 172.21.15.59:0/2524337962 conn(0x7f35bc0637c0 msgr2=0x7f35bc070f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:25.208 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.208+0000 7f35c142d700 1 -- 172.21.15.59:0/2524337962 shutdown_connections 2023-09-30T02:06:25.208 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.208+0000 7f35c142d700 1 -- 172.21.15.59:0/2524337962 wait complete. 2023-09-30T02:06:25.208 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.209+0000 7f35c142d700 1 librados: shutdown 2023-09-30T02:06:25.224 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411333 got 77309411332 for osd.2 2023-09-30T02:06:25.284 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.284+0000 7f27ea4d1700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2023-09-30T02:06:25.284 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.284+0000 7f27ea4d1700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-09-30T02:06:25.284 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.284+0000 7f27ea4d1700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:25.284 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.284+0000 7f27ea4d1700 1 -- 172.21.15.59:0/1035930162 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f27c4005630 con 0x7f27e4142290 2023-09-30T02:06:25.328 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.328+0000 7fa8bb274700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2023-09-30T02:06:25.328 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.328+0000 7fa8bb274700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-09-30T02:06:25.328 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.328+0000 7fa8bb274700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:06:25.328 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.328+0000 7fa8bb274700 1 -- 172.21.15.59:0/144204252 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fa8b41400d0 con 0x7fa8b40a5f40 2023-09-30T02:06:25.339 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.340+0000 7f078d7fa700 1 -- 172.21.15.59:0/2516421565 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f0794019030 con 0x7f0784003a60 2023-09-30T02:06:25.340 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.340+0000 7f078d7fa700 1 --2- 172.21.15.59:0/2516421565 >> v2:172.21.15.110:6800/115870 conn(0x7f077c068890 0x7f077c06ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:25.340 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.341+0000 7f079cea7700 1 --2- 172.21.15.59:0/2516421565 >> v2:172.21.15.110:6800/115870 conn(0x7f077c068890 0x7f077c06ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:25.340 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.341+0000 7f079cea7700 10 monclient: get_auth_request con 0x7f077c068890 auth_method 0 2023-09-30T02:06:25.341 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.341+0000 7f078d7fa700 10 client.4384.objecter ms_handle_connect 0x7f077c068890 2023-09-30T02:06:25.341 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.341+0000 7f079cea7700 1 --2- 172.21.15.59:0/2516421565 >> v2:172.21.15.110:6800/115870 conn(0x7f077c068890 0x7f077c06ad40 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f079806cc10 tx=0x7f0788006d40 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:25.344 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.345+0000 7f078d7fa700 1 -- 172.21.15.59:0/2516421565 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f0794057680 con 0x7f0784003a60 2023-09-30T02:06:25.344 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.345+0000 7f078d7fa700 10 client.4384.objecter ms_dispatch 0x7f0798001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:06:25.345 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.345+0000 7f078d7fa700 3 client.4384.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:06:25.345 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.345+0000 7f078d7fa700 3 client.4384.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:06:25.345 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.345+0000 7f078d7fa700 20 client.4384.objecter dump_active .. 0 homeless 2023-09-30T02:06:25.350 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.351+0000 7fa8a97fa700 1 -- 172.21.15.59:0/144204252 <== mon.2 v2:172.21.15.59:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa8a402a030 con 0x7fa8b40a5f40 2023-09-30T02:06:25.350 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.351+0000 7fa8a97fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-09-30T02:06:25.350 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.351+0000 7fa8a97fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:06:25.351 INFO:teuthology.orchestra.run.smithi059.stdout:64424509446 2023-09-30T02:06:25.353 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.353+0000 7fa8bb274700 10 librados: watch_flush enter 2023-09-30T02:06:25.353 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.353+0000 7fa8bb274700 10 librados: watch_flush exit 2023-09-30T02:06:25.354 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.353+0000 7fa8bb274700 20 client.4328.objecter shutdown clearing up homeless session... 2023-09-30T02:06:25.354 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.353+0000 7fa8bb274700 10 client.4328.objecter successfully canceled tick 2023-09-30T02:06:25.354 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.353+0000 7fa8bb274700 1 -- 172.21.15.59:0/144204252 >> v2:172.21.15.110:6800/115870 conn(0x7fa898068890 msgr2=0x7fa89806ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:25.354 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.353+0000 7fa8bb274700 1 --2- 172.21.15.59:0/144204252 >> v2:172.21.15.110:6800/115870 conn(0x7fa898068890 0x7fa89806ad40 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa8b00097c0 tx=0x7fa8b0006ce0 comp rx=0 tx=0).stop 2023-09-30T02:06:25.354 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.353+0000 7fa8bb274700 10 monclient: shutdown 2023-09-30T02:06:25.354 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.353+0000 7fa8bb274700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:25.354 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.353+0000 7fa8bb274700 1 -- 172.21.15.59:0/144204252 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa8b40a5f40 msgr2=0x7fa8b41407c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:25.354 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.353+0000 7fa8bb274700 1 --2- 172.21.15.59:0/144204252 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa8b40a5f40 0x7fa8b41407c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa8b40b1ff0 tx=0x7fa8a4016000 comp rx=0 tx=0).stop 2023-09-30T02:06:25.354 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.354+0000 7fa8bb274700 1 -- 172.21.15.59:0/144204252 shutdown_connections 2023-09-30T02:06:25.355 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.354+0000 7fa8bb274700 1 --2- 172.21.15.59:0/144204252 >> v2:172.21.15.110:6800/115870 conn(0x7fa898068890 0x7fa89806ad40 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:25.355 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.354+0000 7fa8bb274700 1 --2- 172.21.15.59:0/144204252 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa8b4142290 0x7fa8b4151340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:25.355 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.354+0000 7fa8bb274700 1 --2- 172.21.15.59:0/144204252 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa8b40a5b70 0x7fa8b4140280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:25.355 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.354+0000 7fa8bb274700 1 --2- 172.21.15.59:0/144204252 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa8b40a5f40 0x7fa8b41407c0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:25.355 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.354+0000 7fa8bb274700 1 -- 172.21.15.59:0/144204252 >> 172.21.15.59:0/144204252 conn(0x7fa8b40631f0 msgr2=0x7fa8b4141b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:25.355 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.354+0000 7fa8bb274700 1 -- 172.21.15.59:0/144204252 shutdown_connections 2023-09-30T02:06:25.355 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.354+0000 7fa8bb274700 1 -- 172.21.15.59:0/144204252 wait complete. 2023-09-30T02:06:25.355 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.354+0000 7fa8bb274700 1 librados: shutdown 2023-09-30T02:06:25.355 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.354+0000 7f078d7fa700 1 -- 172.21.15.59:0/2516421565 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f07940532c0 con 0x7f0784003a60 2023-09-30T02:06:25.355 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.354+0000 7f078d7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:25.356 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.354+0000 7f078d7fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:06:25.367 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509446 for osd.4 2023-09-30T02:06:25.427 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.428+0000 7f27cbfff700 1 -- 172.21.15.59:0/1035930162 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f27dc051370 con 0x7f27e4142290 2023-09-30T02:06:25.428 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.428+0000 7f27cbfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-09-30T02:06:25.428 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.428+0000 7f27cbfff700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:06:25.428 INFO:teuthology.orchestra.run.smithi059.stdout:81604378629 2023-09-30T02:06:25.431 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.431+0000 7f27ea4d1700 10 librados: watch_flush enter 2023-09-30T02:06:25.431 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.431+0000 7f27ea4d1700 10 librados: watch_flush exit 2023-09-30T02:06:25.432 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.431+0000 7f27ea4d1700 20 client.4381.objecter shutdown clearing up homeless session... 2023-09-30T02:06:25.432 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.431+0000 7f27ea4d1700 10 client.4381.objecter successfully canceled tick 2023-09-30T02:06:25.432 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.432+0000 7f27ea4d1700 1 -- 172.21.15.59:0/1035930162 >> v2:172.21.15.110:6800/115870 conn(0x7f27d0068990 msgr2=0x7f27d006ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:25.432 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.432+0000 7f27ea4d1700 1 --2- 172.21.15.59:0/1035930162 >> v2:172.21.15.110:6800/115870 conn(0x7f27d0068990 0x7f27d006ae40 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f27d40099e0 tx=0x7f27d4016000 comp rx=0 tx=0).stop 2023-09-30T02:06:25.432 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.432+0000 7f27ea4d1700 10 monclient: shutdown 2023-09-30T02:06:25.432 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.432+0000 7f27ea4d1700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:25.432 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.432+0000 7f27ea4d1700 1 -- 172.21.15.59:0/1035930162 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f27e4142290 msgr2=0x7f27e413e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:25.432 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.432+0000 7f27ea4d1700 1 --2- 172.21.15.59:0/1035930162 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f27e4142290 0x7f27e413e100 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f27dc00bc30 tx=0x7f27dc009840 comp rx=0 tx=0).stop 2023-09-30T02:06:25.432 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.432+0000 7f27ea4d1700 1 -- 172.21.15.59:0/1035930162 shutdown_connections 2023-09-30T02:06:25.432 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.432+0000 7f27ea4d1700 1 --2- 172.21.15.59:0/1035930162 >> v2:172.21.15.110:6800/115870 conn(0x7f27d0068990 0x7f27d006ae40 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:25.432 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.432+0000 7f27ea4d1700 1 --2- 172.21.15.59:0/1035930162 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f27e4142290 0x7f27e413e100 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:25.433 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.432+0000 7f27ea4d1700 1 --2- 172.21.15.59:0/1035930162 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f27e40a5b70 0x7f27e41400f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:25.433 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.432+0000 7f27ea4d1700 1 --2- 172.21.15.59:0/1035930162 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f27e40a5f40 0x7f27e4154970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:25.433 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.432+0000 7f27ea4d1700 1 -- 172.21.15.59:0/1035930162 >> 172.21.15.59:0/1035930162 conn(0x7f27e40631f0 msgr2=0x7f27e40635c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:25.433 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.432+0000 7f27ea4d1700 1 -- 172.21.15.59:0/1035930162 shutdown_connections 2023-09-30T02:06:25.433 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.432+0000 7f27ea4d1700 1 -- 172.21.15.59:0/1035930162 wait complete. 2023-09-30T02:06:25.433 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.432+0000 7f27ea4d1700 1 librados: shutdown 2023-09-30T02:06:25.445 INFO:tasks.ceph.ceph_manager.ceph:need seq 81604378630 got 81604378629 for osd.3 2023-09-30T02:06:25.488 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.489+0000 7f1d54ac5700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2023-09-30T02:06:25.488 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.489+0000 7f1d54ac5700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-09-30T02:06:25.488 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.489+0000 7f1d54ac5700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:06:25.489 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.489+0000 7f1d54ac5700 1 -- 172.21.15.59:0/3857864768 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f1d5013e530 con 0x7f1d500b1500 2023-09-30T02:06:25.521 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.521+0000 7f1d467fc700 1 -- 172.21.15.59:0/3857864768 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1d38017020 con 0x7f1d500b1500 2023-09-30T02:06:25.521 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.521+0000 7f1d467fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-09-30T02:06:25.521 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.521+0000 7f1d467fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:06:25.521 INFO:teuthology.orchestra.run.smithi059.stdout:77309411333 2023-09-30T02:06:25.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.525+0000 7f1d54ac5700 10 librados: watch_flush enter 2023-09-30T02:06:25.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.525+0000 7f1d54ac5700 10 librados: watch_flush exit 2023-09-30T02:06:25.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.525+0000 7f1d54ac5700 20 client.4341.objecter shutdown clearing up homeless session... 2023-09-30T02:06:25.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.525+0000 7f1d54ac5700 10 client.4341.objecter successfully canceled tick 2023-09-30T02:06:25.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.525+0000 7f1d54ac5700 1 -- 172.21.15.59:0/3857864768 >> v2:172.21.15.110:6800/115870 conn(0x7f1d30068910 msgr2=0x7f1d3006adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:25.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.525+0000 7f1d54ac5700 1 --2- 172.21.15.59:0/3857864768 >> v2:172.21.15.110:6800/115870 conn(0x7f1d30068910 0x7f1d3006adc0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1d400097c0 tx=0x7f1d40006ce0 comp rx=0 tx=0).stop 2023-09-30T02:06:25.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.525+0000 7f1d54ac5700 10 monclient: shutdown 2023-09-30T02:06:25.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.525+0000 7f1d54ac5700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:25.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.525+0000 7f1d54ac5700 1 -- 172.21.15.59:0/3857864768 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d500b1500 msgr2=0x7f1d5013ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:25.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.525+0000 7f1d54ac5700 1 --2- 172.21.15.59:0/3857864768 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d500b1500 0x7f1d5013ece0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1d3800c560 tx=0x7f1d38001020 comp rx=0 tx=0).stop 2023-09-30T02:06:25.526 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.526+0000 7f1d54ac5700 1 -- 172.21.15.59:0/3857864768 shutdown_connections 2023-09-30T02:06:25.526 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.526+0000 7f1d54ac5700 1 --2- 172.21.15.59:0/3857864768 >> v2:172.21.15.110:6800/115870 conn(0x7f1d30068910 0x7f1d3006adc0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:25.526 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.526+0000 7f1d54ac5700 1 --2- 172.21.15.59:0/3857864768 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1d500a5b70 0x7f1d50140c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:25.526 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.526+0000 7f1d54ac5700 1 --2- 172.21.15.59:0/3857864768 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1d50142290 0x7f1d5014e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:25.527 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.526+0000 7f1d54ac5700 1 --2- 172.21.15.59:0/3857864768 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d500b1500 0x7f1d5013ece0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:25.527 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.526+0000 7f1d54ac5700 1 -- 172.21.15.59:0/3857864768 >> 172.21.15.59:0/3857864768 conn(0x7f1d500631f0 msgr2=0x7f1d50141b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:25.527 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.527+0000 7f1d54ac5700 1 -- 172.21.15.59:0/3857864768 shutdown_connections 2023-09-30T02:06:25.527 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.527+0000 7f1d54ac5700 1 -- 172.21.15.59:0/3857864768 wait complete. 2023-09-30T02:06:25.527 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.527+0000 7f1d54ac5700 1 librados: shutdown 2023-09-30T02:06:25.545 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411333 got 77309411333 for osd.1 2023-09-30T02:06:25.545 DEBUG:teuthology.parallel:result is None 2023-09-30T02:06:25.569 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.569+0000 7f079f90c700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2023-09-30T02:06:25.569 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.569+0000 7f079f90c700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-09-30T02:06:25.569 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.569+0000 7f079f90c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:25.569 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.569+0000 7f079f90c700 1 -- 172.21.15.59:0/2516421565 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f0778005630 con 0x7f0784003a60 2023-09-30T02:06:25.572 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.573+0000 7f078d7fa700 1 -- 172.21.15.59:0/2516421565 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f07940568e0 con 0x7f0784003a60 2023-09-30T02:06:25.572 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.573+0000 7f078d7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-09-30T02:06:25.573 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.573+0000 7f078d7fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:06:25.573 INFO:teuthology.orchestra.run.smithi059.stdout:64424509446 2023-09-30T02:06:25.576 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.575+0000 7f079f90c700 10 librados: watch_flush enter 2023-09-30T02:06:25.576 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.575+0000 7f079f90c700 10 librados: watch_flush exit 2023-09-30T02:06:25.576 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.575+0000 7f079f90c700 20 client.4384.objecter shutdown clearing up homeless session... 2023-09-30T02:06:25.576 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.575+0000 7f079f90c700 10 client.4384.objecter successfully canceled tick 2023-09-30T02:06:25.576 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.576+0000 7f079f90c700 1 -- 172.21.15.59:0/2516421565 >> v2:172.21.15.110:6800/115870 conn(0x7f077c068890 msgr2=0x7f077c06ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:25.576 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.576+0000 7f079f90c700 1 --2- 172.21.15.59:0/2516421565 >> v2:172.21.15.110:6800/115870 conn(0x7f077c068890 0x7f077c06ad40 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f079806cc10 tx=0x7f0788006d40 comp rx=0 tx=0).stop 2023-09-30T02:06:25.576 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.576+0000 7f079f90c700 10 monclient: shutdown 2023-09-30T02:06:25.576 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.576+0000 7f079f90c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:25.576 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.576+0000 7f079f90c700 1 -- 172.21.15.59:0/2516421565 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0784003a60 msgr2=0x7f079806af00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:25.577 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.576+0000 7f079f90c700 1 --2- 172.21.15.59:0/2516421565 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0784003a60 0x7f079806af00 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f0794005fa0 tx=0x7f0794015000 comp rx=0 tx=0).stop 2023-09-30T02:06:25.577 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.576+0000 7f079f90c700 1 -- 172.21.15.59:0/2516421565 shutdown_connections 2023-09-30T02:06:25.577 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.576+0000 7f079f90c700 1 --2- 172.21.15.59:0/2516421565 >> v2:172.21.15.110:6800/115870 conn(0x7f077c068890 0x7f077c06ad40 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:25.577 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.576+0000 7f079f90c700 1 --2- 172.21.15.59:0/2516421565 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0784003a60 0x7f079806af00 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:25.577 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.576+0000 7f079f90c700 1 --2- 172.21.15.59:0/2516421565 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f079806d6b0 0x7f079815da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:25.577 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.576+0000 7f079f90c700 1 --2- 172.21.15.59:0/2516421565 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f079806b690 0x7f079806d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:25.577 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.576+0000 7f079f90c700 1 -- 172.21.15.59:0/2516421565 >> 172.21.15.59:0/2516421565 conn(0x7f07980631f0 msgr2=0x7f0798070f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:25.577 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.576+0000 7f079f90c700 1 -- 172.21.15.59:0/2516421565 shutdown_connections 2023-09-30T02:06:25.577 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.576+0000 7f079f90c700 1 -- 172.21.15.59:0/2516421565 wait complete. 2023-09-30T02:06:25.577 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:25.576+0000 7f079f90c700 1 librados: shutdown 2023-09-30T02:06:25.589 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509446 for osd.5 2023-09-30T02:06:25.589 DEBUG:teuthology.parallel:result is None 2023-09-30T02:06:26.006 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-09-30T02:06:26.103 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-09-30T02:06:26.147 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-09-30T02:06:26.151 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.150+0000 7f6f01605700 10 monclient: get_monmap_and_config 2023-09-30T02:06:26.151 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.150+0000 7f6f01605700 10 monclient: build_initial_monmap 2023-09-30T02:06:26.151 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.150+0000 7f6f01605700 10 monclient: monmap: 2023-09-30T02:06:26.151 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:26.152 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:26.152 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:06:26.151126+0000 2023-09-30T02:06:26.152 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:06:26.151126+0000 2023-09-30T02:06:26.152 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:26.152 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:26.152 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:06:26.152 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:06:26.152 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:06:26.152 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:06:26.152 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:06:26.152 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:26.153 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.151+0000 7f6f01605700 1 Processor -- start 2023-09-30T02:06:26.153 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.151+0000 7f6f01605700 1 -- start start 2023-09-30T02:06:26.153 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.151+0000 7f6f01605700 10 monclient: init 2023-09-30T02:06:26.153 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.152+0000 7f6f01605700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:26.153 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.152+0000 7f6f01605700 10 monclient: _add_conns ranks=[2,3,0,4,1] 2023-09-30T02:06:26.153 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.152+0000 7f6f01605700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6efc06c620 0x7f6efc06e770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:26.153 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.152+0000 7f6f01605700 10 monclient(hunting): picked mon.noname-b con 0x7f6efc06c620 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:26.153 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.152+0000 7f6f01605700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f6efc06c9f0 0x7f6efc13b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:26.153 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.152+0000 7f6f01605700 10 monclient(hunting): picked mon.noname-a con 0x7f6efc06c9f0 addr v2:172.21.15.59:3300/0 2023-09-30T02:06:26.153 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.152+0000 7f6f01605700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6efc13b6e0 addr v1:172.21.15.59:6789/0 2023-09-30T02:06:26.153 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.152+0000 7f6f01605700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6efc06ef60 con 0x7f6efc13b6e0 2023-09-30T02:06:26.153 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.152+0000 7f6f01605700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:26.154 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.152+0000 7f6f01605700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6efc06ecb0 con 0x7f6efc06c9f0 2023-09-30T02:06:26.154 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.152+0000 7f6f01605700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:26.154 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.152+0000 7f6f01605700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6efc06daa0 con 0x7f6efc06c620 2023-09-30T02:06:26.154 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.152+0000 7f6f01605700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:26.154 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.152+0000 7f6f01605700 10 monclient(hunting): authenticate will time out at 1831.454187s 2023-09-30T02:06:26.155 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.152+0000 7f6efaffd700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6efc06c620 0x7f6efc06e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:26.155 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.153+0000 7f6efaffd700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6efc06c620 0x7f6efc06e770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:47846/0 (socket says 172.21.15.59:47846) 2023-09-30T02:06:26.155 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.153+0000 7f6efaffd700 1 -- 172.21.15.59:0/925329457 learned_addr learned my addr 172.21.15.59:0/925329457 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:06:26.155 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.153+0000 7f6efaffd700 10 monclient(hunting): get_auth_request con 0x7f6efc06c620 auth_method 0 2023-09-30T02:06:26.155 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.153+0000 7f6efaffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:26.155 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.153+0000 7f6efaffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:26.155 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.153+0000 7f6efaffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:26.155 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.153+0000 7f6efa7fc700 1 --2- 172.21.15.59:0/925329457 >> v2:172.21.15.59:3300/0 conn(0x7f6efc06c9f0 0x7f6efc13b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:26.156 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.154+0000 7f6efaffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:26.156 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.154+0000 7f6efaffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:26.156 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.154+0000 7f6efaffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:26.156 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.155+0000 7f6efa7fc700 10 monclient(hunting): get_auth_request con 0x7f6efc06c9f0 auth_method 0 2023-09-30T02:06:26.156 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.155+0000 7f6efa7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:26.156 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.155+0000 7f6efa7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:26.156 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.155+0000 7f6efa7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:26.156 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.156+0000 7f6efa7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:26.156 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.156+0000 7f6efa7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:26.157 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.156+0000 7f6efa7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:26.157 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.157+0000 7f6efaffd700 10 monclient(hunting): handle_auth_done global_id 4331 payload 274 2023-09-30T02:06:26.157 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.157+0000 7f6efaffd700 1 -- 172.21.15.59:0/925329457 >> v2:172.21.15.59:3300/0 conn(0x7f6efc06c9f0 msgr2=0x7f6efc13b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:06:26.157 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.157+0000 7f6efaffd700 1 --2- 172.21.15.59:0/925329457 >> v2:172.21.15.59:3300/0 conn(0x7f6efc06c9f0 0x7f6efc13b1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:26.157 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.157+0000 7f6efaffd700 1 -- 172.21.15.59:0/925329457 >> v1:172.21.15.59:6789/0 conn(0x7f6efc13b6e0 legacy=0x7f6efc13e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:26.157 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.157+0000 7f6efaffd700 10 monclient: _finish_hunting 0 2023-09-30T02:06:26.157 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.157+0000 7f6efaffd700 1 monclient: found mon.noname-b 2023-09-30T02:06:26.157 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.157+0000 7f6efaffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:26.158 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.157+0000 7f6efaffd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.59:3301/0 2023-09-30T02:06:26.158 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.157+0000 7f6efaffd700 1 -- 172.21.15.59:0/925329457 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6efc13bb60 con 0x7f6efc06c620 2023-09-30T02:06:26.162 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.163+0000 7f6efaffd700 1 --2- 172.21.15.59:0/925329457 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6efc06c620 0x7f6efc06e770 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6eec0099b0 tx=0x7f6eec009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f00a307e99df26b server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:26.168 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.169+0000 7f6ef9ffb700 1 -- 172.21.15.59:0/925329457 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eec009ea0 con 0x7f6efc06c620 2023-09-30T02:06:26.169 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.169+0000 7f6ef9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:26.169 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.169+0000 7f6ef9ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:06:26.169 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.169+0000 7f6ef9ffb700 10 monclient: dump: 2023-09-30T02:06:26.169 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:26.169 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:26.169 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:26.169 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:26.169 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:26.169 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:26.169 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:26.170 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:26.170 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:26.170 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:26.170 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.169+0000 7f6ef9ffb700 10 monclient: _finish_auth 0 2023-09-30T02:06:26.170 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.169+0000 7f6ef9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:26.170 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.169+0000 7f6ef9ffb700 1 -- 172.21.15.59:0/925329457 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6eec00b860 con 0x7f6efc06c620 2023-09-30T02:06:26.170 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.169+0000 7f6ef9ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:26.170 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.169+0000 7f6ef9ffb700 1 -- 172.21.15.59:0/925329457 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eec00bc20 con 0x7f6efc06c620 2023-09-30T02:06:26.170 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.169+0000 7f6ef9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:26.170 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.169+0000 7f6ef9ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:06:26.170 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.169+0000 7f6ef9ffb700 10 monclient: dump: 2023-09-30T02:06:26.171 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:26.171 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:26.171 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:26.171 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:26.171 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:26.171 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:26.171 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:26.171 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:26.171 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:26.171 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:26.173 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.169+0000 7f6f01605700 5 monclient: authenticate success, global_id 4331 2023-09-30T02:06:26.173 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.169+0000 7f6f01605700 10 monclient: get_monmap_and_config success 2023-09-30T02:06:26.173 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.169+0000 7f6f01605700 4 set_mon_vals no callback set 2023-09-30T02:06:26.173 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.170+0000 7f6f01605700 10 monclient: shutdown 2023-09-30T02:06:26.173 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.170+0000 7f6f01605700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:26.173 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.170+0000 7f6f01605700 1 -- 172.21.15.59:0/925329457 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6efc06c620 msgr2=0x7f6efc06e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:26.174 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.170+0000 7f6f01605700 1 --2- 172.21.15.59:0/925329457 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6efc06c620 0x7f6efc06e770 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6eec0099b0 tx=0x7f6eec009250 comp rx=0 tx=0).stop 2023-09-30T02:06:26.174 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.170+0000 7f6f01605700 1 -- 172.21.15.59:0/925329457 shutdown_connections 2023-09-30T02:06:26.174 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.170+0000 7f6f01605700 1 --2- 172.21.15.59:0/925329457 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6efc06c620 0x7f6efc06e770 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:26.174 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.170+0000 7f6f01605700 1 --2- 172.21.15.59:0/925329457 >> v2:172.21.15.59:3300/0 conn(0x7f6efc06c9f0 0x7f6efc13b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:26.174 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.170+0000 7f6f01605700 1 -- 172.21.15.59:0/925329457 >> 172.21.15.59:0/925329457 conn(0x7f6efc069da0 msgr2=0x7f6efc068730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:26.174 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.171+0000 7f6f01605700 1 -- 172.21.15.59:0/925329457 shutdown_connections 2023-09-30T02:06:26.174 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.171+0000 7f6f01605700 1 -- 172.21.15.59:0/925329457 wait complete. 2023-09-30T02:06:26.174 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.172+0000 7f6f01605700 10 monclient: build_initial_monmap 2023-09-30T02:06:26.174 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.172+0000 7f6f01605700 10 monclient: monmap: 2023-09-30T02:06:26.174 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:26.174 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:06:26.174 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:06:26.175 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:06:26.175 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:26.175 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:06:26.175 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:06:26.175 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:06:26.175 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:06:26.175 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:26.175 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.172+0000 7f6f01605700 1 librados: starting msgr at 2023-09-30T02:06:26.175 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.172+0000 7f6f01605700 1 librados: starting objecter 2023-09-30T02:06:26.175 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.172+0000 7f6f01605700 1 Processor -- start 2023-09-30T02:06:26.175 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.172+0000 7f6f01605700 1 -- start start 2023-09-30T02:06:26.175 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.172+0000 7f6f01605700 1 librados: setting wanted keys 2023-09-30T02:06:26.176 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.172+0000 7f6f01605700 1 librados: calling monclient init 2023-09-30T02:06:26.176 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.172+0000 7f6f01605700 10 monclient: init 2023-09-30T02:06:26.176 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.173+0000 7f6f01605700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:26.176 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.173+0000 7f6f01605700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T02:06:26.176 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.173+0000 7f6f01605700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6efc06c620 0x7f6efc13c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:26.176 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.173+0000 7f6f01605700 10 monclient(hunting): picked mon.noname-c con 0x7f6efc06c620 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:26.176 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.173+0000 7f6f01605700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6efc06c9f0 0x7f6efc13d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:26.176 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.173+0000 7f6f01605700 10 monclient(hunting): picked mon.noname-b con 0x7f6efc06c9f0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:06:26.176 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.173+0000 7f6f01605700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6efc13b6e0 0x7f6efc139560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:26.176 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.173+0000 7f6f01605700 10 monclient(hunting): picked mon.noname-a con 0x7f6efc13b6e0 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:06:26.176 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.173+0000 7f6f01605700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:26.176 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.173+0000 7f6f01605700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6efc13aef0 con 0x7f6efc13b6e0 2023-09-30T02:06:26.177 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.173+0000 7f6f01605700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:26.177 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.173+0000 7f6f01605700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6efc138df0 con 0x7f6efc06c9f0 2023-09-30T02:06:26.177 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.173+0000 7f6f01605700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:26.177 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.173+0000 7f6f01605700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6efc1390d0 con 0x7f6efc06c620 2023-09-30T02:06:26.177 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.173+0000 7f6f01605700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:26.177 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.173+0000 7f6f01605700 10 monclient(hunting): authenticate will time out at 1831.475619s 2023-09-30T02:06:26.178 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.174+0000 7f6efa7fc700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6efc06c9f0 0x7f6efc13d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:26.179 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.174+0000 7f6efb7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6efc13b6e0 0x7f6efc139560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:26.179 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.174+0000 7f6efa7fc700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6efc06c9f0 0x7f6efc13d4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:39240/0 (socket says 172.21.15.59:39240) 2023-09-30T02:06:26.179 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.174+0000 7f6efa7fc700 1 -- 172.21.15.59:0/1439181173 learned_addr learned my addr 172.21.15.59:0/1439181173 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:06:26.179 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.174+0000 7f6efa7fc700 10 monclient(hunting): get_auth_request con 0x7f6efc06c9f0 auth_method 0 2023-09-30T02:06:26.179 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.174+0000 7f6efa7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:26.179 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.174+0000 7f6efa7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:26.179 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.174+0000 7f6efa7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:26.179 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.175+0000 7f6efb7fe700 10 monclient(hunting): get_auth_request con 0x7f6efc13b6e0 auth_method 0 2023-09-30T02:06:26.179 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.175+0000 7f6efb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:26.179 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.175+0000 7f6efb7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:26.179 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.175+0000 7f6efb7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:26.179 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.175+0000 7f6efa7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:26.180 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.175+0000 7f6efa7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:26.180 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.175+0000 7f6efa7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:26.180 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.175+0000 7f6efb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:26.180 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.175+0000 7f6efb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:26.180 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.175+0000 7f6efb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:26.180 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.178+0000 7f6efaffd700 1 --2- 172.21.15.59:0/1439181173 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6efc06c620 0x7f6efc13c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:26.180 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.178+0000 7f6efaffd700 10 monclient(hunting): get_auth_request con 0x7f6efc06c620 auth_method 0 2023-09-30T02:06:26.180 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.178+0000 7f6efaffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:26.180 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.178+0000 7f6efaffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:26.180 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.178+0000 7f6efaffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:26.180 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.178+0000 7f6efaffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:26.181 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.178+0000 7f6efaffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:26.181 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.178+0000 7f6efaffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:26.181 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.179+0000 7f6efa7fc700 10 monclient(hunting): handle_auth_done global_id 4387 payload 824 2023-09-30T02:06:26.181 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.179+0000 7f6efa7fc700 1 -- 172.21.15.59:0/1439181173 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6efc06c620 msgr2=0x7f6efc13c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:26.181 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.179+0000 7f6efa7fc700 1 --2- 172.21.15.59:0/1439181173 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6efc06c620 0x7f6efc13c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:26.181 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.179+0000 7f6efa7fc700 1 -- 172.21.15.59:0/1439181173 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6efc13b6e0 msgr2=0x7f6efc139560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:26.181 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.179+0000 7f6efa7fc700 1 --2- 172.21.15.59:0/1439181173 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6efc13b6e0 0x7f6efc139560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:26.181 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.179+0000 7f6efa7fc700 10 monclient: _finish_hunting 0 2023-09-30T02:06:26.181 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.179+0000 7f6efa7fc700 1 monclient: found mon.noname-b 2023-09-30T02:06:26.181 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.179+0000 7f6efa7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:26.181 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.179+0000 7f6efa7fc700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:06:26.181 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.179+0000 7f6efa7fc700 1 -- 172.21.15.59:0/1439181173 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6eec0094f0 con 0x7f6efc06c9f0 2023-09-30T02:06:26.182 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.182+0000 7f6efa7fc700 1 --2- 172.21.15.59:0/1439181173 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6efc06c9f0 0x7f6efc13d4f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6ef000ec80 tx=0x7f6ef00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:26.182 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.182+0000 7f6ee2ffd700 10 client.?.objecter ms_handle_connect 0x7f6efc06c9f0 2023-09-30T02:06:26.182 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.182+0000 7f6ee2ffd700 10 client.?.objecter resend_mon_ops 2023-09-30T02:06:26.186 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.186+0000 7f6ee2ffd700 1 -- 172.21.15.59:0/1439181173 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef000c850 con 0x7f6efc06c9f0 2023-09-30T02:06:26.186 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.186+0000 7f6ee2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:26.186 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.186+0000 7f6ee2ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:06:26.186 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.186+0000 7f6ee2ffd700 10 monclient: dump: 2023-09-30T02:06:26.186 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:26.186 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:26.186 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:26.186 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:26.186 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:26.187 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:26.187 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:26.187 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:26.187 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:26.187 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:26.187 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.186+0000 7f6ee2ffd700 10 monclient: _finish_auth 0 2023-09-30T02:06:26.187 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.186+0000 7f6ee2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:26.187 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.186+0000 7f6ee2ffd700 1 -- 172.21.15.59:0/1439181173 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ef000c9b0 con 0x7f6efc06c9f0 2023-09-30T02:06:26.187 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.186+0000 7f6f01605700 5 monclient: authenticate success, global_id 4387 2023-09-30T02:06:26.187 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.186+0000 7f6f01605700 10 monclient: _renew_subs 2023-09-30T02:06:26.188 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.186+0000 7f6f01605700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:26.188 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.186+0000 7f6f01605700 1 -- 172.21.15.59:0/1439181173 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6efc13a890 con 0x7f6efc06c9f0 2023-09-30T02:06:26.188 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.186+0000 7f6f01605700 10 client.4387.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:06:26.188 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.186+0000 7f6f01605700 10 monclient: _renew_subs 2023-09-30T02:06:26.188 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.186+0000 7f6f01605700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:26.188 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.186+0000 7f6f01605700 1 -- 172.21.15.59:0/1439181173 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6efc139df0 con 0x7f6efc06c9f0 2023-09-30T02:06:26.188 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.186+0000 7f6f01605700 1 librados: init done 2023-09-30T02:06:26.188 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.186+0000 7f6ee2ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:26.188 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.186+0000 7f6ee2ffd700 1 -- 172.21.15.59:0/1439181173 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef000cd50 con 0x7f6efc06c9f0 2023-09-30T02:06:26.188 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.186+0000 7f6ee2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:26.188 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.186+0000 7f6ee2ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:06:26.188 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.186+0000 7f6ee2ffd700 10 monclient: dump: 2023-09-30T02:06:26.188 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:26.189 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:26.189 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:26.189 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:26.189 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:26.189 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:26.189 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:26.189 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:26.189 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:26.189 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:26.189 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.186+0000 7f6ef97fa700 4 set_mon_vals no callback set 2023-09-30T02:06:26.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.187+0000 7f6f01605700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:26.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.187+0000 7f6f01605700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:26.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.187+0000 7f6f01605700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:26.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.187+0000 7f6f01605700 1 -- 172.21.15.59:0/1439181173 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6edc0050b0 con 0x7f6efc06c9f0 2023-09-30T02:06:26.225 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-09-30T02:06:26.250 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.251+0000 7fa6ddf9f700 10 monclient: get_monmap_and_config 2023-09-30T02:06:26.251 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.251+0000 7fa6ddf9f700 10 monclient: build_initial_monmap 2023-09-30T02:06:26.251 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.251+0000 7fa6ddf9f700 10 monclient: monmap: 2023-09-30T02:06:26.251 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:26.251 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:26.251 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:06:26.251912+0000 2023-09-30T02:06:26.251 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:06:26.251912+0000 2023-09-30T02:06:26.251 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:26.251 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:26.251 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:06:26.252 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:06:26.252 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:06:26.252 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:06:26.252 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:06:26.252 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:26.253 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.252+0000 7fa6ddf9f700 1 Processor -- start 2023-09-30T02:06:26.253 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.252+0000 7fa6ddf9f700 1 -- start start 2023-09-30T02:06:26.253 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.252+0000 7fa6ddf9f700 10 monclient: init 2023-09-30T02:06:26.253 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.253+0000 7fa6ddf9f700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:26.254 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.253+0000 7fa6ddf9f700 10 monclient: _add_conns ranks=[1,3,0,4,2] 2023-09-30T02:06:26.254 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.253+0000 7fa6ddf9f700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa6d80a5b70 addr v1:172.21.15.110:6789/0 2023-09-30T02:06:26.254 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.253+0000 7fa6ddf9f700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa6d8142290 0x7fa6d80b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:26.254 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.253+0000 7fa6ddf9f700 10 monclient(hunting): picked mon.noname-a con 0x7fa6d8142290 addr v2:172.21.15.59:3300/0 2023-09-30T02:06:26.254 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.253+0000 7fa6ddf9f700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa6d80a5f40 addr v1:172.21.15.59:6789/0 2023-09-30T02:06:26.254 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.253+0000 7fa6ddf9f700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6d80a75a0 con 0x7fa6d80a5f40 2023-09-30T02:06:26.254 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.253+0000 7fa6ddf9f700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6d80a6360 con 0x7fa6d80a5b70 2023-09-30T02:06:26.254 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.253+0000 7fa6ddf9f700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:26.254 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.253+0000 7fa6ddf9f700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6d80af3f0 con 0x7fa6d8142290 2023-09-30T02:06:26.255 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.253+0000 7fa6ddf9f700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:26.255 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.253+0000 7fa6ddf9f700 10 monclient(hunting): authenticate will time out at 1831.555305s 2023-09-30T02:06:26.255 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.254+0000 7fa6d6ffd700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa6d8142290 0x7fa6d80b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:26.255 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.254+0000 7fa6d7fff700 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fa6d80a5f40 0x7fa6d813af30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:35134/0 (socket says 172.21.15.59:35134) 2023-09-30T02:06:26.255 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.254+0000 7fa6d7fff700 1 -- 172.21.15.59:0/2286827648 learned_addr learned my addr 172.21.15.59:0/2286827648 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:06:26.256 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.255+0000 7fa6d6ffd700 10 monclient(hunting): get_auth_request con 0x7fa6d8142290 auth_method 0 2023-09-30T02:06:26.256 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.255+0000 7fa6d6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:26.256 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.255+0000 7fa6d6ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:26.256 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.255+0000 7fa6d6ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:26.256 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.256+0000 7fa6d6ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:26.256 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.256+0000 7fa6d6ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:26.256 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.256+0000 7fa6d6ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:26.260 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.260+0000 7fa6d6ffd700 10 monclient(hunting): handle_auth_done global_id 4353 payload 274 2023-09-30T02:06:26.260 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.260+0000 7fa6d6ffd700 1 -- 172.21.15.59:0/2286827648 >> v1:172.21.15.110:6789/0 conn(0x7fa6d80a5b70 legacy=0x7fa6d8141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:06:26.260 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.260+0000 7fa6d6ffd700 1 -- 172.21.15.59:0/2286827648 >> v1:172.21.15.59:6789/0 conn(0x7fa6d80a5f40 legacy=0x7fa6d813af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:26.260 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.260+0000 7fa6d6ffd700 10 monclient: _finish_hunting 0 2023-09-30T02:06:26.260 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.260+0000 7fa6d6ffd700 1 monclient: found mon.noname-a 2023-09-30T02:06:26.260 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.261+0000 7fa6d6ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:26.260 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.261+0000 7fa6d6ffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.59:3300/0 2023-09-30T02:06:26.261 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.261+0000 7fa6d6ffd700 1 -- 172.21.15.59:0/2286827648 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6d813b640 con 0x7fa6d8142290 2023-09-30T02:06:26.265 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.266+0000 7fa6d6ffd700 1 --2- 172.21.15.59:0/2286827648 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa6d8142290 0x7fa6d80b1500 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa6cc0098e0 tx=0x7fa6cc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c5b5eac13dc1284 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:26.272 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.273+0000 7fa6d67fc700 1 -- 172.21.15.59:0/2286827648 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6cc009e80 con 0x7fa6d8142290 2023-09-30T02:06:26.273 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.273+0000 7fa6d67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:26.273 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.273+0000 7fa6d67fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T02:06:26.273 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.273+0000 7fa6d67fc700 10 monclient: dump: 2023-09-30T02:06:26.273 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:26.273 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:26.273 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:26.273 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:26.273 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:26.273 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:26.273 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:26.274 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:26.274 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:26.274 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:26.274 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.273+0000 7fa6d67fc700 10 monclient: _finish_auth 0 2023-09-30T02:06:26.274 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.273+0000 7fa6d67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:26.274 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.273+0000 7fa6d67fc700 1 -- 172.21.15.59:0/2286827648 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6cc00b820 con 0x7fa6d8142290 2023-09-30T02:06:26.274 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.273+0000 7fa6ddf9f700 5 monclient: authenticate success, global_id 4353 2023-09-30T02:06:26.274 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.273+0000 7fa6ddf9f700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:06:26.274 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.273+0000 7fa6d67fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:26.275 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.273+0000 7fa6d67fc700 1 -- 172.21.15.59:0/2286827648 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6cc00bba0 con 0x7fa6d8142290 2023-09-30T02:06:26.275 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.273+0000 7fa6d67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:26.275 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.273+0000 7fa6d67fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T02:06:26.275 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.273+0000 7fa6d67fc700 10 monclient: dump: 2023-09-30T02:06:26.275 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:26.275 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:26.275 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:26.275 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:26.275 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:26.276 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:26.276 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:26.276 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:26.276 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:26.276 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:26.276 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.273+0000 7fa6ddf9f700 10 monclient: get_monmap_and_config success 2023-09-30T02:06:26.276 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.273+0000 7fa6ddf9f700 4 set_mon_vals no callback set 2023-09-30T02:06:26.279 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.274+0000 7fa6ddf9f700 10 monclient: shutdown 2023-09-30T02:06:26.279 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.274+0000 7fa6ddf9f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:26.279 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.274+0000 7fa6ddf9f700 1 -- 172.21.15.59:0/2286827648 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa6d8142290 msgr2=0x7fa6d80b1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:26.279 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.274+0000 7fa6ddf9f700 1 --2- 172.21.15.59:0/2286827648 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa6d8142290 0x7fa6d80b1500 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa6cc0098e0 tx=0x7fa6cc0091a0 comp rx=0 tx=0).stop 2023-09-30T02:06:26.279 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.274+0000 7fa6ddf9f700 1 -- 172.21.15.59:0/2286827648 shutdown_connections 2023-09-30T02:06:26.280 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.274+0000 7fa6ddf9f700 1 --2- 172.21.15.59:0/2286827648 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa6d8142290 0x7fa6d80b1500 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:26.280 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.274+0000 7fa6ddf9f700 1 -- 172.21.15.59:0/2286827648 >> 172.21.15.59:0/2286827648 conn(0x7fa6d80631f0 msgr2=0x7fa6d80635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:26.280 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.274+0000 7fa6ddf9f700 1 -- 172.21.15.59:0/2286827648 shutdown_connections 2023-09-30T02:06:26.280 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.274+0000 7fa6ddf9f700 1 -- 172.21.15.59:0/2286827648 wait complete. 2023-09-30T02:06:26.280 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.275+0000 7fa6ddf9f700 10 monclient: build_initial_monmap 2023-09-30T02:06:26.280 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.275+0000 7fa6ddf9f700 10 monclient: monmap: 2023-09-30T02:06:26.280 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:26.280 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:06:26.280 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:06:26.280 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:06:26.280 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:26.281 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:06:26.281 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:06:26.281 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:06:26.281 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:06:26.281 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:26.281 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.275+0000 7fa6ddf9f700 1 librados: starting msgr at 2023-09-30T02:06:26.281 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.275+0000 7fa6ddf9f700 1 librados: starting objecter 2023-09-30T02:06:26.281 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.275+0000 7fa6ddf9f700 1 Processor -- start 2023-09-30T02:06:26.281 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.276+0000 7fa6ddf9f700 1 -- start start 2023-09-30T02:06:26.281 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.276+0000 7fa6ddf9f700 1 librados: setting wanted keys 2023-09-30T02:06:26.281 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.276+0000 7fa6ddf9f700 1 librados: calling monclient init 2023-09-30T02:06:26.282 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.276+0000 7fa6ddf9f700 10 monclient: init 2023-09-30T02:06:26.282 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.276+0000 7fa6ddf9f700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:26.282 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.276+0000 7fa6ddf9f700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T02:06:26.282 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.276+0000 7fa6ddf9f700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa6d80a5b70 0x7fa6d8151230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:26.282 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.276+0000 7fa6ddf9f700 10 monclient(hunting): picked mon.noname-b con 0x7fa6d80a5b70 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:06:26.282 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.276+0000 7fa6ddf9f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa6d80a5f40 0x7fa6d8151770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:26.282 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.276+0000 7fa6ddf9f700 10 monclient(hunting): picked mon.noname-a con 0x7fa6d80a5f40 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:06:26.282 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.276+0000 7fa6ddf9f700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa6d8142290 0x7fa6d813fe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:26.282 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.276+0000 7fa6ddf9f700 10 monclient(hunting): picked mon.noname-c con 0x7fa6d8142290 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:26.282 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.276+0000 7fa6ddf9f700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:26.282 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.276+0000 7fa6ddf9f700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6d813fa10 con 0x7fa6d80a5f40 2023-09-30T02:06:26.283 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.276+0000 7fa6ddf9f700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:26.283 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.276+0000 7fa6ddf9f700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6d813c030 con 0x7fa6d80a5b70 2023-09-30T02:06:26.283 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.276+0000 7fa6ddf9f700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:26.283 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.276+0000 7fa6ddf9f700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6d8151cb0 con 0x7fa6d8142290 2023-09-30T02:06:26.283 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.276+0000 7fa6ddf9f700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:26.283 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.276+0000 7fa6ddf9f700 10 monclient(hunting): authenticate will time out at 1831.578907s 2023-09-30T02:06:26.283 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.277+0000 7fa6d6ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa6d80a5f40 0x7fa6d8151770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:26.283 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.277+0000 7fa6d7fff700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa6d8142290 0x7fa6d813fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:26.283 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.278+0000 7fa6d6ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa6d80a5f40 0x7fa6d8151770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52182/0 (socket says 172.21.15.59:52182) 2023-09-30T02:06:26.283 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.278+0000 7fa6d6ffd700 1 -- 172.21.15.59:0/939460102 learned_addr learned my addr 172.21.15.59:0/939460102 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:06:26.283 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.278+0000 7fa6d6ffd700 10 monclient(hunting): get_auth_request con 0x7fa6d80a5f40 auth_method 0 2023-09-30T02:06:26.283 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.278+0000 7fa6d6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:26.284 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.278+0000 7fa6d6ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:26.284 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.278+0000 7fa6d6ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:26.284 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.278+0000 7fa6d7fff700 10 monclient(hunting): get_auth_request con 0x7fa6d8142290 auth_method 0 2023-09-30T02:06:26.284 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.278+0000 7fa6d7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:26.284 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.278+0000 7fa6d7fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:26.284 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.278+0000 7fa6d7fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:26.284 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.278+0000 7fa6d6ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:26.284 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.278+0000 7fa6d6ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:26.284 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.278+0000 7fa6d6ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:26.284 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.279+0000 7fa6d7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:26.284 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.279+0000 7fa6d7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:26.284 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.279+0000 7fa6d7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:26.285 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.282+0000 7fa6d6ffd700 10 monclient(hunting): handle_auth_done global_id 4359 payload 824 2023-09-30T02:06:26.286 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.282+0000 7fa6d6ffd700 1 -- 172.21.15.59:0/939460102 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa6d8142290 msgr2=0x7fa6d813fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:26.286 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.282+0000 7fa6d6ffd700 1 --2- 172.21.15.59:0/939460102 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa6d8142290 0x7fa6d813fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:26.286 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.282+0000 7fa6d6ffd700 1 -- 172.21.15.59:0/939460102 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa6d80a5b70 msgr2=0x7fa6d8151230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:26.286 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.282+0000 7fa6d6ffd700 1 --2- 172.21.15.59:0/939460102 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa6d80a5b70 0x7fa6d8151230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:26.286 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.282+0000 7fa6d6ffd700 10 monclient: _finish_hunting 0 2023-09-30T02:06:26.286 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.282+0000 7fa6d6ffd700 1 monclient: found mon.noname-a 2023-09-30T02:06:26.286 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.282+0000 7fa6d6ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:26.286 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.282+0000 7fa6d6ffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.59:3300/0 2023-09-30T02:06:26.286 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.282+0000 7fa6d6ffd700 1 -- 172.21.15.59:0/939460102 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6cc009420 con 0x7fa6d80a5f40 2023-09-30T02:06:26.286 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.285+0000 7fa6d6ffd700 1 --2- 172.21.15.59:0/939460102 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa6d80a5f40 0x7fa6d8151770 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa6cc0030c0 tx=0x7fa6cc016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:26.286 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.285+0000 7fa6bf7fe700 10 client.?.objecter ms_handle_connect 0x7fa6d80a5f40 2023-09-30T02:06:26.287 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.285+0000 7fa6bf7fe700 10 client.?.objecter resend_mon_ops 2023-09-30T02:06:26.290 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.290+0000 7fa6bf7fe700 1 -- 172.21.15.59:0/939460102 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6cc00ba60 con 0x7fa6d80a5f40 2023-09-30T02:06:26.290 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.290+0000 7fa6bf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:26.290 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.290+0000 7fa6bf7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T02:06:26.290 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.290+0000 7fa6bf7fe700 10 monclient: dump: 2023-09-30T02:06:26.290 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:26.290 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:26.290 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:26.290 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:26.290 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:26.291 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:26.291 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:26.291 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:26.291 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:26.291 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:26.291 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.290+0000 7fa6bf7fe700 10 monclient: _finish_auth 0 2023-09-30T02:06:26.291 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.290+0000 7fa6bf7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:26.291 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.290+0000 7fa6bf7fe700 1 -- 172.21.15.59:0/939460102 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6cc020070 con 0x7fa6d80a5f40 2023-09-30T02:06:26.291 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.290+0000 7fa6bf7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:26.291 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.290+0000 7fa6bf7fe700 1 -- 172.21.15.59:0/939460102 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6cc00bbe0 con 0x7fa6d80a5f40 2023-09-30T02:06:26.291 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.290+0000 7fa6bf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:26.291 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.290+0000 7fa6bf7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T02:06:26.292 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.290+0000 7fa6bf7fe700 10 monclient: dump: 2023-09-30T02:06:26.292 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:26.292 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:26.292 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:26.292 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:26.292 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:26.292 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:26.292 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:26.292 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:26.292 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:26.292 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:26.293 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.290+0000 7fa6ddf9f700 5 monclient: authenticate success, global_id 4359 2023-09-30T02:06:26.293 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.290+0000 7fa6ddf9f700 10 monclient: _renew_subs 2023-09-30T02:06:26.293 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.290+0000 7fa6ddf9f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:06:26.293 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.290+0000 7fa6ddf9f700 1 -- 172.21.15.59:0/939460102 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6d81403a0 con 0x7fa6d80a5f40 2023-09-30T02:06:26.293 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.290+0000 7fa6d5ffb700 4 set_mon_vals no callback set 2023-09-30T02:06:26.293 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.290+0000 7fa6ddf9f700 10 client.4359.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:06:26.293 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.290+0000 7fa6ddf9f700 10 monclient: _renew_subs 2023-09-30T02:06:26.293 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.290+0000 7fa6ddf9f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:06:26.293 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.290+0000 7fa6ddf9f700 1 -- 172.21.15.59:0/939460102 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6d8140780 con 0x7fa6d80a5f40 2023-09-30T02:06:26.293 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.290+0000 7fa6ddf9f700 1 librados: init done 2023-09-30T02:06:26.294 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.291+0000 7fa6ddf9f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:26.294 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.291+0000 7fa6ddf9f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:26.294 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.291+0000 7fa6ddf9f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:06:26.294 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.291+0000 7fa6ddf9f700 1 -- 172.21.15.59:0/939460102 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6d8151e70 con 0x7fa6d80a5f40 2023-09-30T02:06:26.306 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.307+0000 7f3fbf625700 10 monclient: get_monmap_and_config 2023-09-30T02:06:26.306 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.307+0000 7f3fbf625700 10 monclient: build_initial_monmap 2023-09-30T02:06:26.306 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.307+0000 7f3fbf625700 10 monclient: monmap: 2023-09-30T02:06:26.306 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:26.307 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:26.307 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:06:26.307470+0000 2023-09-30T02:06:26.307 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:06:26.307470+0000 2023-09-30T02:06:26.307 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:26.307 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:26.307 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:06:26.307 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:06:26.307 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:06:26.307 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:06:26.307 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:06:26.307 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:26.308 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.307+0000 7f3fbf625700 1 Processor -- start 2023-09-30T02:06:26.308 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.308+0000 7f3fbf625700 1 -- start start 2023-09-30T02:06:26.309 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.308+0000 7f3fbf625700 10 monclient: init 2023-09-30T02:06:26.309 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.308+0000 7f3fbf625700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:26.309 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.308+0000 7f3fbf625700 10 monclient: _add_conns ranks=[3,1,0,2,4] 2023-09-30T02:06:26.309 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.308+0000 7f3fbf625700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f3fb80a5b70 0x7f3fb80b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:26.309 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.308+0000 7f3fbf625700 10 monclient(hunting): picked mon.noname-a con 0x7f3fb80a5b70 addr v2:172.21.15.59:3300/0 2023-09-30T02:06:26.309 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.308+0000 7f3fbf625700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3fb80a5f40 addr v1:172.21.15.110:6789/0 2023-09-30T02:06:26.309 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.308+0000 7f3fbf625700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3fb8142290 addr v1:172.21.15.59:6789/0 2023-09-30T02:06:26.309 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.308+0000 7f3fbf625700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fb80a75a0 con 0x7f3fb8142290 2023-09-30T02:06:26.309 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.308+0000 7f3fbf625700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fb80af3f0 con 0x7f3fb80a5f40 2023-09-30T02:06:26.309 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.308+0000 7f3fbf625700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:26.310 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.308+0000 7f3fbf625700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fb80a63e0 con 0x7f3fb80a5b70 2023-09-30T02:06:26.310 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.308+0000 7f3fbf625700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:26.310 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.308+0000 7f3fbf625700 10 monclient(hunting): authenticate will time out at 1831.610788s 2023-09-30T02:06:26.324 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.324+0000 7fa6bf7fe700 1 -- 172.21.15.59:0/939460102 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fa6cc022030 con 0x7fa6d80a5f40 2023-09-30T02:06:26.324 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.324+0000 7fa6bf7fe700 1 --2- 172.21.15.59:0/939460102 >> v2:172.21.15.110:6800/115870 conn(0x7fa6c4068960 0x7fa6c406ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:26.324 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.325+0000 7fa6bf7fe700 1 -- 172.21.15.59:0/939460102 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fa6cc086150 con 0x7fa6d80a5f40 2023-09-30T02:06:26.324 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.325+0000 7fa6bf7fe700 10 client.4359.objecter ms_dispatch 0x7fa6d8001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:06:26.325 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.325+0000 7fa6bf7fe700 3 client.4359.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:06:26.325 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.325+0000 7fa6bf7fe700 3 client.4359.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:06:26.325 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.325+0000 7fa6d77fe700 1 --2- 172.21.15.59:0/939460102 >> v2:172.21.15.110:6800/115870 conn(0x7fa6c4068960 0x7fa6c406ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:26.325 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.325+0000 7fa6bf7fe700 20 client.4359.objecter dump_active .. 0 homeless 2023-09-30T02:06:26.325 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.325+0000 7fa6d77fe700 10 monclient: get_auth_request con 0x7fa6c4068960 auth_method 0 2023-09-30T02:06:26.325 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.325+0000 7fa6bf7fe700 10 client.4359.objecter ms_handle_connect 0x7fa6c4068960 2023-09-30T02:06:26.325 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.325+0000 7fa6d77fe700 1 --2- 172.21.15.59:0/939460102 >> v2:172.21.15.110:6800/115870 conn(0x7fa6c4068960 0x7fa6c406ae10 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa6c800ad10 tx=0x7fa6c8009370 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:26.365 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.366+0000 7f1cb29c0700 10 monclient: get_monmap_and_config 2023-09-30T02:06:26.365 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.366+0000 7f1cb29c0700 10 monclient: build_initial_monmap 2023-09-30T02:06:26.366 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.366+0000 7f1cb29c0700 10 monclient: monmap: 2023-09-30T02:06:26.366 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:26.366 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:26.366 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:06:26.366683+0000 2023-09-30T02:06:26.366 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:06:26.366683+0000 2023-09-30T02:06:26.366 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:26.366 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:26.367 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:06:26.367 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:06:26.367 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:06:26.367 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:06:26.367 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:06:26.367 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:26.367 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-09-30T02:06:26.369 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.367+0000 7f1cb29c0700 1 Processor -- start 2023-09-30T02:06:26.369 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.367+0000 7f1cb29c0700 1 -- start start 2023-09-30T02:06:26.369 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.367+0000 7f1cb29c0700 10 monclient: init 2023-09-30T02:06:26.369 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.367+0000 7f1cb29c0700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:26.369 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.367+0000 7f1cb29c0700 10 monclient: _add_conns ranks=[0,3,1,4,2] 2023-09-30T02:06:26.369 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.367+0000 7f1cb29c0700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1cac0a5b70 addr v1:172.21.15.59:6789/0 2023-09-30T02:06:26.370 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.367+0000 7f1cb29c0700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f1cac142290 0x7f1cac0b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:26.370 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.367+0000 7f1cb29c0700 10 monclient(hunting): picked mon.noname-a con 0x7f1cac142290 addr v2:172.21.15.59:3300/0 2023-09-30T02:06:26.370 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.367+0000 7f1cb29c0700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1cac0a5f40 addr v1:172.21.15.110:6789/0 2023-09-30T02:06:26.370 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.367+0000 7f1cb29c0700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cac0a75a0 con 0x7f1cac0a5b70 2023-09-30T02:06:26.370 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.367+0000 7f1cb29c0700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cac0a6360 con 0x7f1cac0a5f40 2023-09-30T02:06:26.370 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.367+0000 7f1cb29c0700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:26.370 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.367+0000 7f1cb29c0700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cac0af3f0 con 0x7f1cac142290 2023-09-30T02:06:26.370 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.367+0000 7f1cb29c0700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:26.371 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.367+0000 7f1cb29c0700 10 monclient(hunting): authenticate will time out at 1831.669970s 2023-09-30T02:06:26.440 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.440+0000 7f6ee2ffd700 1 -- 172.21.15.59:0/1439181173 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f6ef0013680 con 0x7f6efc06c9f0 2023-09-30T02:06:26.440 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.440+0000 7f6ee2ffd700 1 --2- 172.21.15.59:0/1439181173 >> v2:172.21.15.110:6800/115870 conn(0x7f6ee80688c0 0x7f6ee806ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:26.440 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.440+0000 7f6ee2ffd700 1 -- 172.21.15.59:0/1439181173 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f6ef00848a0 con 0x7f6efc06c9f0 2023-09-30T02:06:26.441 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.440+0000 7f6ee2ffd700 10 client.4387.objecter ms_dispatch 0x7f6efc001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:06:26.441 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.440+0000 7f6ee2ffd700 3 client.4387.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:06:26.441 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.440+0000 7f6ee2ffd700 3 client.4387.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:06:26.441 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.441+0000 7f6ee2ffd700 20 client.4387.objecter dump_active .. 0 homeless 2023-09-30T02:06:26.441 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.441+0000 7f6efaffd700 1 --2- 172.21.15.59:0/1439181173 >> v2:172.21.15.110:6800/115870 conn(0x7f6ee80688c0 0x7f6ee806ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:26.441 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.441+0000 7f6efaffd700 10 monclient: get_auth_request con 0x7f6ee80688c0 auth_method 0 2023-09-30T02:06:26.442 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.441+0000 7f6ee2ffd700 10 client.4387.objecter ms_handle_connect 0x7f6ee80688c0 2023-09-30T02:06:26.442 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.441+0000 7f6efaffd700 1 --2- 172.21.15.59:0/1439181173 >> v2:172.21.15.110:6800/115870 conn(0x7f6ee80688c0 0x7f6ee806ad70 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6eec0099b0 tx=0x7f6eec017000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:26.446 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-09-30T02:06:26.449 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.449+0000 7f6ee2ffd700 1 -- 172.21.15.59:0/1439181173 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6ef0052140 con 0x7f6efc06c9f0 2023-09-30T02:06:26.450 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.449+0000 7f6ee2ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:26.450 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.449+0000 7f6ee2ffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:06:26.450 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.451+0000 7f1cb0f5d700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f1cac0a5f40 0x7f1cac13af30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.59:54938/0 (socket says 172.21.15.59:54938) 2023-09-30T02:06:26.450 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.451+0000 7f1cb0f5d700 1 -- 172.21.15.59:0/1126432236 learned_addr learned my addr 172.21.15.59:0/1126432236 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:06:26.451 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.452+0000 7f3fbcbc0700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f3fb80a5f40 0x7f3fb8141b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.59:54928/0 (socket says 172.21.15.59:54928) 2023-09-30T02:06:26.451 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.452+0000 7f3fbcbc0700 1 -- 172.21.15.59:0/642111424 learned_addr learned my addr 172.21.15.59:0/642111424 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:06:26.520 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.521+0000 7f4fe81db700 10 monclient: get_monmap_and_config 2023-09-30T02:06:26.521 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.521+0000 7f4fe81db700 10 monclient: build_initial_monmap 2023-09-30T02:06:26.521 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.521+0000 7f4fe81db700 10 monclient: monmap: 2023-09-30T02:06:26.521 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:26.521 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:26.521 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:06:26.521628+0000 2023-09-30T02:06:26.521 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:06:26.521628+0000 2023-09-30T02:06:26.521 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:26.521 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:26.521 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:06:26.521 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:06:26.522 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:06:26.522 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:06:26.522 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:06:26.522 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:26.523 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.521+0000 7f4fe81db700 1 Processor -- start 2023-09-30T02:06:26.523 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.522+0000 7f4fe81db700 1 -- start start 2023-09-30T02:06:26.523 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.522+0000 7f4fe81db700 10 monclient: init 2023-09-30T02:06:26.523 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.522+0000 7f4fe81db700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:26.523 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.522+0000 7f4fe81db700 10 monclient: _add_conns ranks=[3,4,1,2,0] 2023-09-30T02:06:26.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.522+0000 7f4fe81db700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f4fe00a5b70 0x7f4fe00b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:26.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.522+0000 7f4fe81db700 10 monclient(hunting): picked mon.noname-a con 0x7f4fe00a5b70 addr v2:172.21.15.59:3300/0 2023-09-30T02:06:26.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.522+0000 7f4fe81db700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4fe00a5f40 0x7f4fe0141b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:26.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.522+0000 7f4fe81db700 10 monclient(hunting): picked mon.noname-c con 0x7f4fe00a5f40 addr v2:172.21.15.110:3300/0 2023-09-30T02:06:26.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.522+0000 7f4fe81db700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4fe01420c0 addr v1:172.21.15.110:6789/0 2023-09-30T02:06:26.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.522+0000 7f4fe81db700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fe00a75a0 con 0x7f4fe01420c0 2023-09-30T02:06:26.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.522+0000 7f4fe81db700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:26.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.522+0000 7f4fe81db700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fe00a63e0 con 0x7f4fe00a5b70 2023-09-30T02:06:26.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.522+0000 7f4fe81db700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:26.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.522+0000 7f4fe81db700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fe00af3f0 con 0x7f4fe00a5f40 2023-09-30T02:06:26.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.522+0000 7f4fe81db700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:26.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.522+0000 7f4fe81db700 10 monclient(hunting): authenticate will time out at 1831.824921s 2023-09-30T02:06:26.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.523+0000 7f4fe5776700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4fe00a5f40 0x7f4fe0141b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:26.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.524+0000 7f4fe5776700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f4fe00a5f40 0x7f4fe0141b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:39250/0 (socket says 172.21.15.59:39250) 2023-09-30T02:06:26.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.524+0000 7f4fe5776700 1 -- 172.21.15.59:0/1754539642 learned_addr learned my addr 172.21.15.59:0/1754539642 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:06:26.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.524+0000 7f4fe5776700 10 monclient(hunting): get_auth_request con 0x7f4fe00a5f40 auth_method 0 2023-09-30T02:06:26.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.524+0000 7f4fe5776700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:26.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.524+0000 7f4fe5776700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:26.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.524+0000 7f4fe5776700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:26.526 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.525+0000 7f4fe5776700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:26.526 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.525+0000 7f4fe5776700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:26.526 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.525+0000 7f4fe5776700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:26.526 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.527+0000 7f4fe5776700 10 monclient(hunting): handle_auth_done global_id 4390 payload 274 2023-09-30T02:06:26.526 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.527+0000 7f4fe5776700 1 -- 172.21.15.59:0/1754539642 >> v2:172.21.15.59:3300/0 conn(0x7f4fe00a5b70 msgr2=0x7f4fe00b1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:06:26.527 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.527+0000 7f4fe5776700 1 --2- 172.21.15.59:0/1754539642 >> v2:172.21.15.59:3300/0 conn(0x7f4fe00a5b70 0x7f4fe00b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:26.527 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.527+0000 7f4fe5776700 1 -- 172.21.15.59:0/1754539642 >> v1:172.21.15.110:6789/0 conn(0x7f4fe01420c0 legacy=0x7f4fe013e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:26.527 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.527+0000 7f4fe5776700 10 monclient: _finish_hunting 0 2023-09-30T02:06:26.527 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.527+0000 7f4fe5776700 1 monclient: found mon.noname-c 2023-09-30T02:06:26.527 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.527+0000 7f4fe5776700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:26.527 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.527+0000 7f4fe5776700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3300/0 2023-09-30T02:06:26.527 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.527+0000 7f4fe5776700 1 -- 172.21.15.59:0/1754539642 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fe01424e0 con 0x7f4fe00a5f40 2023-09-30T02:06:26.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.531+0000 7f4fe5776700 1 --2- 172.21.15.59:0/1754539642 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4fe00a5f40 0x7f4fe0141b80 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f4fe00b1fc0 tx=0x7f4fd0009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=29d5584943d98e4f server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:26.535 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.535+0000 7f4fe4f75700 1 -- 172.21.15.59:0/1754539642 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fd000b860 con 0x7f4fe00a5f40 2023-09-30T02:06:26.535 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.535+0000 7f4fe4f75700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:26.535 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.535+0000 7f4fe4f75700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:06:26.535 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.535+0000 7f4fe4f75700 10 monclient: dump: 2023-09-30T02:06:26.535 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:26.535 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:26.535 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:26.535 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:26.535 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:26.536 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:26.536 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:26.536 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:26.536 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:26.536 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:26.536 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.535+0000 7f4fe4f75700 10 monclient: _finish_auth 0 2023-09-30T02:06:26.536 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.535+0000 7f4fe4f75700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:26.536 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.535+0000 7f4fe4f75700 1 -- 172.21.15.59:0/1754539642 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fd000b9c0 con 0x7f4fe00a5f40 2023-09-30T02:06:26.536 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.535+0000 7f4fe4f75700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:26.536 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.535+0000 7f4fe4f75700 1 -- 172.21.15.59:0/1754539642 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fd000bd60 con 0x7f4fe00a5f40 2023-09-30T02:06:26.537 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.535+0000 7f4fe81db700 5 monclient: authenticate success, global_id 4390 2023-09-30T02:06:26.537 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.535+0000 7f4fe81db700 10 monclient: get_monmap_and_config success 2023-09-30T02:06:26.537 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.535+0000 7f4fe81db700 4 set_mon_vals no callback set 2023-09-30T02:06:26.537 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.535+0000 7f4fe4f75700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:26.537 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.535+0000 7f4fe4f75700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:06:26.537 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.535+0000 7f4fe4f75700 10 monclient: dump: 2023-09-30T02:06:26.537 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:26.537 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:26.537 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:26.537 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:26.538 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:26.538 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:26.538 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:26.538 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:26.538 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:26.538 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:26.540 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.536+0000 7f4fe81db700 10 monclient: shutdown 2023-09-30T02:06:26.541 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.536+0000 7f4fe81db700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:26.541 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.536+0000 7f4fe81db700 1 -- 172.21.15.59:0/1754539642 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4fe00a5f40 msgr2=0x7f4fe0141b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:26.541 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.536+0000 7f4fe81db700 1 --2- 172.21.15.59:0/1754539642 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4fe00a5f40 0x7f4fe0141b80 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f4fe00b1fc0 tx=0x7f4fd0009250 comp rx=0 tx=0).stop 2023-09-30T02:06:26.541 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.536+0000 7f4fe81db700 1 -- 172.21.15.59:0/1754539642 shutdown_connections 2023-09-30T02:06:26.541 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.536+0000 7f4fe81db700 1 --2- 172.21.15.59:0/1754539642 >> v2:172.21.15.59:3300/0 conn(0x7f4fe00a5b70 0x7f4fe00b1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:26.541 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.536+0000 7f4fe81db700 1 --2- 172.21.15.59:0/1754539642 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4fe00a5f40 0x7f4fe0141b80 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:26.541 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.536+0000 7f4fe81db700 1 -- 172.21.15.59:0/1754539642 >> 172.21.15.59:0/1754539642 conn(0x7f4fe00631f0 msgr2=0x7f4fe00635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:26.541 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.536+0000 7f4fe81db700 1 -- 172.21.15.59:0/1754539642 shutdown_connections 2023-09-30T02:06:26.541 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.536+0000 7f4fe81db700 1 -- 172.21.15.59:0/1754539642 wait complete. 2023-09-30T02:06:26.542 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.537+0000 7f4fe81db700 10 monclient: build_initial_monmap 2023-09-30T02:06:26.542 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.537+0000 7f4fe81db700 10 monclient: monmap: 2023-09-30T02:06:26.542 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:26.542 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:06:26.542 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:06:26.542 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:06:26.542 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:26.542 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:06:26.542 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:06:26.543 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:06:26.543 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:06:26.543 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:26.543 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.537+0000 7f4fe81db700 1 librados: starting msgr at 2023-09-30T02:06:26.543 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.537+0000 7f4fe81db700 1 librados: starting objecter 2023-09-30T02:06:26.543 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.537+0000 7f4fe81db700 1 Processor -- start 2023-09-30T02:06:26.543 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.537+0000 7f4fe81db700 1 -- start start 2023-09-30T02:06:26.543 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.537+0000 7f4fe81db700 1 librados: setting wanted keys 2023-09-30T02:06:26.543 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.537+0000 7f4fe81db700 1 librados: calling monclient init 2023-09-30T02:06:26.543 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.537+0000 7f4fe81db700 10 monclient: init 2023-09-30T02:06:26.543 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.538+0000 7f4fe81db700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:26.543 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.538+0000 7f4fe81db700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T02:06:26.544 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.538+0000 7f4fe81db700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4fe00a5b70 0x7f4fe013ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:26.544 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.538+0000 7f4fe81db700 10 monclient(hunting): picked mon.noname-c con 0x7f4fe00a5b70 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:26.544 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.538+0000 7f4fe81db700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4fe00a5f40 0x7f4fe0135560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:26.544 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.538+0000 7f4fe81db700 10 monclient(hunting): picked mon.noname-b con 0x7f4fe00a5f40 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:06:26.544 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.538+0000 7f4fe81db700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4fe01420c0 0x7f4fe01347c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:26.544 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.538+0000 7f4fe81db700 10 monclient(hunting): picked mon.noname-a con 0x7f4fe01420c0 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:06:26.544 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.538+0000 7f4fe81db700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:26.544 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.538+0000 7f4fe81db700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fe013d530 con 0x7f4fe01420c0 2023-09-30T02:06:26.544 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.538+0000 7f4fe81db700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:26.544 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.538+0000 7f4fe81db700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fe013a0a0 con 0x7f4fe00a5f40 2023-09-30T02:06:26.544 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.538+0000 7f4fe81db700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:26.544 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.538+0000 7f4fe81db700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fe0134d00 con 0x7f4fe00a5b70 2023-09-30T02:06:26.545 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.538+0000 7f4fe81db700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:26.545 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.538+0000 7f4fe81db700 10 monclient(hunting): authenticate will time out at 1831.840905s 2023-09-30T02:06:26.545 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.539+0000 7f4fe5776700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4fe00a5f40 0x7f4fe0135560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:26.545 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.539+0000 7f4fe5776700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4fe00a5f40 0x7f4fe0135560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:39266/0 (socket says 172.21.15.59:39266) 2023-09-30T02:06:26.545 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.539+0000 7f4fe5776700 1 -- 172.21.15.59:0/495107266 learned_addr learned my addr 172.21.15.59:0/495107266 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:06:26.545 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.539+0000 7f4fe5776700 10 monclient(hunting): get_auth_request con 0x7f4fe00a5f40 auth_method 0 2023-09-30T02:06:26.545 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.539+0000 7f4fe5776700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:26.545 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.539+0000 7f4fe5776700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:26.545 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.539+0000 7f4fe5776700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:26.545 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.540+0000 7f4fe5776700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:26.546 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.540+0000 7f4fe5776700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:26.546 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.540+0000 7f4fe5776700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:26.546 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.543+0000 7f4fe5776700 10 monclient(hunting): handle_auth_done global_id 4393 payload 824 2023-09-30T02:06:26.547 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.543+0000 7f4fe5776700 1 -- 172.21.15.59:0/495107266 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4fe00a5b70 msgr2=0x7f4fe013ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:26.547 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.543+0000 7f4fe5776700 1 --2- 172.21.15.59:0/495107266 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4fe00a5b70 0x7f4fe013ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:26.547 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.543+0000 7f4fe5776700 1 -- 172.21.15.59:0/495107266 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4fe01420c0 msgr2=0x7f4fe01347c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:26.547 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.543+0000 7f4fe5776700 1 --2- 172.21.15.59:0/495107266 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4fe01420c0 0x7f4fe01347c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:26.547 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.543+0000 7f4fe5776700 10 monclient: _finish_hunting 0 2023-09-30T02:06:26.547 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.543+0000 7f4fe5776700 1 monclient: found mon.noname-b 2023-09-30T02:06:26.547 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.543+0000 7f4fe5776700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:26.547 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.543+0000 7f4fe5776700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:06:26.547 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.543+0000 7f4fe5776700 1 -- 172.21.15.59:0/495107266 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fd00094f0 con 0x7f4fe00a5f40 2023-09-30T02:06:26.547 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.546+0000 7f4fe5776700 1 --2- 172.21.15.59:0/495107266 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4fe00a5f40 0x7f4fe0135560 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4fd0009290 tx=0x7f4fd0017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:26.548 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.546+0000 7f4fd5ffb700 10 client.?.objecter ms_handle_connect 0x7f4fe00a5f40 2023-09-30T02:06:26.548 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.546+0000 7f4fd5ffb700 10 client.?.objecter resend_mon_ops 2023-09-30T02:06:26.549 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.550+0000 7f4fd5ffb700 1 -- 172.21.15.59:0/495107266 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fd0011e60 con 0x7f4fe00a5f40 2023-09-30T02:06:26.549 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.550+0000 7f4fd5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:26.550 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.550+0000 7f4fd5ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:06:26.550 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.550+0000 7f4fd5ffb700 10 monclient: dump: 2023-09-30T02:06:26.550 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:26.550 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:26.550 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:26.550 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:26.550 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:26.550 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:26.550 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:26.550 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:26.551 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:26.551 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:26.551 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.550+0000 7f4fd5ffb700 10 monclient: _finish_auth 0 2023-09-30T02:06:26.551 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.550+0000 7f4fd5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:26.551 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.550+0000 7f4fd5ffb700 1 -- 172.21.15.59:0/495107266 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fd001a660 con 0x7f4fe00a5f40 2023-09-30T02:06:26.551 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.550+0000 7f4fe81db700 5 monclient: authenticate success, global_id 4393 2023-09-30T02:06:26.551 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.550+0000 7f4fe81db700 10 monclient: _renew_subs 2023-09-30T02:06:26.551 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.550+0000 7f4fe81db700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:26.551 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.550+0000 7f4fe81db700 1 -- 172.21.15.59:0/495107266 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fe013c3e0 con 0x7f4fe00a5f40 2023-09-30T02:06:26.551 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.550+0000 7f4fd5ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:26.551 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.550+0000 7f4fd5ffb700 1 -- 172.21.15.59:0/495107266 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fd001aa20 con 0x7f4fe00a5f40 2023-09-30T02:06:26.552 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.550+0000 7f4fd5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:26.552 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.550+0000 7f4fd5ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:06:26.552 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.550+0000 7f4fd5ffb700 10 monclient: dump: 2023-09-30T02:06:26.552 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:26.552 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:26.552 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:26.552 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:26.552 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:26.552 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:26.552 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:26.552 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:26.552 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:26.553 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:26.553 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.550+0000 7f4fd7fff700 4 set_mon_vals no callback set 2023-09-30T02:06:26.553 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.550+0000 7f4fe81db700 10 client.4393.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:06:26.554 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.550+0000 7f4fe81db700 10 monclient: _renew_subs 2023-09-30T02:06:26.554 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.550+0000 7f4fe81db700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:26.554 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.550+0000 7f4fe81db700 1 -- 172.21.15.59:0/495107266 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fe013bcb0 con 0x7f4fe00a5f40 2023-09-30T02:06:26.554 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.550+0000 7f4fe81db700 1 librados: init done 2023-09-30T02:06:26.554 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.552+0000 7f4fe81db700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:26.554 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.552+0000 7f4fe81db700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:26.554 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.552+0000 7f4fe81db700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:26.554 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.552+0000 7f4fe81db700 1 -- 172.21.15.59:0/495107266 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fe013d160 con 0x7f4fe00a5f40 2023-09-30T02:06:26.589 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.590+0000 7f5bc8b6d700 10 monclient: get_monmap_and_config 2023-09-30T02:06:26.590 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.590+0000 7f5bc8b6d700 10 monclient: build_initial_monmap 2023-09-30T02:06:26.590 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.590+0000 7f5bc8b6d700 10 monclient: monmap: 2023-09-30T02:06:26.590 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:26.590 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:26.590 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:06:26.590703+0000 2023-09-30T02:06:26.590 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:06:26.590703+0000 2023-09-30T02:06:26.590 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:26.590 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:26.590 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:06:26.590 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:06:26.590 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:06:26.590 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:06:26.591 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:06:26.591 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:26.591 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.590+0000 7f5bc8b6d700 1 Processor -- start 2023-09-30T02:06:26.592 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.591+0000 7f5bc8b6d700 1 -- start start 2023-09-30T02:06:26.592 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.591+0000 7f5bc8b6d700 10 monclient: init 2023-09-30T02:06:26.592 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.592+0000 7f5bc8b6d700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:26.592 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.592+0000 7f5bc8b6d700 10 monclient: _add_conns ranks=[1,2,0,3,4] 2023-09-30T02:06:26.592 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.592+0000 7f5bc8b6d700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5bc40a5b70 addr v1:172.21.15.110:6789/0 2023-09-30T02:06:26.592 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.592+0000 7f5bc8b6d700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5bc4142290 0x7f5bc40b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:26.592 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.592+0000 7f5bc8b6d700 10 monclient(hunting): picked mon.noname-b con 0x7f5bc4142290 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:26.592 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.592+0000 7f5bc8b6d700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5bc40a5f40 addr v1:172.21.15.59:6789/0 2023-09-30T02:06:26.592 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.592+0000 7f5bc8b6d700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bc40a6380 con 0x7f5bc40a5f40 2023-09-30T02:06:26.592 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.592+0000 7f5bc8b6d700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bc40a75a0 con 0x7f5bc40a5b70 2023-09-30T02:06:26.593 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.592+0000 7f5bc8b6d700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:26.593 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.592+0000 7f5bc8b6d700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc40af3f0 con 0x7f5bc4142290 2023-09-30T02:06:26.593 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.592+0000 7f5bc8b6d700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:26.593 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.592+0000 7f5bc8b6d700 10 monclient(hunting): authenticate will time out at 1831.894457s 2023-09-30T02:06:26.593 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.593+0000 7f5bc1d9b700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5bc4142290 0x7f5bc40b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:26.594 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.593+0000 7f5bc1d9b700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5bc4142290 0x7f5bc40b1500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:47890/0 (socket says 172.21.15.59:47890) 2023-09-30T02:06:26.594 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.593+0000 7f5bc1d9b700 1 -- 172.21.15.59:0/1633275500 learned_addr learned my addr 172.21.15.59:0/1633275500 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:06:26.594 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.593+0000 7f5bc1d9b700 10 monclient(hunting): get_auth_request con 0x7f5bc4142290 auth_method 0 2023-09-30T02:06:26.594 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.593+0000 7f5bc1d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:26.594 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.593+0000 7f5bc1d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:26.594 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.593+0000 7f5bc1d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:26.594 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.594+0000 7f5bc1d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:26.594 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.594+0000 7f5bc1d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:26.594 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.594+0000 7f5bc1d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:26.595 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.596+0000 7f5bc1d9b700 10 monclient(hunting): handle_auth_done global_id 4340 payload 274 2023-09-30T02:06:26.595 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.596+0000 7f5bc1d9b700 1 -- 172.21.15.59:0/1633275500 >> v1:172.21.15.110:6789/0 conn(0x7f5bc40a5b70 legacy=0x7f5bc4141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:06:26.595 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.596+0000 7f5bc1d9b700 1 -- 172.21.15.59:0/1633275500 >> v1:172.21.15.59:6789/0 conn(0x7f5bc40a5f40 legacy=0x7f5bc413af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:06:26.595 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.596+0000 7f5bc1d9b700 10 monclient: _finish_hunting 0 2023-09-30T02:06:26.595 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.596+0000 7f5bc1d9b700 1 monclient: found mon.noname-b 2023-09-30T02:06:26.596 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.596+0000 7f5bc1d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:26.596 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.596+0000 7f5bc1d9b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.59:3301/0 2023-09-30T02:06:26.596 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.596+0000 7f5bc1d9b700 1 -- 172.21.15.59:0/1633275500 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bc413b640 con 0x7f5bc4142290 2023-09-30T02:06:26.596 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.597+0000 7f3fbd3c1700 1 --2- 172.21.15.59:0/642111424 >> v2:172.21.15.59:3300/0 conn(0x7f3fb80a5b70 0x7f3fb80b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:26.597 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.598+0000 7f5bc1d9b700 1 --2- 172.21.15.59:0/1633275500 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5bc4142290 0x7f5bc40b1500 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5bb80099e0 tx=0x7f5bb8009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ded7f2a26b98bf8b server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:26.598 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.599+0000 7f3fbd3c1700 10 monclient(hunting): get_auth_request con 0x7f3fb80a5b70 auth_method 0 2023-09-30T02:06:26.599 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.599+0000 7f3fbd3c1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:26.599 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.599+0000 7f3fbd3c1700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:26.599 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.599+0000 7f3fbd3c1700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:26.603 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.604+0000 7f5bc159a700 1 -- 172.21.15.59:0/1633275500 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bb800b7e0 con 0x7f5bc4142290 2023-09-30T02:06:26.604 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.604+0000 7f5bc159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:26.604 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.604+0000 7f5bc159a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:06:26.604 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.604+0000 7f5bc159a700 10 monclient: dump: 2023-09-30T02:06:26.604 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:26.604 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:26.604 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:26.604 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:26.604 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:26.604 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:26.604 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:26.604 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:26.605 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:26.605 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:26.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.604+0000 7f5bc159a700 10 monclient: _finish_auth 0 2023-09-30T02:06:26.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.604+0000 7f5bc159a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:26.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.604+0000 7f5bc159a700 1 -- 172.21.15.59:0/1633275500 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bb800b940 con 0x7f5bc4142290 2023-09-30T02:06:26.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.604+0000 7f5bc8b6d700 5 monclient: authenticate success, global_id 4340 2023-09-30T02:06:26.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.604+0000 7f5bc159a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:26.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.604+0000 7f5bc8b6d700 10 monclient: get_monmap_and_config success 2023-09-30T02:06:26.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.604+0000 7f5bc8b6d700 4 set_mon_vals no callback set 2023-09-30T02:06:26.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.604+0000 7f5bc159a700 1 -- 172.21.15.59:0/1633275500 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bb800bce0 con 0x7f5bc4142290 2023-09-30T02:06:26.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.604+0000 7f5bc159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:26.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.604+0000 7f5bc159a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:06:26.606 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.604+0000 7f5bc159a700 10 monclient: dump: 2023-09-30T02:06:26.606 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:26.606 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:26.606 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:26.606 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:26.606 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:26.606 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:26.606 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:26.606 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:26.606 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:26.606 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:26.608 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.605+0000 7f5bc8b6d700 10 monclient: shutdown 2023-09-30T02:06:26.609 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.605+0000 7f5bc8b6d700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:26.609 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.605+0000 7f5bc8b6d700 1 -- 172.21.15.59:0/1633275500 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5bc4142290 msgr2=0x7f5bc40b1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:26.609 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.605+0000 7f5bc8b6d700 1 --2- 172.21.15.59:0/1633275500 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5bc4142290 0x7f5bc40b1500 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f5bb80099e0 tx=0x7f5bb8009250 comp rx=0 tx=0).stop 2023-09-30T02:06:26.609 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.605+0000 7f5bc8b6d700 1 -- 172.21.15.59:0/1633275500 shutdown_connections 2023-09-30T02:06:26.609 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.605+0000 7f5bc8b6d700 1 --2- 172.21.15.59:0/1633275500 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5bc4142290 0x7f5bc40b1500 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:26.609 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.605+0000 7f5bc8b6d700 1 -- 172.21.15.59:0/1633275500 >> 172.21.15.59:0/1633275500 conn(0x7f5bc40631f0 msgr2=0x7f5bc40635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:26.609 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.605+0000 7f5bc8b6d700 1 -- 172.21.15.59:0/1633275500 shutdown_connections 2023-09-30T02:06:26.609 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.605+0000 7f5bc8b6d700 1 -- 172.21.15.59:0/1633275500 wait complete. 2023-09-30T02:06:26.609 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.606+0000 7f5bc8b6d700 10 monclient: build_initial_monmap 2023-09-30T02:06:26.609 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.606+0000 7f5bc8b6d700 10 monclient: monmap: 2023-09-30T02:06:26.609 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:26.610 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:06:26.610 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:06:26.610 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:06:26.610 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:26.610 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:06:26.610 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:06:26.610 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:06:26.610 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:06:26.610 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:26.610 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.606+0000 7f5bc8b6d700 1 librados: starting msgr at 2023-09-30T02:06:26.610 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.606+0000 7f5bc8b6d700 1 librados: starting objecter 2023-09-30T02:06:26.610 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.606+0000 7f5bc8b6d700 1 Processor -- start 2023-09-30T02:06:26.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.606+0000 7f5bc8b6d700 1 -- start start 2023-09-30T02:06:26.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.606+0000 7f5bc8b6d700 1 librados: setting wanted keys 2023-09-30T02:06:26.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.606+0000 7f5bc8b6d700 1 librados: calling monclient init 2023-09-30T02:06:26.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.606+0000 7f5bc8b6d700 10 monclient: init 2023-09-30T02:06:26.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.607+0000 7f5bc8b6d700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:26.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.607+0000 7f5bc8b6d700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T02:06:26.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.607+0000 7f5bc8b6d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5bc40a5b70 0x7f5bc413d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:26.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.607+0000 7f5bc8b6d700 10 monclient(hunting): picked mon.noname-b con 0x7f5bc40a5b70 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:06:26.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.607+0000 7f5bc8b6d700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5bc40a5f40 0x7f5bc413fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:26.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.607+0000 7f5bc8b6d700 10 monclient(hunting): picked mon.noname-c con 0x7f5bc40a5f40 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:26.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.607+0000 7f5bc8b6d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5bc4142290 0x7f5bc413df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:26.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.607+0000 7f5bc8b6d700 10 monclient(hunting): picked mon.noname-a con 0x7f5bc4142290 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:06:26.612 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.607+0000 7f5bc8b6d700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:26.612 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.607+0000 7f5bc8b6d700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc40a7810 con 0x7f5bc4142290 2023-09-30T02:06:26.612 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.607+0000 7f5bc8b6d700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:26.612 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.607+0000 7f5bc8b6d700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc4151580 con 0x7f5bc40a5b70 2023-09-30T02:06:26.612 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.607+0000 7f5bc8b6d700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:26.612 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.607+0000 7f5bc8b6d700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc4151860 con 0x7f5bc40a5f40 2023-09-30T02:06:26.612 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.607+0000 7f5bc8b6d700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:26.612 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.607+0000 7f5bc8b6d700 10 monclient(hunting): authenticate will time out at 1831.909873s 2023-09-30T02:06:26.612 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.608+0000 7f5bc1d9b700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5bc40a5f40 0x7f5bc413fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:26.612 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.609+0000 7f5bc1d9b700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5bc40a5f40 0x7f5bc413fcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:47902/0 (socket says 172.21.15.59:47902) 2023-09-30T02:06:26.612 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.609+0000 7f5bc1d9b700 1 -- 172.21.15.59:0/1883605283 learned_addr learned my addr 172.21.15.59:0/1883605283 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:06:26.613 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.609+0000 7f5bc1d9b700 10 monclient(hunting): get_auth_request con 0x7f5bc40a5f40 auth_method 0 2023-09-30T02:06:26.613 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.609+0000 7f5bc1d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:26.613 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.609+0000 7f5bc1d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:26.613 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.609+0000 7f5bc1d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:26.613 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.610+0000 7f5bc1d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:26.613 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.610+0000 7f5bc1d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:26.614 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.610+0000 7f5bc1d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:26.614 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.610+0000 7f3fbd3c1700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:26.614 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.610+0000 7f3fbd3c1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:26.614 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.610+0000 7f3fbd3c1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:26.615 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.613+0000 7f5bc1d9b700 10 monclient(hunting): handle_auth_done global_id 4343 payload 824 2023-09-30T02:06:26.615 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.613+0000 7f5bc1d9b700 1 -- 172.21.15.59:0/1883605283 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5bc40a5b70 msgr2=0x7f5bc413d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:26.615 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.613+0000 7f5bc1d9b700 1 --2- 172.21.15.59:0/1883605283 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5bc40a5b70 0x7f5bc413d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:26.615 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.613+0000 7f5bc1d9b700 1 -- 172.21.15.59:0/1883605283 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5bc4142290 msgr2=0x7f5bc413df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:26.615 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.613+0000 7f5bc1d9b700 1 --2- 172.21.15.59:0/1883605283 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5bc4142290 0x7f5bc413df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:26.615 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.613+0000 7f5bc1d9b700 10 monclient: _finish_hunting 0 2023-09-30T02:06:26.615 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.613+0000 7f5bc1d9b700 1 monclient: found mon.noname-c 2023-09-30T02:06:26.615 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.613+0000 7f5bc1d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:26.615 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.613+0000 7f5bc1d9b700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.59:3301/0 2023-09-30T02:06:26.616 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.614+0000 7f5bc1d9b700 1 -- 172.21.15.59:0/1883605283 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bb80094f0 con 0x7f5bc40a5f40 2023-09-30T02:06:26.616 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.616+0000 7f3fbd3c1700 10 monclient(hunting): handle_auth_done global_id 4362 payload 274 2023-09-30T02:06:26.616 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.616+0000 7f3fbd3c1700 1 -- 172.21.15.59:0/642111424 >> v1:172.21.15.110:6789/0 conn(0x7f3fb80a5f40 legacy=0x7f3fb8141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:26.616 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.616+0000 7f3fbd3c1700 1 -- 172.21.15.59:0/642111424 >> v1:172.21.15.59:6789/0 conn(0x7f3fb8142290 legacy=0x7f3fb813af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:06:26.616 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.616+0000 7f3fbd3c1700 10 monclient: _finish_hunting 0 2023-09-30T02:06:26.616 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.616+0000 7f3fbd3c1700 1 monclient: found mon.noname-a 2023-09-30T02:06:26.617 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.616+0000 7f3fbd3c1700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:26.617 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.616+0000 7f3fbd3c1700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.59:3300/0 2023-09-30T02:06:26.617 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.616+0000 7f3fbd3c1700 1 -- 172.21.15.59:0/642111424 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fb813b640 con 0x7f3fb80a5b70 2023-09-30T02:06:26.617 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.617+0000 7f5bc1d9b700 1 --2- 172.21.15.59:0/1883605283 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5bc40a5f40 0x7f5bc413fcf0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5bb8003080 tx=0x7f5bb8016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:26.617 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.617+0000 7f5bb27fc700 10 client.?.objecter ms_handle_connect 0x7f5bc40a5f40 2023-09-30T02:06:26.617 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.617+0000 7f5bb27fc700 10 client.?.objecter resend_mon_ops 2023-09-30T02:06:26.618 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.617+0000 7f4fd5ffb700 1 -- 172.21.15.59:0/495107266 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f4fd002f480 con 0x7f4fe00a5f40 2023-09-30T02:06:26.619 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.618+0000 7f4fd5ffb700 1 --2- 172.21.15.59:0/495107266 >> v2:172.21.15.110:6800/115870 conn(0x7f4fc40688c0 0x7f4fc406ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:26.619 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.618+0000 7f4fd5ffb700 1 -- 172.21.15.59:0/495107266 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f4fd0085a30 con 0x7f4fe00a5f40 2023-09-30T02:06:26.619 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.618+0000 7f4fd5ffb700 10 client.4393.objecter ms_dispatch 0x7f4fe0001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:06:26.619 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.618+0000 7f4fd5ffb700 3 client.4393.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:06:26.619 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.618+0000 7f4fd5ffb700 3 client.4393.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:06:26.619 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.618+0000 7f4fe5f77700 1 --2- 172.21.15.59:0/495107266 >> v2:172.21.15.110:6800/115870 conn(0x7f4fc40688c0 0x7f4fc406ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:26.619 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.618+0000 7f4fe5f77700 10 monclient: get_auth_request con 0x7f4fc40688c0 auth_method 0 2023-09-30T02:06:26.619 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.618+0000 7f4fd5ffb700 20 client.4393.objecter dump_active .. 0 homeless 2023-09-30T02:06:26.619 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.619+0000 7f4fd5ffb700 10 client.4393.objecter ms_handle_connect 0x7f4fc40688c0 2023-09-30T02:06:26.619 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.619+0000 7f4fe5f77700 1 --2- 172.21.15.59:0/495107266 >> v2:172.21.15.110:6800/115870 conn(0x7f4fc40688c0 0x7f4fc406ad70 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4fdc006fd0 tx=0x7f4fdc006db0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:26.620 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.619+0000 7f1cab7fe700 1 --2- 172.21.15.59:0/1126432236 >> v2:172.21.15.59:3300/0 conn(0x7f1cac142290 0x7f1cac0b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:26.620 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.619+0000 7f1cab7fe700 10 monclient(hunting): get_auth_request con 0x7f1cac142290 auth_method 0 2023-09-30T02:06:26.620 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.619+0000 7f1cab7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:26.620 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.619+0000 7f1cab7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:26.620 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.619+0000 7f1cab7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:26.620 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.620+0000 7f3fbd3c1700 1 --2- 172.21.15.59:0/642111424 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3fb80a5b70 0x7f3fb80b1500 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f3fb40098e0 tx=0x7f3fb40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b38f92885fbf59 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:26.622 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.622+0000 7f5bb27fc700 1 -- 172.21.15.59:0/1883605283 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bb8011ea0 con 0x7f5bc40a5f40 2023-09-30T02:06:26.622 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.622+0000 7f5bb27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:26.622 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.622+0000 7f5bb27fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:06:26.622 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.622+0000 7f5bb27fc700 10 monclient: dump: 2023-09-30T02:06:26.622 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:26.623 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:26.623 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:26.623 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:26.623 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:26.623 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:26.623 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:26.623 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:26.623 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:26.623 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:26.623 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.622+0000 7f5bb27fc700 10 monclient: _finish_auth 0 2023-09-30T02:06:26.623 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.622+0000 7f5bb27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:26.624 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.622+0000 7f5bb27fc700 1 -- 172.21.15.59:0/1883605283 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bb801c670 con 0x7f5bc40a5f40 2023-09-30T02:06:26.624 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.622+0000 7f5bb27fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:26.624 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.622+0000 7f5bb27fc700 1 -- 172.21.15.59:0/1883605283 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bb801ca10 con 0x7f5bc40a5f40 2023-09-30T02:06:26.624 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.622+0000 7f5bc8b6d700 5 monclient: authenticate success, global_id 4343 2023-09-30T02:06:26.624 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.622+0000 7f5bc8b6d700 10 monclient: _renew_subs 2023-09-30T02:06:26.624 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.622+0000 7f5bc8b6d700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:06:26.624 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.622+0000 7f5bc8b6d700 1 -- 172.21.15.59:0/1883605283 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bc413e480 con 0x7f5bc40a5f40 2023-09-30T02:06:26.624 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.622+0000 7f5bc0d99700 4 set_mon_vals no callback set 2023-09-30T02:06:26.624 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.622+0000 7f5bb27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:26.624 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.622+0000 7f5bb27fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:06:26.624 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.623+0000 7f5bc8b6d700 10 client.4343.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:06:26.625 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.622+0000 7f5bb27fc700 10 monclient: dump: 2023-09-30T02:06:26.625 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:26.625 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:26.625 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:26.625 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:26.625 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:26.625 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:26.625 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:26.625 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:26.625 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:26.625 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:26.625 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.623+0000 7f5bc8b6d700 10 monclient: _renew_subs 2023-09-30T02:06:26.625 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.623+0000 7f5bc8b6d700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:06:26.626 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.623+0000 7f5bc8b6d700 1 -- 172.21.15.59:0/1883605283 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bc4151fc0 con 0x7f5bc40a5f40 2023-09-30T02:06:26.626 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.623+0000 7f5bc8b6d700 1 librados: init done 2023-09-30T02:06:26.626 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.624+0000 7f5bc8b6d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:26.626 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.624+0000 7f5bc8b6d700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:26.626 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.624+0000 7f5bc8b6d700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:06:26.626 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.624+0000 7f5bc8b6d700 1 -- 172.21.15.59:0/1883605283 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bc4151fc0 con 0x7f5bc40a5f40 2023-09-30T02:06:26.632 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.633+0000 7fa6bf7fe700 1 -- 172.21.15.59:0/939460102 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa6cc053a70 con 0x7fa6d80a5f40 2023-09-30T02:06:26.632 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.633+0000 7fa6bf7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:26.632 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.633+0000 7fa6bf7fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:06:26.636 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.637+0000 7f1cab7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:26.637 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.637+0000 7f1cab7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:26.637 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.637+0000 7f1cab7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:26.640 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.641+0000 7f1cab7fe700 10 monclient(hunting): handle_auth_done global_id 4365 payload 274 2023-09-30T02:06:26.641 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.641+0000 7f1cab7fe700 1 -- 172.21.15.59:0/1126432236 >> v1:172.21.15.110:6789/0 conn(0x7f1cac0a5f40 legacy=0x7f1cac13af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:26.641 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.641+0000 7f1cab7fe700 1 -- 172.21.15.59:0/1126432236 >> v1:172.21.15.59:6789/0 conn(0x7f1cac0a5b70 legacy=0x7f1cac141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:26.641 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.641+0000 7f1cab7fe700 10 monclient: _finish_hunting 0 2023-09-30T02:06:26.641 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.641+0000 7f1cab7fe700 1 monclient: found mon.noname-a 2023-09-30T02:06:26.641 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.641+0000 7f1cab7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:26.641 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.641+0000 7f1cab7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.59:3300/0 2023-09-30T02:06:26.641 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.641+0000 7f1cab7fe700 1 -- 172.21.15.59:0/1126432236 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cac13b640 con 0x7f1cac142290 2023-09-30T02:06:26.643 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.644+0000 7f3faffff700 1 -- 172.21.15.59:0/642111424 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb4009e80 con 0x7f3fb80a5b70 2023-09-30T02:06:26.643 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.644+0000 7f3faffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:26.644 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.644+0000 7f3faffff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T02:06:26.644 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.644+0000 7f3faffff700 10 monclient: dump: 2023-09-30T02:06:26.644 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:26.644 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:26.644 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:26.644 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:26.644 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:26.644 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:26.644 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:26.645 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:26.645 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:26.645 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:26.645 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.644+0000 7f3faffff700 10 monclient: _finish_auth 0 2023-09-30T02:06:26.645 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.644+0000 7f3faffff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:26.645 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.644+0000 7f3faffff700 1 -- 172.21.15.59:0/642111424 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fb400b820 con 0x7f3fb80a5b70 2023-09-30T02:06:26.645 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.644+0000 7f3fbf625700 5 monclient: authenticate success, global_id 4362 2023-09-30T02:06:26.645 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.644+0000 7f3fbf625700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:06:26.645 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.644+0000 7f3faffff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:26.645 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.644+0000 7f3fbf625700 10 monclient: get_monmap_and_config success 2023-09-30T02:06:26.645 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.644+0000 7f3faffff700 1 -- 172.21.15.59:0/642111424 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb400bba0 con 0x7f3fb80a5b70 2023-09-30T02:06:26.646 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.644+0000 7f3fbf625700 4 set_mon_vals no callback set 2023-09-30T02:06:26.646 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.644+0000 7f3faffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:26.646 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.644+0000 7f3faffff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T02:06:26.646 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.644+0000 7f3faffff700 10 monclient: dump: 2023-09-30T02:06:26.646 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:26.646 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:26.646 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:26.646 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:26.646 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:26.646 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:26.647 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:26.647 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:26.647 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:26.647 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:26.649 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.644+0000 7f3fbf625700 10 monclient: shutdown 2023-09-30T02:06:26.649 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.644+0000 7f3fbf625700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:26.649 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.644+0000 7f3fbf625700 1 -- 172.21.15.59:0/642111424 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3fb80a5b70 msgr2=0x7f3fb80b1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:26.649 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.644+0000 7f3fbf625700 1 --2- 172.21.15.59:0/642111424 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3fb80a5b70 0x7f3fb80b1500 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f3fb40098e0 tx=0x7f3fb40091a0 comp rx=0 tx=0).stop 2023-09-30T02:06:26.649 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.645+0000 7f3fbf625700 1 -- 172.21.15.59:0/642111424 shutdown_connections 2023-09-30T02:06:26.649 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.645+0000 7f3fbf625700 1 --2- 172.21.15.59:0/642111424 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3fb80a5b70 0x7f3fb80b1500 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:26.649 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.645+0000 7f3fbf625700 1 -- 172.21.15.59:0/642111424 >> 172.21.15.59:0/642111424 conn(0x7f3fb80631f0 msgr2=0x7f3fb80635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:26.649 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.646+0000 7f3fbf625700 1 -- 172.21.15.59:0/642111424 shutdown_connections 2023-09-30T02:06:26.650 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.646+0000 7f3fbf625700 1 -- 172.21.15.59:0/642111424 wait complete. 2023-09-30T02:06:26.650 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.646+0000 7f3fbf625700 10 monclient: build_initial_monmap 2023-09-30T02:06:26.650 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.646+0000 7f3fbf625700 10 monclient: monmap: 2023-09-30T02:06:26.650 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:26.650 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:06:26.650 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:06:26.650 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:06:26.650 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:26.650 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:06:26.650 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:06:26.650 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:06:26.650 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:06:26.650 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:26.651 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.647+0000 7f3fbf625700 1 librados: starting msgr at 2023-09-30T02:06:26.651 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.647+0000 7f3fbf625700 1 librados: starting objecter 2023-09-30T02:06:26.651 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.647+0000 7f3fbf625700 1 Processor -- start 2023-09-30T02:06:26.651 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.647+0000 7f3fbf625700 1 -- start start 2023-09-30T02:06:26.651 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.647+0000 7f3fbf625700 1 librados: setting wanted keys 2023-09-30T02:06:26.651 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.647+0000 7f3fbf625700 1 librados: calling monclient init 2023-09-30T02:06:26.651 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.647+0000 7f3fbf625700 10 monclient: init 2023-09-30T02:06:26.651 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.648+0000 7f3fbf625700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:26.651 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.648+0000 7f3fbf625700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T02:06:26.651 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.648+0000 7f3fbf625700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3fb80a5b70 0x7f3fb813c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:26.653 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.648+0000 7f3fbf625700 10 monclient(hunting): picked mon.noname-b con 0x7f3fb80a5b70 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:06:26.653 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.648+0000 7f3fbf625700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3fb80a5f40 0x7f3fb813ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:26.653 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.648+0000 7f3fbf625700 10 monclient(hunting): picked mon.noname-c con 0x7f3fb80a5f40 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:26.653 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.648+0000 7f3fbf625700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3fb8142290 0x7f3fb81562c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:26.653 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.648+0000 7f3fbf625700 10 monclient(hunting): picked mon.noname-a con 0x7f3fb8142290 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:06:26.653 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.648+0000 7f3fbf625700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:26.653 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.648+0000 7f3fbf625700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fb813f110 con 0x7f3fb8142290 2023-09-30T02:06:26.653 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.648+0000 7f3fbf625700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:26.653 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.648+0000 7f3fbf625700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fb8155b50 con 0x7f3fb80a5b70 2023-09-30T02:06:26.653 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.648+0000 7f3fbf625700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:26.654 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.648+0000 7f3fbf625700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fb8155da0 con 0x7f3fb80a5f40 2023-09-30T02:06:26.654 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.648+0000 7f3fbf625700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:26.654 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.648+0000 7f3fbf625700 10 monclient(hunting): authenticate will time out at 1831.950158s 2023-09-30T02:06:26.654 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.649+0000 7f1cab7fe700 1 --2- 172.21.15.59:0/1126432236 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1cac142290 0x7f1cac0b1500 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f1ca00098e0 tx=0x7f1ca00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99895c172bd70e54 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:26.654 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.654+0000 7f3fbdbc2700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3fb8142290 0x7f3fb81562c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:26.657 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.657+0000 7f3fbdbc2700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3fb8142290 0x7f3fb81562c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52240/0 (socket says 172.21.15.59:52240) 2023-09-30T02:06:26.657 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.657+0000 7f3fbdbc2700 1 -- 172.21.15.59:0/1323244403 learned_addr learned my addr 172.21.15.59:0/1323244403 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:06:26.657 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.657+0000 7f3fbdbc2700 10 monclient(hunting): get_auth_request con 0x7f3fb8142290 auth_method 0 2023-09-30T02:06:26.657 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.657+0000 7f3fbdbc2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:26.657 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.657+0000 7f3fbdbc2700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:26.657 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.657+0000 7f3fbdbc2700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:26.658 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.659+0000 7f3fbdbc2700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:26.658 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.659+0000 7f3fbdbc2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:26.658 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.659+0000 7f3fbdbc2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:26.663 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.664+0000 7f3fbdbc2700 10 monclient(hunting): handle_auth_done global_id 4368 payload 824 2023-09-30T02:06:26.663 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.664+0000 7f3fbdbc2700 1 -- 172.21.15.59:0/1323244403 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3fb80a5f40 msgr2=0x7f3fb813ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:26.663 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.664+0000 7f3fbdbc2700 1 --2- 172.21.15.59:0/1323244403 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3fb80a5f40 0x7f3fb813ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:26.663 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.664+0000 7f3fbdbc2700 1 -- 172.21.15.59:0/1323244403 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3fb80a5b70 msgr2=0x7f3fb813c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:26.663 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.664+0000 7f3fbdbc2700 1 --2- 172.21.15.59:0/1323244403 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3fb80a5b70 0x7f3fb813c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:26.664 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.664+0000 7f3fbdbc2700 10 monclient: _finish_hunting 0 2023-09-30T02:06:26.664 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.664+0000 7f3fbdbc2700 1 monclient: found mon.noname-a 2023-09-30T02:06:26.664 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.664+0000 7f3fbdbc2700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:26.664 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.664+0000 7f3fbdbc2700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.59:3300/0 2023-09-30T02:06:26.664 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.664+0000 7f3fbdbc2700 1 -- 172.21.15.59:0/1323244403 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fb4009420 con 0x7f3fb8142290 2023-09-30T02:06:26.665 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.666+0000 7f1caaffd700 1 -- 172.21.15.59:0/1126432236 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ca0009e80 con 0x7f1cac142290 2023-09-30T02:06:26.665 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.666+0000 7f1caaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:26.666 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.666+0000 7f1caaffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T02:06:26.666 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.666+0000 7f1caaffd700 10 monclient: dump: 2023-09-30T02:06:26.666 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:26.666 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:26.666 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:26.666 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:26.666 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:26.666 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:26.666 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:26.666 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:26.666 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:26.667 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:26.667 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.666+0000 7f1caaffd700 10 monclient: _finish_auth 0 2023-09-30T02:06:26.667 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.666+0000 7f1caaffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:26.667 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.666+0000 7f1caaffd700 1 -- 172.21.15.59:0/1126432236 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ca000b860 con 0x7f1cac142290 2023-09-30T02:06:26.667 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.666+0000 7f1caaffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:26.667 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.666+0000 7f1caaffd700 1 -- 172.21.15.59:0/1126432236 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ca000bbe0 con 0x7f1cac142290 2023-09-30T02:06:26.667 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.666+0000 7f1caaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:26.667 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.666+0000 7f1caaffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T02:06:26.667 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.666+0000 7f1caaffd700 10 monclient: dump: 2023-09-30T02:06:26.667 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:26.667 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:26.668 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:26.668 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:26.668 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:26.668 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:26.668 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:26.668 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:26.668 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:26.668 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:26.669 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.669+0000 7f1cb29c0700 5 monclient: authenticate success, global_id 4365 2023-09-30T02:06:26.669 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.669+0000 7f1cb29c0700 10 monclient: get_monmap_and_config success 2023-09-30T02:06:26.669 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.669+0000 7f1cb29c0700 4 set_mon_vals no callback set 2023-09-30T02:06:26.669 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.670+0000 7f1cb29c0700 10 monclient: shutdown 2023-09-30T02:06:26.669 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.670+0000 7f1cb29c0700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:26.669 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.670+0000 7f1cb29c0700 1 -- 172.21.15.59:0/1126432236 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1cac142290 msgr2=0x7f1cac0b1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:26.669 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.670+0000 7f1cb29c0700 1 --2- 172.21.15.59:0/1126432236 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1cac142290 0x7f1cac0b1500 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f1ca00098e0 tx=0x7f1ca00091a0 comp rx=0 tx=0).stop 2023-09-30T02:06:26.669 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.670+0000 7f1cb29c0700 1 -- 172.21.15.59:0/1126432236 shutdown_connections 2023-09-30T02:06:26.670 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.670+0000 7f1cb29c0700 1 --2- 172.21.15.59:0/1126432236 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1cac142290 0x7f1cac0b1500 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:26.670 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.670+0000 7f1cb29c0700 1 -- 172.21.15.59:0/1126432236 >> 172.21.15.59:0/1126432236 conn(0x7f1cac0631f0 msgr2=0x7f1cac0635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:26.670 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.670+0000 7f5bb27fc700 1 -- 172.21.15.59:0/1883605283 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f5bb801a450 con 0x7f5bc40a5f40 2023-09-30T02:06:26.670 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.671+0000 7f5bb27fc700 1 --2- 172.21.15.59:0/1883605283 >> v2:172.21.15.110:6800/115870 conn(0x7f5ba4068910 0x7f5ba406adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:26.671 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.671+0000 7f5bb27fc700 1 -- 172.21.15.59:0/1883605283 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f5bb80868e0 con 0x7f5bc40a5f40 2023-09-30T02:06:26.671 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.671+0000 7f5bb27fc700 10 client.4343.objecter ms_dispatch 0x7f5bc4001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:06:26.671 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.671+0000 7f5bb27fc700 3 client.4343.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:06:26.671 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.671+0000 7f5bb27fc700 3 client.4343.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:06:26.671 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.671+0000 7f5bc259c700 1 --2- 172.21.15.59:0/1883605283 >> v2:172.21.15.110:6800/115870 conn(0x7f5ba4068910 0x7f5ba406adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:26.671 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.671+0000 7f5bc259c700 10 monclient: get_auth_request con 0x7f5ba4068910 auth_method 0 2023-09-30T02:06:26.671 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.671+0000 7f5bb27fc700 20 client.4343.objecter dump_active .. 0 homeless 2023-09-30T02:06:26.671 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.672+0000 7f5bb27fc700 10 client.4343.objecter ms_handle_connect 0x7f5ba4068910 2023-09-30T02:06:26.672 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.672+0000 7f5bc259c700 1 --2- 172.21.15.59:0/1883605283 >> v2:172.21.15.110:6800/115870 conn(0x7f5ba4068910 0x7f5ba406adc0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5bb40097c0 tx=0x7f5bb4006ce0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:26.673 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.674+0000 7f1cb29c0700 1 -- 172.21.15.59:0/1126432236 shutdown_connections 2023-09-30T02:06:26.673 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.674+0000 7f1cb29c0700 1 -- 172.21.15.59:0/1126432236 wait complete. 2023-09-30T02:06:26.674 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.675+0000 7f1cb29c0700 10 monclient: build_initial_monmap 2023-09-30T02:06:26.674 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.675+0000 7f1cb29c0700 10 monclient: monmap: 2023-09-30T02:06:26.674 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:26.675 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:06:26.675 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:06:26.675 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:06:26.675 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:26.675 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:06:26.675 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:06:26.675 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:06:26.675 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:06:26.675 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:26.675 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.675+0000 7f1cb29c0700 1 librados: starting msgr at 2023-09-30T02:06:26.675 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.675+0000 7f1cb29c0700 1 librados: starting objecter 2023-09-30T02:06:26.676 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.675+0000 7f1cb29c0700 1 Processor -- start 2023-09-30T02:06:26.676 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.675+0000 7f1cb29c0700 1 -- start start 2023-09-30T02:06:26.676 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.675+0000 7f1cb29c0700 1 librados: setting wanted keys 2023-09-30T02:06:26.676 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.675+0000 7f1cb29c0700 1 librados: calling monclient init 2023-09-30T02:06:26.676 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.675+0000 7f1cb29c0700 10 monclient: init 2023-09-30T02:06:26.677 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.676+0000 7f1cb29c0700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:26.677 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.676+0000 7f1cb29c0700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T02:06:26.677 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.676+0000 7f1cb29c0700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1cac0a5b70 0x7f1cac143e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:26.677 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.676+0000 7f1cb29c0700 10 monclient(hunting): picked mon.noname-a con 0x7f1cac0a5b70 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:06:26.677 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.676+0000 7f1cb29c0700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1cac0a5f40 0x7f1cac139140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:26.677 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.676+0000 7f1cb29c0700 10 monclient(hunting): picked mon.noname-c con 0x7f1cac0a5f40 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:26.677 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.676+0000 7f1cb29c0700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1cac142290 0x7f1cac13f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:26.677 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.676+0000 7f1cb29c0700 10 monclient(hunting): picked mon.noname-b con 0x7f1cac142290 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:06:26.677 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.676+0000 7f1cb29c0700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:26.677 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.676+0000 7f1cb29c0700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cac1399f0 con 0x7f1cac0a5b70 2023-09-30T02:06:26.677 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.676+0000 7f1cb29c0700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:26.678 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.676+0000 7f1cb29c0700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cac139680 con 0x7f1cac142290 2023-09-30T02:06:26.678 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.676+0000 7f1cb29c0700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:26.678 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.676+0000 7f1cb29c0700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cac13ac20 con 0x7f1cac0a5f40 2023-09-30T02:06:26.678 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.676+0000 7f1cb29c0700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:26.678 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.676+0000 7f1cb29c0700 10 monclient(hunting): authenticate will time out at 1831.978842s 2023-09-30T02:06:26.686 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.687+0000 7f1cabfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1cac0a5b70 0x7f1cac143e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:26.687 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.687+0000 7f3fbdbc2700 1 --2- 172.21.15.59:0/1323244403 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3fb8142290 0x7f3fb81562c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f3fb000bbb0 tx=0x7f3fb0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:26.687 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.687+0000 7f3fad7fa700 10 client.?.objecter ms_handle_connect 0x7f3fb8142290 2023-09-30T02:06:26.687 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.687+0000 7f3fad7fa700 10 client.?.objecter resend_mon_ops 2023-09-30T02:06:26.687 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.687+0000 7f1cabfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1cac0a5b70 0x7f1cac143e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52246/0 (socket says 172.21.15.59:52246) 2023-09-30T02:06:26.687 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.687+0000 7f1cabfff700 1 -- 172.21.15.59:0/2892307729 learned_addr learned my addr 172.21.15.59:0/2892307729 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:06:26.687 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.687+0000 7f1cabfff700 10 monclient(hunting): get_auth_request con 0x7f1cac0a5b70 auth_method 0 2023-09-30T02:06:26.687 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.687+0000 7f1cabfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:26.687 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.687+0000 7f1cabfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:26.688 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.687+0000 7f1cabfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:26.688 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.689+0000 7f1cabfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:26.688 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.689+0000 7f1cabfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:26.688 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.689+0000 7f1cabfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:26.695 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.696+0000 7f1cabfff700 10 monclient(hunting): handle_auth_done global_id 4371 payload 824 2023-09-30T02:06:26.695 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.696+0000 7f1cabfff700 1 -- 172.21.15.59:0/2892307729 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1cac0a5f40 msgr2=0x7f1cac139140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:26.695 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.696+0000 7f1cabfff700 1 --2- 172.21.15.59:0/2892307729 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1cac0a5f40 0x7f1cac139140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:26.695 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.696+0000 7f1cabfff700 1 -- 172.21.15.59:0/2892307729 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1cac142290 msgr2=0x7f1cac13f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:26.695 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.696+0000 7f1cabfff700 1 --2- 172.21.15.59:0/2892307729 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1cac142290 0x7f1cac13f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:26.695 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.696+0000 7f1cabfff700 10 monclient: _finish_hunting 0 2023-09-30T02:06:26.695 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.696+0000 7f1cabfff700 1 monclient: found mon.noname-a 2023-09-30T02:06:26.696 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.696+0000 7f1cabfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:26.696 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.696+0000 7f1cabfff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.59:3300/0 2023-09-30T02:06:26.696 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.696+0000 7f1cabfff700 1 -- 172.21.15.59:0/2892307729 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ca0009420 con 0x7f1cac0a5b70 2023-09-30T02:06:26.699 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.700+0000 7f1cabfff700 1 --2- 172.21.15.59:0/2892307729 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1cac0a5b70 0x7f1cac143e90 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c00c5c0 tx=0x7f1c9c00c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:26.699 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.700+0000 7f1c93fff700 10 client.?.objecter ms_handle_connect 0x7f1cac0a5b70 2023-09-30T02:06:26.699 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.700+0000 7f1c93fff700 10 client.?.objecter resend_mon_ops 2023-09-30T02:06:26.706 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.706+0000 7f6f01605700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2023-09-30T02:06:26.706 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.706+0000 7f6f01605700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2023-09-30T02:06:26.706 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.706+0000 7f6f01605700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:26.706 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.706+0000 7f6f01605700 1 -- 172.21.15.59:0/1439181173 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f6edc005630 con 0x7f6efc06c9f0 2023-09-30T02:06:26.782 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.782+0000 7f3fad7fa700 1 -- 172.21.15.59:0/1323244403 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb00092e0 con 0x7f3fb8142290 2023-09-30T02:06:26.782 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.782+0000 7f3fad7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:26.782 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.782+0000 7f3fad7fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T02:06:26.782 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.782+0000 7f3fad7fa700 10 monclient: dump: 2023-09-30T02:06:26.782 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:26.783 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:26.783 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:26.783 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:26.783 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:26.783 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:26.783 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:26.783 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:26.783 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:26.783 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:26.783 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.782+0000 7f3fad7fa700 10 monclient: _finish_auth 0 2023-09-30T02:06:26.783 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.782+0000 7f3fad7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:26.783 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.782+0000 7f3fbf625700 5 monclient: authenticate success, global_id 4368 2023-09-30T02:06:26.784 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.782+0000 7f3fbf625700 10 monclient: _renew_subs 2023-09-30T02:06:26.784 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.782+0000 7f3fbf625700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:06:26.784 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.782+0000 7f3fbf625700 1 -- 172.21.15.59:0/1323244403 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fb8156860 con 0x7f3fb8142290 2023-09-30T02:06:26.784 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.782+0000 7f3fbf625700 10 client.4368.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:06:26.784 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.782+0000 7f3fbf625700 10 monclient: _renew_subs 2023-09-30T02:06:26.784 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.782+0000 7f3fbf625700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:06:26.784 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.782+0000 7f3fbf625700 1 -- 172.21.15.59:0/1323244403 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fb806acc0 con 0x7f3fb8142290 2023-09-30T02:06:26.784 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.782+0000 7f3fbf625700 1 librados: init done 2023-09-30T02:06:26.785 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.784+0000 7f3fbf625700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:26.785 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.784+0000 7f3fbf625700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:26.785 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.784+0000 7f3fbf625700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:06:26.785 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.784+0000 7f3fbf625700 1 -- 172.21.15.59:0/1323244403 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fb8155fc0 con 0x7f3fb8142290 2023-09-30T02:06:26.787 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.787+0000 7f3fad7fa700 1 -- 172.21.15.59:0/1323244403 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fb0009980 con 0x7f3fb8142290 2023-09-30T02:06:26.787 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.787+0000 7f3fad7fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:26.787 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.787+0000 7f3fad7fa700 1 -- 172.21.15.59:0/1323244403 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb000c9b0 con 0x7f3fb8142290 2023-09-30T02:06:26.787 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.787+0000 7f3fad7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:26.787 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.787+0000 7f3fad7fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T02:06:26.787 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.787+0000 7f3fad7fa700 10 monclient: dump: 2023-09-30T02:06:26.787 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:26.787 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:26.787 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:26.787 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:26.787 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:26.788 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:26.788 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:26.788 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:26.788 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:26.788 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:26.788 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.787+0000 7f3faf7fe700 4 set_mon_vals no callback set 2023-09-30T02:06:26.790 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.790+0000 7f1c93fff700 1 -- 172.21.15.59:0/2892307729 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c9c011590 con 0x7f1cac0a5b70 2023-09-30T02:06:26.790 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.790+0000 7f1c93fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:26.790 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.790+0000 7f1c93fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T02:06:26.790 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.790+0000 7f1c93fff700 10 monclient: dump: 2023-09-30T02:06:26.790 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:26.790 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:26.790 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:26.790 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:26.790 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:26.790 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:26.790 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:26.791 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:26.791 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:26.791 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:26.791 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.790+0000 7f1c93fff700 10 monclient: _finish_auth 0 2023-09-30T02:06:26.791 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.790+0000 7f1c93fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:26.791 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.790+0000 7f1c93fff700 1 -- 172.21.15.59:0/2892307729 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c9c011710 con 0x7f1cac0a5b70 2023-09-30T02:06:26.791 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.790+0000 7f1cb29c0700 5 monclient: authenticate success, global_id 4371 2023-09-30T02:06:26.791 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.790+0000 7f1c93fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:26.791 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.790+0000 7f1c93fff700 1 -- 172.21.15.59:0/2892307729 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c9c011ad0 con 0x7f1cac0a5b70 2023-09-30T02:06:26.791 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.790+0000 7f1cb29c0700 10 monclient: _renew_subs 2023-09-30T02:06:26.791 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.790+0000 7f1cb29c0700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:06:26.791 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.790+0000 7f1cb29c0700 1 -- 172.21.15.59:0/2892307729 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cac1413c0 con 0x7f1cac0a5b70 2023-09-30T02:06:26.792 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.790+0000 7f1caa7fc700 4 set_mon_vals no callback set 2023-09-30T02:06:26.792 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.790+0000 7f1c93fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:26.792 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.790+0000 7f1c93fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T02:06:26.792 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.790+0000 7f1c93fff700 10 monclient: dump: 2023-09-30T02:06:26.792 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:26.792 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:26.792 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:26.792 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:26.792 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:26.792 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:26.792 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:26.792 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:26.793 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:26.793 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:26.793 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.790+0000 7f1cb29c0700 10 client.4371.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:06:26.793 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.790+0000 7f1cb29c0700 10 monclient: _renew_subs 2023-09-30T02:06:26.793 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.790+0000 7f1cb29c0700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:06:26.793 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.790+0000 7f1cb29c0700 1 -- 172.21.15.59:0/2892307729 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cac13fd90 con 0x7f1cac0a5b70 2023-09-30T02:06:26.793 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.790+0000 7f1cb29c0700 1 librados: init done 2023-09-30T02:06:26.794 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.791+0000 7f1cb29c0700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:26.794 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.791+0000 7f1cb29c0700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:26.794 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.791+0000 7f1cb29c0700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:06:26.794 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.791+0000 7f1cb29c0700 1 -- 172.21.15.59:0/2892307729 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cac139b30 con 0x7f1cac0a5b70 2023-09-30T02:06:26.819 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.820+0000 7f4fd5ffb700 1 -- 172.21.15.59:0/495107266 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4fd0053370 con 0x7f4fe00a5f40 2023-09-30T02:06:26.820 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.820+0000 7f4fd5ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:26.820 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.820+0000 7f4fd5ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:06:26.909 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.910+0000 7f6ee2ffd700 1 -- 172.21.15.59:0/1439181173 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6ef0055760 con 0x7f6efc06c9f0 2023-09-30T02:06:26.910 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.910+0000 7f6ee2ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2023-09-30T02:06:26.910 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.910+0000 7f6ee2ffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:06:26.910 INFO:teuthology.orchestra.run.smithi059.stdout:60129542151 2023-09-30T02:06:26.912 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.913+0000 7f6f01605700 10 librados: watch_flush enter 2023-09-30T02:06:26.912 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.913+0000 7f6f01605700 10 librados: watch_flush exit 2023-09-30T02:06:26.913 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.913+0000 7f6f01605700 20 client.4387.objecter shutdown clearing up homeless session... 2023-09-30T02:06:26.913 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.913+0000 7f6f01605700 10 client.4387.objecter successfully canceled tick 2023-09-30T02:06:26.913 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.913+0000 7f6f01605700 1 -- 172.21.15.59:0/1439181173 >> v2:172.21.15.110:6800/115870 conn(0x7f6ee80688c0 msgr2=0x7f6ee806ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:26.913 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.913+0000 7f6f01605700 1 --2- 172.21.15.59:0/1439181173 >> v2:172.21.15.110:6800/115870 conn(0x7f6ee80688c0 0x7f6ee806ad70 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6eec0099b0 tx=0x7f6eec017000 comp rx=0 tx=0).stop 2023-09-30T02:06:26.913 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.913+0000 7f6f01605700 10 monclient: shutdown 2023-09-30T02:06:26.913 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.913+0000 7f6f01605700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:26.913 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.913+0000 7f6f01605700 1 -- 172.21.15.59:0/1439181173 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6efc06c9f0 msgr2=0x7f6efc13d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:26.913 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.913+0000 7f6f01605700 1 --2- 172.21.15.59:0/1439181173 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6efc06c9f0 0x7f6efc13d4f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f6ef000ec80 tx=0x7f6ef00091a0 comp rx=0 tx=0).stop 2023-09-30T02:06:26.914 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.914+0000 7f6f01605700 1 -- 172.21.15.59:0/1439181173 shutdown_connections 2023-09-30T02:06:26.914 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.914+0000 7f6f01605700 1 --2- 172.21.15.59:0/1439181173 >> v2:172.21.15.110:6800/115870 conn(0x7f6ee80688c0 0x7f6ee806ad70 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:26.914 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.914+0000 7f6f01605700 1 --2- 172.21.15.59:0/1439181173 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6efc13b6e0 0x7f6efc139560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:26.914 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.914+0000 7f6f01605700 1 --2- 172.21.15.59:0/1439181173 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6efc06c620 0x7f6efc13c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:26.914 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.914+0000 7f6f01605700 1 --2- 172.21.15.59:0/1439181173 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6efc06c9f0 0x7f6efc13d4f0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:26.915 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.914+0000 7f6f01605700 1 -- 172.21.15.59:0/1439181173 >> 172.21.15.59:0/1439181173 conn(0x7f6efc069da0 msgr2=0x7f6efc0b0e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:26.915 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.915+0000 7f6f01605700 1 -- 172.21.15.59:0/1439181173 shutdown_connections 2023-09-30T02:06:26.915 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.915+0000 7f6f01605700 1 -- 172.21.15.59:0/1439181173 wait complete. 2023-09-30T02:06:26.915 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.915+0000 7f6f01605700 1 librados: shutdown 2023-09-30T02:06:26.932 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542151 for osd.7 2023-09-30T02:06:26.932 DEBUG:teuthology.parallel:result is None 2023-09-30T02:06:26.933 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.932+0000 7f1c93fff700 1 -- 172.21.15.59:0/2892307729 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f1c9c012420 con 0x7f1cac0a5b70 2023-09-30T02:06:26.933 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.932+0000 7f1c93fff700 1 --2- 172.21.15.59:0/2892307729 >> v2:172.21.15.110:6800/115870 conn(0x7f1c98068990 0x7f1c9806ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:26.933 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.933+0000 7f1cab7fe700 1 --2- 172.21.15.59:0/2892307729 >> v2:172.21.15.110:6800/115870 conn(0x7f1c98068990 0x7f1c9806ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:26.933 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.933+0000 7f1cab7fe700 10 monclient: get_auth_request con 0x7f1c98068990 auth_method 0 2023-09-30T02:06:26.933 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.933+0000 7f1cab7fe700 1 --2- 172.21.15.59:0/2892307729 >> v2:172.21.15.110:6800/115870 conn(0x7f1c98068990 0x7f1c9806ae40 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1cac144690 tx=0x7f1ca0017000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:26.933 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.933+0000 7f1c93fff700 10 client.4371.objecter ms_handle_connect 0x7f1c98068990 2023-09-30T02:06:26.958 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.959+0000 7fa6ddf9f700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2023-09-30T02:06:26.959 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.959+0000 7fa6ddf9f700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-09-30T02:06:26.959 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.959+0000 7fa6ddf9f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:06:26.959 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:26.959+0000 7fa6ddf9f700 1 -- 172.21.15.59:0/939460102 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fa6d8151e70 con 0x7fa6d80a5f40 2023-09-30T02:06:27.001 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.002+0000 7f5bb27fc700 1 -- 172.21.15.59:0/1883605283 <== mon.2 v2:172.21.15.59:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5bb8054180 con 0x7f5bc40a5f40 2023-09-30T02:06:27.002 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.002+0000 7f5bb27fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:27.002 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.002+0000 7f5bb27fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:06:27.121 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.122+0000 7f4fe81db700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2023-09-30T02:06:27.122 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.122+0000 7f4fe81db700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-09-30T02:06:27.122 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.122+0000 7f4fe81db700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:27.122 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.122+0000 7f4fe81db700 1 -- 172.21.15.59:0/495107266 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f4fe0064a00 con 0x7f4fe00a5f40 2023-09-30T02:06:27.125 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.125+0000 7f4fd5ffb700 1 -- 172.21.15.59:0/495107266 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4fd0056990 con 0x7f4fe00a5f40 2023-09-30T02:06:27.125 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.126+0000 7f4fd5ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-09-30T02:06:27.125 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.126+0000 7f4fd5ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:06:27.125 INFO:teuthology.orchestra.run.smithi059.stdout:64424509447 2023-09-30T02:06:27.128 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.128+0000 7f4fe81db700 10 librados: watch_flush enter 2023-09-30T02:06:27.128 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.128+0000 7f4fe81db700 10 librados: watch_flush exit 2023-09-30T02:06:27.128 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.128+0000 7f4fe81db700 20 client.4393.objecter shutdown clearing up homeless session... 2023-09-30T02:06:27.128 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.128+0000 7f4fe81db700 10 client.4393.objecter successfully canceled tick 2023-09-30T02:06:27.128 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.128+0000 7f4fe81db700 1 -- 172.21.15.59:0/495107266 >> v2:172.21.15.110:6800/115870 conn(0x7f4fc40688c0 msgr2=0x7f4fc406ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:27.129 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.128+0000 7f4fe81db700 1 --2- 172.21.15.59:0/495107266 >> v2:172.21.15.110:6800/115870 conn(0x7f4fc40688c0 0x7f4fc406ad70 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4fdc006fd0 tx=0x7f4fdc006db0 comp rx=0 tx=0).stop 2023-09-30T02:06:27.129 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.128+0000 7f4fe81db700 10 monclient: shutdown 2023-09-30T02:06:27.129 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.128+0000 7f4fe81db700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:27.129 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.128+0000 7f4fe81db700 1 -- 172.21.15.59:0/495107266 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4fe00a5f40 msgr2=0x7f4fe0135560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:27.129 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.128+0000 7f4fe81db700 1 --2- 172.21.15.59:0/495107266 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4fe00a5f40 0x7f4fe0135560 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4fd0009290 tx=0x7f4fd0017000 comp rx=0 tx=0).stop 2023-09-30T02:06:27.129 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.128+0000 7f4fe81db700 1 -- 172.21.15.59:0/495107266 shutdown_connections 2023-09-30T02:06:27.129 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.128+0000 7f4fe81db700 1 --2- 172.21.15.59:0/495107266 >> v2:172.21.15.110:6800/115870 conn(0x7f4fc40688c0 0x7f4fc406ad70 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:27.129 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.128+0000 7f4fe81db700 1 --2- 172.21.15.59:0/495107266 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4fe01420c0 0x7f4fe01347c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:27.129 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.128+0000 7f4fe81db700 1 --2- 172.21.15.59:0/495107266 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4fe00a5b70 0x7f4fe013ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:27.129 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.128+0000 7f4fe81db700 1 --2- 172.21.15.59:0/495107266 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4fe00a5f40 0x7f4fe0135560 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:27.129 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.128+0000 7f4fe81db700 1 -- 172.21.15.59:0/495107266 >> 172.21.15.59:0/495107266 conn(0x7f4fe00631f0 msgr2=0x7f4fe0144db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:27.129 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.128+0000 7f4fe81db700 1 -- 172.21.15.59:0/495107266 shutdown_connections 2023-09-30T02:06:27.130 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.128+0000 7f4fe81db700 1 -- 172.21.15.59:0/495107266 wait complete. 2023-09-30T02:06:27.130 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.128+0000 7f4fe81db700 1 librados: shutdown 2023-09-30T02:06:27.141 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509447 for osd.4 2023-09-30T02:06:27.141 DEBUG:teuthology.parallel:result is None 2023-09-30T02:06:27.168 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.169+0000 7f3fad7fa700 1 -- 172.21.15.59:0/1323244403 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f3fb0012070 con 0x7f3fb8142290 2023-09-30T02:06:27.169 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.170+0000 7f3fad7fa700 1 --2- 172.21.15.59:0/1323244403 >> v2:172.21.15.110:6800/115870 conn(0x7f3f9c068890 0x7f3f9c06ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:27.169 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.170+0000 7f3fad7fa700 1 -- 172.21.15.59:0/1323244403 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f3fb0083a80 con 0x7f3fb8142290 2023-09-30T02:06:27.170 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.170+0000 7f3fad7fa700 10 client.4368.objecter ms_dispatch 0x7f3fb8001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:06:27.170 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.170+0000 7f3fad7fa700 3 client.4368.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:06:27.170 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.170+0000 7f3fad7fa700 3 client.4368.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:06:27.170 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.170+0000 7f3fad7fa700 20 client.4368.objecter dump_active .. 0 homeless 2023-09-30T02:06:27.170 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.170+0000 7f3fbd3c1700 1 --2- 172.21.15.59:0/1323244403 >> v2:172.21.15.110:6800/115870 conn(0x7f3f9c068890 0x7f3f9c06ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:27.170 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.170+0000 7f3fbd3c1700 10 monclient: get_auth_request con 0x7f3f9c068890 auth_method 0 2023-09-30T02:06:27.170 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.171+0000 7f3fad7fa700 10 client.4368.objecter ms_handle_connect 0x7f3f9c068890 2023-09-30T02:06:27.170 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.171+0000 7f3fbd3c1700 1 --2- 172.21.15.59:0/1323244403 >> v2:172.21.15.110:6800/115870 conn(0x7f3f9c068890 0x7f3f9c06ad40 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3fb40098e0 tx=0x7f3fb4016000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:27.199 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.200+0000 7f3fad7fa700 1 -- 172.21.15.59:0/1323244403 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3fb004e5a0 con 0x7f3fb8142290 2023-09-30T02:06:27.199 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.200+0000 7f3fad7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:27.199 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.200+0000 7f3fad7fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:06:27.221 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.221+0000 7f5bc8b6d700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2023-09-30T02:06:27.221 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.221+0000 7f5bc8b6d700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-09-30T02:06:27.221 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.221+0000 7f5bc8b6d700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:06:27.221 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.221+0000 7f5bc8b6d700 1 -- 172.21.15.59:0/1883605283 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f5bc4151fc0 con 0x7f5bc40a5f40 2023-09-30T02:06:27.230 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.230+0000 7f5bb27fc700 1 -- 172.21.15.59:0/1883605283 <== mon.2 v2:172.21.15.59:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5bb80577a0 con 0x7f5bc40a5f40 2023-09-30T02:06:27.230 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.231+0000 7f5bb27fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-09-30T02:06:27.230 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.231+0000 7f5bb27fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:06:27.231 INFO:teuthology.orchestra.run.smithi059.stdout:81604378630 2023-09-30T02:06:27.233 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.233+0000 7f5bc8b6d700 10 librados: watch_flush enter 2023-09-30T02:06:27.233 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.233+0000 7f5bc8b6d700 10 librados: watch_flush exit 2023-09-30T02:06:27.233 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.233+0000 7f5bc8b6d700 20 client.4343.objecter shutdown clearing up homeless session... 2023-09-30T02:06:27.233 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.233+0000 7f5bc8b6d700 10 client.4343.objecter successfully canceled tick 2023-09-30T02:06:27.233 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.233+0000 7f5bc8b6d700 1 -- 172.21.15.59:0/1883605283 >> v2:172.21.15.110:6800/115870 conn(0x7f5ba4068910 msgr2=0x7f5ba406adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:27.234 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.233+0000 7f5bc8b6d700 1 --2- 172.21.15.59:0/1883605283 >> v2:172.21.15.110:6800/115870 conn(0x7f5ba4068910 0x7f5ba406adc0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5bb40097c0 tx=0x7f5bb4006ce0 comp rx=0 tx=0).stop 2023-09-30T02:06:27.234 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.233+0000 7f5bc8b6d700 10 monclient: shutdown 2023-09-30T02:06:27.234 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.233+0000 7f5bc8b6d700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:27.234 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.233+0000 7f5bc8b6d700 1 -- 172.21.15.59:0/1883605283 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5bc40a5f40 msgr2=0x7f5bc413fcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:27.234 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.233+0000 7f5bc8b6d700 1 --2- 172.21.15.59:0/1883605283 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5bc40a5f40 0x7f5bc413fcf0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5bb8003080 tx=0x7f5bb8016000 comp rx=0 tx=0).stop 2023-09-30T02:06:27.234 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.233+0000 7f5bc8b6d700 1 -- 172.21.15.59:0/1883605283 shutdown_connections 2023-09-30T02:06:27.234 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.233+0000 7f5bc8b6d700 1 --2- 172.21.15.59:0/1883605283 >> v2:172.21.15.110:6800/115870 conn(0x7f5ba4068910 0x7f5ba406adc0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:27.234 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.233+0000 7f5bc8b6d700 1 --2- 172.21.15.59:0/1883605283 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5bc40a5b70 0x7f5bc413d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:27.234 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.233+0000 7f5bc8b6d700 1 --2- 172.21.15.59:0/1883605283 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5bc4142290 0x7f5bc413df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:27.234 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.233+0000 7f5bc8b6d700 1 --2- 172.21.15.59:0/1883605283 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5bc40a5f40 0x7f5bc413fcf0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:27.235 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.233+0000 7f5bc8b6d700 1 -- 172.21.15.59:0/1883605283 >> 172.21.15.59:0/1883605283 conn(0x7f5bc40631f0 msgr2=0x7f5bc40b0be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:27.235 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.233+0000 7f5bc8b6d700 1 -- 172.21.15.59:0/1883605283 shutdown_connections 2023-09-30T02:06:27.235 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.233+0000 7f5bc8b6d700 1 -- 172.21.15.59:0/1883605283 wait complete. 2023-09-30T02:06:27.235 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.233+0000 7f5bc8b6d700 1 librados: shutdown 2023-09-30T02:06:27.247 INFO:tasks.ceph.ceph_manager.ceph:need seq 81604378630 got 81604378630 for osd.3 2023-09-30T02:06:27.247 DEBUG:teuthology.parallel:result is None 2023-09-30T02:06:27.291 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.291+0000 7f1c93fff700 1 -- 172.21.15.59:0/2892307729 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f1c9c054ff0 con 0x7f1cac0a5b70 2023-09-30T02:06:27.291 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.291+0000 7f1c93fff700 10 client.4371.objecter ms_dispatch 0x7f1cac001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:06:27.291 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.291+0000 7f1c93fff700 3 client.4371.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:06:27.291 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.291+0000 7f1c93fff700 3 client.4371.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:06:27.291 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.291+0000 7f1c93fff700 20 client.4371.objecter dump_active .. 0 homeless 2023-09-30T02:06:27.316 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.317+0000 7fa6bf7fe700 1 -- 172.21.15.59:0/939460102 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa6cc057090 con 0x7fa6d80a5f40 2023-09-30T02:06:27.316 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.317+0000 7fa6bf7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-09-30T02:06:27.317 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.317+0000 7fa6bf7fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:06:27.317 INFO:teuthology.orchestra.run.smithi059.stdout:77309411335 2023-09-30T02:06:27.321 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.321+0000 7fa6ddf9f700 10 librados: watch_flush enter 2023-09-30T02:06:27.321 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.321+0000 7fa6ddf9f700 10 librados: watch_flush exit 2023-09-30T02:06:27.321 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.321+0000 7fa6ddf9f700 20 client.4359.objecter shutdown clearing up homeless session... 2023-09-30T02:06:27.321 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.321+0000 7fa6ddf9f700 10 client.4359.objecter successfully canceled tick 2023-09-30T02:06:27.321 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.321+0000 7fa6ddf9f700 1 -- 172.21.15.59:0/939460102 >> v2:172.21.15.110:6800/115870 conn(0x7fa6c4068960 msgr2=0x7fa6c406ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:27.321 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.321+0000 7fa6ddf9f700 1 --2- 172.21.15.59:0/939460102 >> v2:172.21.15.110:6800/115870 conn(0x7fa6c4068960 0x7fa6c406ae10 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa6c800ad10 tx=0x7fa6c8009370 comp rx=0 tx=0).stop 2023-09-30T02:06:27.321 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.321+0000 7fa6ddf9f700 10 monclient: shutdown 2023-09-30T02:06:27.321 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.321+0000 7fa6ddf9f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:27.321 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.321+0000 7fa6ddf9f700 1 -- 172.21.15.59:0/939460102 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa6d80a5f40 msgr2=0x7fa6d8151770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:27.321 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.321+0000 7fa6ddf9f700 1 --2- 172.21.15.59:0/939460102 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa6d80a5f40 0x7fa6d8151770 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa6cc0030c0 tx=0x7fa6cc016000 comp rx=0 tx=0).stop 2023-09-30T02:06:27.322 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.321+0000 7fa6ddf9f700 1 -- 172.21.15.59:0/939460102 shutdown_connections 2023-09-30T02:06:27.322 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.321+0000 7fa6ddf9f700 1 --2- 172.21.15.59:0/939460102 >> v2:172.21.15.110:6800/115870 conn(0x7fa6c4068960 0x7fa6c406ae10 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:27.322 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.321+0000 7fa6ddf9f700 1 --2- 172.21.15.59:0/939460102 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa6d80a5b70 0x7fa6d8151230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:27.322 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.321+0000 7fa6ddf9f700 1 --2- 172.21.15.59:0/939460102 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa6d8142290 0x7fa6d813fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:27.322 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.321+0000 7fa6ddf9f700 1 --2- 172.21.15.59:0/939460102 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa6d80a5f40 0x7fa6d8151770 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:27.322 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.321+0000 7fa6ddf9f700 1 -- 172.21.15.59:0/939460102 >> 172.21.15.59:0/939460102 conn(0x7fa6d80631f0 msgr2=0x7fa6d80af530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:27.322 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.321+0000 7fa6ddf9f700 1 -- 172.21.15.59:0/939460102 shutdown_connections 2023-09-30T02:06:27.322 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.321+0000 7fa6ddf9f700 1 -- 172.21.15.59:0/939460102 wait complete. 2023-09-30T02:06:27.322 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.321+0000 7fa6ddf9f700 1 librados: shutdown 2023-09-30T02:06:27.335 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411334 got 77309411335 for osd.0 2023-09-30T02:06:27.335 DEBUG:teuthology.parallel:result is None 2023-09-30T02:06:27.449 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.449+0000 7f3fbf625700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2023-09-30T02:06:27.449 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.449+0000 7f3fbf625700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2023-09-30T02:06:27.449 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.449+0000 7f3fbf625700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:06:27.449 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.449+0000 7f3fbf625700 1 -- 172.21.15.59:0/1323244403 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f3fb8064a00 con 0x7f3fb8142290 2023-09-30T02:06:27.600 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.600+0000 7f1c93fff700 1 -- 172.21.15.59:0/2892307729 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1c9c051330 con 0x7f1cac0a5b70 2023-09-30T02:06:27.600 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.601+0000 7f1c93fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:27.600 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.601+0000 7f1c93fff700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:06:27.628 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.629+0000 7f3fad7fa700 1 -- 172.21.15.59:0/1323244403 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3fb0054e30 con 0x7f3fb8142290 2023-09-30T02:06:27.629 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.629+0000 7f3fad7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2023-09-30T02:06:27.629 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.629+0000 7f3fad7fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:06:27.629 INFO:teuthology.orchestra.run.smithi059.stdout:55834574855 2023-09-30T02:06:27.631 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.631+0000 7f3fbf625700 10 librados: watch_flush enter 2023-09-30T02:06:27.631 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.631+0000 7f3fbf625700 10 librados: watch_flush exit 2023-09-30T02:06:27.631 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.631+0000 7f3fbf625700 20 client.4368.objecter shutdown clearing up homeless session... 2023-09-30T02:06:27.631 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.631+0000 7f3fbf625700 10 client.4368.objecter successfully canceled tick 2023-09-30T02:06:27.631 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.631+0000 7f3fbf625700 1 -- 172.21.15.59:0/1323244403 >> v2:172.21.15.110:6800/115870 conn(0x7f3f9c068890 msgr2=0x7f3f9c06ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:27.632 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.631+0000 7f3fbf625700 1 --2- 172.21.15.59:0/1323244403 >> v2:172.21.15.110:6800/115870 conn(0x7f3f9c068890 0x7f3f9c06ad40 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3fb40098e0 tx=0x7f3fb4016000 comp rx=0 tx=0).stop 2023-09-30T02:06:27.632 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.631+0000 7f3fbf625700 10 monclient: shutdown 2023-09-30T02:06:27.632 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.631+0000 7f3fbf625700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:27.632 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.631+0000 7f3fbf625700 1 -- 172.21.15.59:0/1323244403 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3fb8142290 msgr2=0x7f3fb81562c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:27.632 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.631+0000 7f3fbf625700 1 --2- 172.21.15.59:0/1323244403 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3fb8142290 0x7f3fb81562c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f3fb000bbb0 tx=0x7f3fb0009840 comp rx=0 tx=0).stop 2023-09-30T02:06:27.632 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.631+0000 7f3fbf625700 1 -- 172.21.15.59:0/1323244403 shutdown_connections 2023-09-30T02:06:27.632 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.631+0000 7f3fbf625700 1 --2- 172.21.15.59:0/1323244403 >> v2:172.21.15.110:6800/115870 conn(0x7f3f9c068890 0x7f3f9c06ad40 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:27.632 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.632+0000 7f3fbf625700 1 --2- 172.21.15.59:0/1323244403 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3fb80a5b70 0x7f3fb813c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:27.632 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.632+0000 7f3fbf625700 1 --2- 172.21.15.59:0/1323244403 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3fb8142290 0x7f3fb81562c0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:27.632 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.632+0000 7f3fbf625700 1 --2- 172.21.15.59:0/1323244403 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3fb80a5f40 0x7f3fb813ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:27.632 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.632+0000 7f3fbf625700 1 -- 172.21.15.59:0/1323244403 >> 172.21.15.59:0/1323244403 conn(0x7f3fb80631f0 msgr2=0x7f3fb80b4a60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:27.632 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.632+0000 7f3fbf625700 1 -- 172.21.15.59:0/1323244403 shutdown_connections 2023-09-30T02:06:27.633 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.632+0000 7f3fbf625700 1 -- 172.21.15.59:0/1323244403 wait complete. 2023-09-30T02:06:27.633 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.632+0000 7f3fbf625700 1 librados: shutdown 2023-09-30T02:06:27.644 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574855 for osd.6 2023-09-30T02:06:27.644 DEBUG:teuthology.parallel:result is None 2023-09-30T02:06:27.811 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.812+0000 7f1cb29c0700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2023-09-30T02:06:27.812 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.812+0000 7f1cb29c0700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-09-30T02:06:27.812 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.812+0000 7f1cb29c0700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:06:27.812 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.812+0000 7f1cb29c0700 1 -- 172.21.15.59:0/2892307729 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f1cac139b30 con 0x7f1cac0a5b70 2023-09-30T02:06:27.816 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.817+0000 7f1c93fff700 1 -- 172.21.15.59:0/2892307729 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1cac139b30 con 0x7f1cac0a5b70 2023-09-30T02:06:27.816 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.817+0000 7f1c93fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-09-30T02:06:27.816 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.817+0000 7f1c93fff700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:06:27.816 INFO:teuthology.orchestra.run.smithi059.stdout:77309411334 2023-09-30T02:06:27.820 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.820+0000 7f1cb29c0700 10 librados: watch_flush enter 2023-09-30T02:06:27.820 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.820+0000 7f1cb29c0700 10 librados: watch_flush exit 2023-09-30T02:06:27.820 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.820+0000 7f1cb29c0700 20 client.4371.objecter shutdown clearing up homeless session... 2023-09-30T02:06:27.820 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.820+0000 7f1cb29c0700 10 client.4371.objecter successfully canceled tick 2023-09-30T02:06:27.820 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.820+0000 7f1cb29c0700 1 -- 172.21.15.59:0/2892307729 >> v2:172.21.15.110:6800/115870 conn(0x7f1c98068990 msgr2=0x7f1c9806ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:27.820 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.820+0000 7f1cb29c0700 1 --2- 172.21.15.59:0/2892307729 >> v2:172.21.15.110:6800/115870 conn(0x7f1c98068990 0x7f1c9806ae40 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1cac144690 tx=0x7f1ca0017000 comp rx=0 tx=0).stop 2023-09-30T02:06:27.821 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.820+0000 7f1cb29c0700 10 monclient: shutdown 2023-09-30T02:06:27.821 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.820+0000 7f1cb29c0700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:27.821 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.820+0000 7f1cb29c0700 1 -- 172.21.15.59:0/2892307729 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1cac0a5b70 msgr2=0x7f1cac143e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:27.821 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.820+0000 7f1cb29c0700 1 --2- 172.21.15.59:0/2892307729 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1cac0a5b70 0x7f1cac143e90 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c00c5c0 tx=0x7f1c9c00c980 comp rx=0 tx=0).stop 2023-09-30T02:06:27.821 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.820+0000 7f1cb29c0700 1 -- 172.21.15.59:0/2892307729 shutdown_connections 2023-09-30T02:06:27.821 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.820+0000 7f1cb29c0700 1 --2- 172.21.15.59:0/2892307729 >> v2:172.21.15.110:6800/115870 conn(0x7f1c98068990 0x7f1c9806ae40 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:27.821 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.820+0000 7f1cb29c0700 1 --2- 172.21.15.59:0/2892307729 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1cac142290 0x7f1cac13f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:27.821 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.820+0000 7f1cb29c0700 1 --2- 172.21.15.59:0/2892307729 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1cac0a5b70 0x7f1cac143e90 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:27.821 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.820+0000 7f1cb29c0700 1 --2- 172.21.15.59:0/2892307729 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1cac0a5f40 0x7f1cac139140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:27.821 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.820+0000 7f1cb29c0700 1 -- 172.21.15.59:0/2892307729 >> 172.21.15.59:0/2892307729 conn(0x7f1cac0631f0 msgr2=0x7f1cac070f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:27.822 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.820+0000 7f1cb29c0700 1 -- 172.21.15.59:0/2892307729 shutdown_connections 2023-09-30T02:06:27.822 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.820+0000 7f1cb29c0700 1 -- 172.21.15.59:0/2892307729 wait complete. 2023-09-30T02:06:27.822 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.820+0000 7f1cb29c0700 1 librados: shutdown 2023-09-30T02:06:27.833 INFO:tasks.ceph.ceph_manager.ceph:need seq 77309411333 got 77309411334 for osd.2 2023-09-30T02:06:27.833 DEBUG:teuthology.parallel:result is None 2023-09-30T02:06:27.833 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-09-30T02:06:27.834 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T02:06:27.964 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.964+0000 7fa889f0a700 10 monclient: get_monmap_and_config 2023-09-30T02:06:27.964 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.964+0000 7fa889f0a700 10 monclient: build_initial_monmap 2023-09-30T02:06:27.964 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.964+0000 7fa889f0a700 10 monclient: monmap: 2023-09-30T02:06:27.964 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:27.964 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:27.964 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:06:27.965139+0000 2023-09-30T02:06:27.964 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:06:27.965139+0000 2023-09-30T02:06:27.964 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:27.964 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:27.965 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:06:27.965 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:06:27.965 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:06:27.965 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:06:27.965 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:06:27.965 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:27.966 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.965+0000 7fa889f0a700 1 Processor -- start 2023-09-30T02:06:27.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.965+0000 7fa889f0a700 1 -- start start 2023-09-30T02:06:27.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.965+0000 7fa889f0a700 10 monclient: init 2023-09-30T02:06:27.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.966+0000 7fa889f0a700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:27.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.966+0000 7fa889f0a700 10 monclient: _add_conns ranks=[1,3,4,2,0] 2023-09-30T02:06:27.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.966+0000 7fa889f0a700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa8840b2f70 addr v1:172.21.15.110:6789/0 2023-09-30T02:06:27.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.966+0000 7fa889f0a700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa88406dd20 0x7fa88406e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:27.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.966+0000 7fa889f0a700 10 monclient(hunting): picked mon.noname-a con 0x7fa88406dd20 addr v2:172.21.15.59:3300/0 2023-09-30T02:06:27.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.966+0000 7fa889f0a700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fa8840b3340 0x7fa8840abd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:27.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.966+0000 7fa889f0a700 10 monclient(hunting): picked mon.noname-c con 0x7fa8840b3340 addr v2:172.21.15.110:3300/0 2023-09-30T02:06:27.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.966+0000 7fa889f0a700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa884139cc0 con 0x7fa8840b2f70 2023-09-30T02:06:27.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.966+0000 7fa889f0a700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:27.968 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.966+0000 7fa889f0a700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa88406f3c0 con 0x7fa88406dd20 2023-09-30T02:06:27.968 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.966+0000 7fa889f0a700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:27.968 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.966+0000 7fa889f0a700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa88406f6a0 con 0x7fa8840b3340 2023-09-30T02:06:27.968 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.966+0000 7fa889f0a700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:27.968 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.966+0000 7fa889f0a700 10 monclient(hunting): authenticate will time out at 1833.268370s 2023-09-30T02:06:27.968 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.967+0000 7fa882ffd700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa88406dd20 0x7fa88406e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:27.968 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.967+0000 7fa883fff700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fa8840b3340 0x7fa8840abd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:27.968 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.967+0000 7fa882ffd700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa88406dd20 0x7fa88406e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:33986/0 (socket says 172.21.15.59:33986) 2023-09-30T02:06:27.968 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.967+0000 7fa882ffd700 1 -- 172.21.15.59:0/537709926 learned_addr learned my addr 172.21.15.59:0/537709926 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:06:27.968 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.967+0000 7fa882ffd700 10 monclient(hunting): get_auth_request con 0x7fa88406dd20 auth_method 0 2023-09-30T02:06:27.968 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.967+0000 7fa882ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:27.968 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.967+0000 7fa882ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:27.968 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.967+0000 7fa882ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:27.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.968+0000 7fa883fff700 10 monclient(hunting): get_auth_request con 0x7fa8840b3340 auth_method 0 2023-09-30T02:06:27.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.968+0000 7fa883fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:27.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.968+0000 7fa883fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:27.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.968+0000 7fa883fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:27.970 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.968+0000 7fa882ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:27.970 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.968+0000 7fa882ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:27.970 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.968+0000 7fa882ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:27.970 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.969+0000 7fa883fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:27.970 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.969+0000 7fa883fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:27.970 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.969+0000 7fa883fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:27.971 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.971+0000 7fa882ffd700 10 monclient(hunting): handle_auth_done global_id 4374 payload 274 2023-09-30T02:06:27.971 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.971+0000 7fa882ffd700 1 -- 172.21.15.59:0/537709926 >> v2:172.21.15.110:3300/0 conn(0x7fa8840b3340 msgr2=0x7fa8840abd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:06:27.971 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.971+0000 7fa882ffd700 1 --2- 172.21.15.59:0/537709926 >> v2:172.21.15.110:3300/0 conn(0x7fa8840b3340 0x7fa8840abd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:27.971 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.971+0000 7fa882ffd700 1 -- 172.21.15.59:0/537709926 >> v1:172.21.15.110:6789/0 conn(0x7fa8840b2f70 legacy=0x7fa88413f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:27.971 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.971+0000 7fa882ffd700 10 monclient: _finish_hunting 0 2023-09-30T02:06:27.971 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.971+0000 7fa882ffd700 1 monclient: found mon.noname-a 2023-09-30T02:06:27.971 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.971+0000 7fa882ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:27.971 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.971+0000 7fa882ffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.59:3300/0 2023-09-30T02:06:27.971 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.971+0000 7fa882ffd700 1 -- 172.21.15.59:0/537709926 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa88406f980 con 0x7fa88406dd20 2023-09-30T02:06:27.972 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.973+0000 7fa882ffd700 1 --2- 172.21.15.59:0/537709926 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa88406dd20 0x7fa88406e6d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa88406f040 tx=0x7fa86c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25fbb736498441de server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:27.977 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.977+0000 7fa8827fc700 1 -- 172.21.15.59:0/537709926 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa86c009e70 con 0x7fa88406dd20 2023-09-30T02:06:27.977 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.977+0000 7fa8827fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:27.977 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.977+0000 7fa8827fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T02:06:27.977 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.977+0000 7fa8827fc700 10 monclient: dump: 2023-09-30T02:06:27.977 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:27.977 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:27.977 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:27.977 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:27.977 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:27.977 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:27.977 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:27.978 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:27.978 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:27.978 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:27.978 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.977+0000 7fa8827fc700 10 monclient: _finish_auth 0 2023-09-30T02:06:27.978 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.977+0000 7fa8827fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:27.978 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.977+0000 7fa889f0a700 5 monclient: authenticate success, global_id 4374 2023-09-30T02:06:27.978 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.977+0000 7fa889f0a700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:06:27.978 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.977+0000 7fa8827fc700 1 -- 172.21.15.59:0/537709926 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa86c00b860 con 0x7fa88406dd20 2023-09-30T02:06:27.978 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.977+0000 7fa8827fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:27.978 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.977+0000 7fa889f0a700 10 monclient: get_monmap_and_config success 2023-09-30T02:06:27.978 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.977+0000 7fa8827fc700 1 -- 172.21.15.59:0/537709926 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa86c00bc00 con 0x7fa88406dd20 2023-09-30T02:06:27.978 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.977+0000 7fa889f0a700 4 set_mon_vals no callback set 2023-09-30T02:06:27.979 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.977+0000 7fa8827fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:27.979 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.977+0000 7fa8827fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T02:06:27.979 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.977+0000 7fa8827fc700 10 monclient: dump: 2023-09-30T02:06:27.979 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:27.979 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:27.979 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:27.979 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:27.979 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:27.979 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:27.979 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:27.979 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:27.979 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:27.979 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:27.982 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.978+0000 7fa889f0a700 10 monclient: shutdown 2023-09-30T02:06:27.983 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.978+0000 7fa889f0a700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:27.983 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.978+0000 7fa889f0a700 1 -- 172.21.15.59:0/537709926 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa88406dd20 msgr2=0x7fa88406e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:27.983 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.978+0000 7fa889f0a700 1 --2- 172.21.15.59:0/537709926 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa88406dd20 0x7fa88406e6d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa88406f040 tx=0x7fa86c009250 comp rx=0 tx=0).stop 2023-09-30T02:06:27.983 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.978+0000 7fa889f0a700 1 -- 172.21.15.59:0/537709926 shutdown_connections 2023-09-30T02:06:27.983 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.978+0000 7fa889f0a700 1 --2- 172.21.15.59:0/537709926 >> v2:172.21.15.110:3300/0 conn(0x7fa8840b3340 0x7fa8840abd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:27.983 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.978+0000 7fa889f0a700 1 --2- 172.21.15.59:0/537709926 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa88406dd20 0x7fa88406e6d0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:27.983 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.978+0000 7fa889f0a700 1 -- 172.21.15.59:0/537709926 >> 172.21.15.59:0/537709926 conn(0x7fa8840637c0 msgr2=0x7fa8840b2860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:27.983 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.978+0000 7fa889f0a700 1 -- 172.21.15.59:0/537709926 shutdown_connections 2023-09-30T02:06:27.983 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.978+0000 7fa889f0a700 1 -- 172.21.15.59:0/537709926 wait complete. 2023-09-30T02:06:27.983 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.979+0000 7fa889f0a700 10 monclient: build_initial_monmap 2023-09-30T02:06:27.983 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.979+0000 7fa889f0a700 10 monclient: monmap: 2023-09-30T02:06:27.984 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:27.984 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:06:27.984 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:06:27.984 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:06:27.984 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:27.984 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:06:27.984 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:06:27.984 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:06:27.984 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:06:27.984 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:27.984 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.979+0000 7fa889f0a700 1 librados: starting msgr at 2023-09-30T02:06:27.984 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.979+0000 7fa889f0a700 1 librados: starting objecter 2023-09-30T02:06:27.985 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.979+0000 7fa889f0a700 1 Processor -- start 2023-09-30T02:06:27.985 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.979+0000 7fa889f0a700 1 -- start start 2023-09-30T02:06:27.985 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.979+0000 7fa889f0a700 1 librados: setting wanted keys 2023-09-30T02:06:27.985 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.979+0000 7fa889f0a700 1 librados: calling monclient init 2023-09-30T02:06:27.985 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.979+0000 7fa889f0a700 10 monclient: init 2023-09-30T02:06:27.985 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.980+0000 7fa889f0a700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:27.985 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.980+0000 7fa889f0a700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T02:06:27.985 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.980+0000 7fa889f0a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa88406dd20 0x7fa884151160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:27.985 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.980+0000 7fa889f0a700 10 monclient(hunting): picked mon.noname-a con 0x7fa88406dd20 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:06:27.985 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.980+0000 7fa889f0a700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa8840b2f70 0x7fa884069780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:27.985 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.980+0000 7fa889f0a700 10 monclient(hunting): picked mon.noname-b con 0x7fa8840b2f70 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:06:27.985 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.980+0000 7fa889f0a700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa8840b3340 0x7fa884069cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:27.986 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.980+0000 7fa889f0a700 10 monclient(hunting): picked mon.noname-c con 0x7fa8840b3340 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:27.986 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.980+0000 7fa889f0a700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:27.986 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.980+0000 7fa889f0a700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa88414b600 con 0x7fa88406dd20 2023-09-30T02:06:27.986 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.980+0000 7fa889f0a700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:27.986 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.980+0000 7fa889f0a700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa88406a200 con 0x7fa8840b2f70 2023-09-30T02:06:27.986 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.980+0000 7fa889f0a700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:27.986 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.980+0000 7fa889f0a700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8840b0690 con 0x7fa8840b3340 2023-09-30T02:06:27.986 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.980+0000 7fa889f0a700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:27.986 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.980+0000 7fa889f0a700 10 monclient(hunting): authenticate will time out at 1833.282492s 2023-09-30T02:06:27.986 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.981+0000 7fa882ffd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa8840b2f70 0x7fa884069780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:27.986 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.981+0000 7fa883fff700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa8840b3340 0x7fa884069cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:27.987 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.981+0000 7fa882ffd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa8840b2f70 0x7fa884069780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:38612/0 (socket says 172.21.15.59:38612) 2023-09-30T02:06:27.987 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.981+0000 7fa882ffd700 1 -- 172.21.15.59:0/2848946847 learned_addr learned my addr 172.21.15.59:0/2848946847 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:06:27.987 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.981+0000 7fa882ffd700 10 monclient(hunting): get_auth_request con 0x7fa8840b2f70 auth_method 0 2023-09-30T02:06:27.987 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.981+0000 7fa882ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:27.987 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.981+0000 7fa882ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:27.987 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.981+0000 7fa882ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:27.987 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.981+0000 7fa883fff700 10 monclient(hunting): get_auth_request con 0x7fa8840b3340 auth_method 0 2023-09-30T02:06:27.987 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.981+0000 7fa883fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:27.987 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.981+0000 7fa883fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:27.987 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.981+0000 7fa883fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:27.987 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.981+0000 7fa8837fe700 1 --2- 172.21.15.59:0/2848946847 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa88406dd20 0x7fa884151160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:27.987 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.981+0000 7fa882ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:27.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.981+0000 7fa882ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:27.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.981+0000 7fa882ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:27.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.982+0000 7fa8837fe700 10 monclient(hunting): get_auth_request con 0x7fa88406dd20 auth_method 0 2023-09-30T02:06:27.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.982+0000 7fa8837fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:27.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.982+0000 7fa8837fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:27.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.982+0000 7fa8837fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:27.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.982+0000 7fa883fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:27.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.982+0000 7fa883fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:27.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.982+0000 7fa883fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:27.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.982+0000 7fa8837fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:27.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.982+0000 7fa8837fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:27.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.982+0000 7fa8837fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:27.989 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.984+0000 7fa882ffd700 10 monclient(hunting): handle_auth_done global_id 4399 payload 824 2023-09-30T02:06:27.990 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.985+0000 7fa882ffd700 1 -- 172.21.15.59:0/2848946847 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa8840b3340 msgr2=0x7fa884069cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:27.990 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.985+0000 7fa882ffd700 1 --2- 172.21.15.59:0/2848946847 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa8840b3340 0x7fa884069cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:27.990 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.985+0000 7fa882ffd700 1 -- 172.21.15.59:0/2848946847 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa88406dd20 msgr2=0x7fa884151160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:27.990 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.985+0000 7fa882ffd700 1 --2- 172.21.15.59:0/2848946847 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa88406dd20 0x7fa884151160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:27.990 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.985+0000 7fa882ffd700 10 monclient: _finish_hunting 0 2023-09-30T02:06:27.990 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.985+0000 7fa882ffd700 1 monclient: found mon.noname-b 2023-09-30T02:06:27.990 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.985+0000 7fa882ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:27.990 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.985+0000 7fa882ffd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:06:27.990 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.985+0000 7fa882ffd700 1 -- 172.21.15.59:0/2848946847 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa86c0094f0 con 0x7fa8840b2f70 2023-09-30T02:06:27.990 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.987+0000 7fa882ffd700 1 --2- 172.21.15.59:0/2848946847 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa8840b2f70 0x7fa884069780 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa86c006fd0 tx=0x7fa86c017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:27.990 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.987+0000 7fa87b7fe700 10 client.?.objecter ms_handle_connect 0x7fa8840b2f70 2023-09-30T02:06:27.991 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.987+0000 7fa87b7fe700 10 client.?.objecter resend_mon_ops 2023-09-30T02:06:27.992 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.991+0000 7fa87b7fe700 1 -- 172.21.15.59:0/2848946847 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa86c00b9d0 con 0x7fa8840b2f70 2023-09-30T02:06:27.992 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.991+0000 7fa87b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:27.992 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.991+0000 7fa87b7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:06:27.992 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.991+0000 7fa87b7fe700 10 monclient: dump: 2023-09-30T02:06:27.992 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:27.992 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:27.993 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:27.993 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:27.993 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:27.993 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:27.993 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:27.993 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:27.993 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:27.993 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:27.993 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.991+0000 7fa87b7fe700 10 monclient: _finish_auth 0 2023-09-30T02:06:27.993 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.991+0000 7fa87b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:27.993 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.991+0000 7fa87b7fe700 1 -- 172.21.15.59:0/2848946847 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa86c00bb30 con 0x7fa8840b2f70 2023-09-30T02:06:27.994 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.991+0000 7fa87b7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:27.994 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.991+0000 7fa87b7fe700 1 -- 172.21.15.59:0/2848946847 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa86c011670 con 0x7fa8840b2f70 2023-09-30T02:06:27.994 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.991+0000 7fa87b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:27.994 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.991+0000 7fa87b7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:06:27.994 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.991+0000 7fa87b7fe700 10 monclient: dump: 2023-09-30T02:06:27.994 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:27.994 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:27.994 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:27.994 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:27.994 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:27.994 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:27.994 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:27.995 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:27.995 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:27.995 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:27.995 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.991+0000 7fa881ffb700 4 set_mon_vals no callback set 2023-09-30T02:06:27.995 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.991+0000 7fa889f0a700 5 monclient: authenticate success, global_id 4399 2023-09-30T02:06:27.995 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.991+0000 7fa889f0a700 10 monclient: _renew_subs 2023-09-30T02:06:27.995 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.991+0000 7fa889f0a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:27.995 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.991+0000 7fa889f0a700 1 -- 172.21.15.59:0/2848946847 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8840692e0 con 0x7fa8840b2f70 2023-09-30T02:06:27.995 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.991+0000 7fa889f0a700 10 client.4399.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:06:27.995 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.991+0000 7fa889f0a700 10 monclient: _renew_subs 2023-09-30T02:06:27.995 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.991+0000 7fa889f0a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:27.995 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.991+0000 7fa889f0a700 1 -- 172.21.15.59:0/2848946847 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa884151c00 con 0x7fa8840b2f70 2023-09-30T02:06:27.996 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.991+0000 7fa889f0a700 1 librados: init done 2023-09-30T02:06:27.996 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.992+0000 7fa889f0a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:27.996 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.992+0000 7fa889f0a700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:27.996 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.992+0000 7fa889f0a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:27.996 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:27.992+0000 7fa889f0a700 1 -- 172.21.15.59:0/2848946847 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa884064a00 con 0x7fa8840b2f70 2023-09-30T02:06:28.242 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.242+0000 7fa87b7fe700 1 -- 172.21.15.59:0/2848946847 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fa86c011a60 con 0x7fa8840b2f70 2023-09-30T02:06:28.243 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.243+0000 7fa87b7fe700 1 --2- 172.21.15.59:0/2848946847 >> v2:172.21.15.110:6800/115870 conn(0x7fa8640689e0 0x7fa86406ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:28.243 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.243+0000 7fa87b7fe700 1 -- 172.21.15.59:0/2848946847 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fa86c085f50 con 0x7fa8840b2f70 2023-09-30T02:06:28.243 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.243+0000 7fa87b7fe700 10 client.4399.objecter ms_dispatch 0x7fa884001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:06:28.243 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.243+0000 7fa87b7fe700 3 client.4399.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:06:28.243 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.243+0000 7fa87b7fe700 3 client.4399.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:06:28.244 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.243+0000 7fa87b7fe700 20 client.4399.objecter dump_active .. 0 homeless 2023-09-30T02:06:28.244 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.243+0000 7fa8837fe700 1 --2- 172.21.15.59:0/2848946847 >> v2:172.21.15.110:6800/115870 conn(0x7fa8640689e0 0x7fa86406ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:28.244 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.243+0000 7fa8837fe700 10 monclient: get_auth_request con 0x7fa8640689e0 auth_method 0 2023-09-30T02:06:28.244 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.244+0000 7fa8837fe700 1 --2- 172.21.15.59:0/2848946847 >> v2:172.21.15.110:6800/115870 conn(0x7fa8640689e0 0x7fa86406ae90 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa874003aa0 tx=0x7fa874000bc0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:28.244 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.244+0000 7fa87b7fe700 10 client.4399.objecter ms_handle_connect 0x7fa8640689e0 2023-09-30T02:06:28.251 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.252+0000 7fa87b7fe700 1 -- 172.21.15.59:0/2848946847 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa86c0537f0 con 0x7fa8840b2f70 2023-09-30T02:06:28.251 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.252+0000 7fa87b7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:28.251 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.252+0000 7fa87b7fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:06:28.491 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.491+0000 7fa889f0a700 1 -- 172.21.15.59:0/2848946847 --> v2:172.21.15.110:6800/115870 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa88406e1b0 con 0x7fa8640689e0 2023-09-30T02:06:28.493 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.493+0000 7fa87b7fe700 1 -- 172.21.15.59:0/2848946847 <== mgr.4108 v2:172.21.15.110:6800/115870 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+47789 (secure 0 0 0) 0x7fa88406e1b0 con 0x7fa8640689e0 2023-09-30T02:06:28.494 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:06:28.497 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.496+0000 7fa889f0a700 10 librados: watch_flush enter 2023-09-30T02:06:28.497 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.496+0000 7fa889f0a700 10 librados: watch_flush exit 2023-09-30T02:06:28.497 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.496+0000 7fa889f0a700 20 client.4399.objecter shutdown clearing up homeless session... 2023-09-30T02:06:28.497 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.496+0000 7fa889f0a700 10 client.4399.objecter successfully canceled tick 2023-09-30T02:06:28.497 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.496+0000 7fa889f0a700 1 -- 172.21.15.59:0/2848946847 >> v2:172.21.15.110:6800/115870 conn(0x7fa8640689e0 msgr2=0x7fa86406ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:28.497 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.496+0000 7fa889f0a700 1 --2- 172.21.15.59:0/2848946847 >> v2:172.21.15.110:6800/115870 conn(0x7fa8640689e0 0x7fa86406ae90 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa874003aa0 tx=0x7fa874000bc0 comp rx=0 tx=0).stop 2023-09-30T02:06:28.497 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.496+0000 7fa889f0a700 10 monclient: shutdown 2023-09-30T02:06:28.497 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.496+0000 7fa889f0a700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:28.497 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.496+0000 7fa889f0a700 1 -- 172.21.15.59:0/2848946847 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa8840b2f70 msgr2=0x7fa884069780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:28.497 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.496+0000 7fa889f0a700 1 --2- 172.21.15.59:0/2848946847 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa8840b2f70 0x7fa884069780 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa86c006fd0 tx=0x7fa86c017000 comp rx=0 tx=0).stop 2023-09-30T02:06:28.497 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.496+0000 7fa889f0a700 1 -- 172.21.15.59:0/2848946847 shutdown_connections 2023-09-30T02:06:28.498 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.496+0000 7fa889f0a700 1 --2- 172.21.15.59:0/2848946847 >> v2:172.21.15.110:6800/115870 conn(0x7fa8640689e0 0x7fa86406ae90 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:28.498 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.496+0000 7fa889f0a700 1 --2- 172.21.15.59:0/2848946847 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa8840b3340 0x7fa884069cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:28.498 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.496+0000 7fa889f0a700 1 --2- 172.21.15.59:0/2848946847 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa88406dd20 0x7fa884151160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:28.498 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.496+0000 7fa889f0a700 1 --2- 172.21.15.59:0/2848946847 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa8840b2f70 0x7fa884069780 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:28.498 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.496+0000 7fa889f0a700 1 -- 172.21.15.59:0/2848946847 >> 172.21.15.59:0/2848946847 conn(0x7fa8840637c0 msgr2=0x7fa8840aa4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:28.498 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.496+0000 7fa889f0a700 1 -- 172.21.15.59:0/2848946847 shutdown_connections 2023-09-30T02:06:28.498 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.496+0000 7fa889f0a700 1 -- 172.21.15.59:0/2848946847 wait complete. 2023-09-30T02:06:28.498 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.496+0000 7fa889f0a700 1 librados: shutdown 2023-09-30T02:06:28.499 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2023-09-30T02:06:28.510 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":42,"stamp":"2023-09-30T02:06:27.585062+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":43464,"kb_used_data":2448,"kb_used_omap":0,"kb_used_meta":40576,"kb_avail":754931256,"statfs":{"total":773094113280,"available":773049606144,"internally_reserved":0,"allocated":2506752,"data_stored":1396000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":41549824},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":14,"num_write_kb":262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.355335"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T02:06:18.350818+0000","last_change":"2023-09-30T02:06:18.210390+0000","last_active":"2023-09-30T02:06:18.350818+0000","last_peered":"2023-09-30T02:06:18.350818+0000","last_clean":"2023-09-30T02:06:18.350818+0000","last_became_active":"2023-09-30T02:06:18.198153+0000","last_became_peered":"2023-09-30T02:06:18.198153+0000","last_unstale":"2023-09-30T02:06:18.350818+0000","last_undegraded":"2023-09-30T02:06:18.350818+0000","last_fullsized":"2023-09-30T02:06:18.350818+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_clean_scrub_stamp":"2023-09-30T02:06:15.800912+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:42:32.739379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T02:06:18.437839+0000","last_change":"2023-09-30T02:06:18.262708+0000","last_active":"2023-09-30T02:06:18.437839+0000","last_peered":"2023-09-30T02:06:18.437839+0000","last_clean":"2023-09-30T02:06:18.437839+0000","last_became_active":"2023-09-30T02:06:18.224464+0000","last_became_peered":"2023-09-30T02:06:18.224464+0000","last_unstale":"2023-09-30T02:06:18.437839+0000","last_undegraded":"2023-09-30T02:06:18.437839+0000","last_fullsized":"2023-09-30T02:06:18.437839+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_clean_scrub_stamp":"2023-09-30T02:06:15.800912+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:13:39.084627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T02:06:18.364779+0000","last_change":"2023-09-30T02:06:17.910244+0000","last_active":"2023-09-30T02:06:18.364779+0000","last_peered":"2023-09-30T02:06:18.364779+0000","last_clean":"2023-09-30T02:06:18.364779+0000","last_became_active":"2023-09-30T02:06:17.855689+0000","last_became_peered":"2023-09-30T02:06:17.855689+0000","last_unstale":"2023-09-30T02:06:18.364779+0000","last_undegraded":"2023-09-30T02:06:18.364779+0000","last_fullsized":"2023-09-30T02:06:18.364779+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_clean_scrub_stamp":"2023-09-30T02:06:15.800912+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:18:08.185701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T02:06:18.359472+0000","last_change":"2023-09-30T02:06:17.806354+0000","last_active":"2023-09-30T02:06:18.359472+0000","last_peered":"2023-09-30T02:06:18.359472+0000","last_clean":"2023-09-30T02:06:18.359472+0000","last_became_active":"2023-09-30T02:06:17.706615+0000","last_became_peered":"2023-09-30T02:06:17.706615+0000","last_unstale":"2023-09-30T02:06:18.359472+0000","last_undegraded":"2023-09-30T02:06:18.359472+0000","last_fullsized":"2023-09-30T02:06:18.359472+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_clean_scrub_stamp":"2023-09-30T02:06:15.800912+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:08:20.786129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"25'76","reported_seq":117,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-09-30T02:06:21.647351+0000","last_change":"2023-09-30T02:06:11.645599+0000","last_active":"2023-09-30T02:06:21.647351+0000","last_peered":"2023-09-30T02:06:21.647351+0000","last_clean":"2023-09-30T02:06:21.647351+0000","last_became_active":"2023-09-30T02:06:11.592631+0000","last_became_peered":"2023-09-30T02:06:11.592631+0000","last_unstale":"2023-09-30T02:06:21.647351+0000","last_undegraded":"2023-09-30T02:06:21.647351+0000","last_fullsized":"2023-09-30T02:06:21.647351+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:06:06.666529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:06:06.666529+0000","last_clean_scrub_stamp":"2023-09-30T02:06:06.666529+0000","objects_scrubbed":0,"log_size":76,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:16:35.859435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T02:06:18.343393+0000","last_change":"2023-09-30T02:06:18.102959+0000","last_active":"2023-09-30T02:06:18.343393+0000","last_peered":"2023-09-30T02:06:18.343393+0000","last_clean":"2023-09-30T02:06:18.343393+0000","last_became_active":"2023-09-30T02:06:18.091407+0000","last_became_peered":"2023-09-30T02:06:18.091407+0000","last_unstale":"2023-09-30T02:06:18.343393+0000","last_undegraded":"2023-09-30T02:06:18.343393+0000","last_fullsized":"2023-09-30T02:06:18.343393+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_clean_scrub_stamp":"2023-09-30T02:06:15.800912+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:34:28.390487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T02:06:18.250861+0000","last_change":"2023-09-30T02:06:17.465018+0000","last_active":"2023-09-30T02:06:18.250861+0000","last_peered":"2023-09-30T02:06:18.250861+0000","last_clean":"2023-09-30T02:06:18.250861+0000","last_became_active":"2023-09-30T02:06:17.420574+0000","last_became_peered":"2023-09-30T02:06:17.420574+0000","last_unstale":"2023-09-30T02:06:18.250861+0000","last_undegraded":"2023-09-30T02:06:18.250861+0000","last_fullsized":"2023-09-30T02:06:18.250861+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_clean_scrub_stamp":"2023-09-30T02:06:15.800912+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:46:28.414555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T02:06:18.354965+0000","last_change":"2023-09-30T02:06:17.896913+0000","last_active":"2023-09-30T02:06:18.354965+0000","last_peered":"2023-09-30T02:06:18.354965+0000","last_clean":"2023-09-30T02:06:18.354965+0000","last_became_active":"2023-09-30T02:06:17.890282+0000","last_became_peered":"2023-09-30T02:06:17.890282+0000","last_unstale":"2023-09-30T02:06:18.354965+0000","last_undegraded":"2023-09-30T02:06:18.354965+0000","last_fullsized":"2023-09-30T02:06:18.354965+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_clean_scrub_stamp":"2023-09-30T02:06:15.800912+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:35:55.671531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-09-30T02:06:17.900193+0000","last_change":"2023-09-30T02:06:17.877329+0000","last_active":"2023-09-30T02:06:17.900193+0000","last_peered":"2023-09-30T02:06:17.900193+0000","last_clean":"2023-09-30T02:06:17.900193+0000","last_became_active":"2023-09-30T02:06:17.836124+0000","last_became_peered":"2023-09-30T02:06:17.836124+0000","last_unstale":"2023-09-30T02:06:17.900193+0000","last_undegraded":"2023-09-30T02:06:17.900193+0000","last_fullsized":"2023-09-30T02:06:17.900193+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_clean_scrub_stamp":"2023-09-30T02:06:15.800912+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T14:04:06.678156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":19,"seq":81604378631,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5208,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366632,"statfs":{"total":96636764160,"available":96631431168,"internally_reserved":0,"allocated":212992,"data_stored":75040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.199999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.113}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.157}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.887}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.225}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.835000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.414999999999999}]}]},{"osd":1,"up_from":18,"seq":77309411334,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6044,"kb_used_data":600,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":94365796,"statfs":{"total":96636764160,"available":96630575104,"internally_reserved":0,"allocated":614400,"data_stored":472880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.931000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":27.437999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":82.474999999999994}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":21.207999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.454000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":30.315000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":31.234000000000002}]}]},{"osd":7,"up_from":14,"seq":60129542152,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5272,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":94366568,"statfs":{"total":96636764160,"available":96631365632,"internally_reserved":0,"allocated":212992,"data_stored":75040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.6859999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.766}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.614000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.484}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.791}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.928000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":22.553000000000001}]}]},{"osd":0,"up_from":18,"seq":77309411335,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5208,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366632,"statfs":{"total":96636764160,"available":96631431168,"internally_reserved":0,"allocated":212992,"data_stored":75040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.007}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.223000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.823}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.632999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.632999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.202000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":21.349}]}]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5208,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366632,"statfs":{"total":96636764160,"available":96631431168,"internally_reserved":0,"allocated":212992,"data_stored":75040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.972}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.997999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.7479999999999993}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.41}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.875}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.504999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":22.341999999999999}]}]},{"osd":4,"up_from":15,"seq":64424509448,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5208,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366632,"statfs":{"total":96636764160,"available":96631431168,"internally_reserved":0,"allocated":212992,"data_stored":75040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.6760000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.0050000000000008}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.378}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.938000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.898}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.773}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.459}]}]},{"osd":5,"up_from":15,"seq":64424509447,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6108,"kb_used_data":600,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":94365732,"statfs":{"total":96636764160,"available":96630509568,"internally_reserved":0,"allocated":614400,"data_stored":472880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.0980000000000008}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.9079999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.776999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.827999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.786999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.885999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.873999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411334,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5208,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366632,"statfs":{"total":96636764160,"available":96631431168,"internally_reserved":0,"allocated":212992,"data_stored":75040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.484999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.294}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.588999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.709}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":22.763000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":21.433}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":27.452999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T02:06:28.512 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T02:06:28.643 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.642+0000 7fdcde614700 10 monclient: get_monmap_and_config 2023-09-30T02:06:28.643 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.642+0000 7fdcde614700 10 monclient: build_initial_monmap 2023-09-30T02:06:28.643 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.643+0000 7fdcde614700 10 monclient: monmap: 2023-09-30T02:06:28.643 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:28.643 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:28.644 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:06:28.644388+0000 2023-09-30T02:06:28.644 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:06:28.644388+0000 2023-09-30T02:06:28.644 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:28.644 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:28.644 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:06:28.644 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:06:28.644 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:06:28.644 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:06:28.644 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:06:28.644 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:28.646 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.643+0000 7fdcde614700 1 Processor -- start 2023-09-30T02:06:28.646 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.643+0000 7fdcde614700 1 -- start start 2023-09-30T02:06:28.646 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.643+0000 7fdcde614700 10 monclient: init 2023-09-30T02:06:28.646 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.644+0000 7fdcde614700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:28.646 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.644+0000 7fdcde614700 10 monclient: _add_conns ranks=[3,2,0,4,1] 2023-09-30T02:06:28.646 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.644+0000 7fdcde614700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fdcd80a5b70 0x7fdcd80b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:28.646 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.644+0000 7fdcde614700 10 monclient(hunting): picked mon.noname-a con 0x7fdcd80a5b70 addr v2:172.21.15.59:3300/0 2023-09-30T02:06:28.646 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.644+0000 7fdcde614700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdcd80a5f40 0x7fdcd8141b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:28.646 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.644+0000 7fdcde614700 10 monclient(hunting): picked mon.noname-b con 0x7fdcd80a5f40 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:28.647 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.644+0000 7fdcde614700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdcd81420c0 addr v1:172.21.15.59:6789/0 2023-09-30T02:06:28.647 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.644+0000 7fdcde614700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcd80a63b0 con 0x7fdcd81420c0 2023-09-30T02:06:28.647 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.644+0000 7fdcde614700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:28.647 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.644+0000 7fdcde614700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcd80a75a0 con 0x7fdcd80a5b70 2023-09-30T02:06:28.647 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.644+0000 7fdcde614700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:28.647 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.644+0000 7fdcde614700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd80af3f0 con 0x7fdcd80a5f40 2023-09-30T02:06:28.647 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.644+0000 7fdcde614700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:28.647 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.644+0000 7fdcde614700 10 monclient(hunting): authenticate will time out at 1833.947555s 2023-09-30T02:06:28.647 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.645+0000 7fdcd77fe700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdcd80a5f40 0x7fdcd8141b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:28.648 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.645+0000 7fdcd7fff700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fdcd80a5b70 0x7fdcd80b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:28.648 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.645+0000 7fdcdcbb1700 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fdcd81420c0 0x7fdcd813e230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:48128/0 (socket says 172.21.15.59:48128) 2023-09-30T02:06:28.648 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.645+0000 7fdcd77fe700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdcd80a5f40 0x7fdcd8141b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:42110/0 (socket says 172.21.15.59:42110) 2023-09-30T02:06:28.648 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.645+0000 7fdcdcbb1700 1 -- 172.21.15.59:0/2800463978 learned_addr learned my addr 172.21.15.59:0/2800463978 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:06:28.648 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.645+0000 7fdcd77fe700 10 monclient(hunting): get_auth_request con 0x7fdcd80a5f40 auth_method 0 2023-09-30T02:06:28.648 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.645+0000 7fdcd77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:28.648 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.645+0000 7fdcd77fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:28.648 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.645+0000 7fdcd77fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:28.648 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.646+0000 7fdcd7fff700 10 monclient(hunting): get_auth_request con 0x7fdcd80a5b70 auth_method 0 2023-09-30T02:06:28.648 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.646+0000 7fdcd7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:28.648 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.646+0000 7fdcd7fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:28.648 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.646+0000 7fdcd7fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:28.649 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.646+0000 7fdcd77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:28.649 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.646+0000 7fdcd77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:28.650 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.646+0000 7fdcd77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:28.650 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.647+0000 7fdcd7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:28.650 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.647+0000 7fdcd7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:28.650 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.647+0000 7fdcd7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:28.650 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.648+0000 7fdcd77fe700 10 monclient(hunting): handle_auth_done global_id 4349 payload 274 2023-09-30T02:06:28.650 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.648+0000 7fdcd77fe700 1 -- 172.21.15.59:0/2800463978 >> v2:172.21.15.59:3300/0 conn(0x7fdcd80a5b70 msgr2=0x7fdcd80b1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:06:28.650 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.648+0000 7fdcd77fe700 1 --2- 172.21.15.59:0/2800463978 >> v2:172.21.15.59:3300/0 conn(0x7fdcd80a5b70 0x7fdcd80b1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:28.650 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.648+0000 7fdcd77fe700 1 -- 172.21.15.59:0/2800463978 >> v1:172.21.15.59:6789/0 conn(0x7fdcd81420c0 legacy=0x7fdcd813e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:28.650 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.648+0000 7fdcd77fe700 10 monclient: _finish_hunting 0 2023-09-30T02:06:28.650 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.648+0000 7fdcd77fe700 1 monclient: found mon.noname-b 2023-09-30T02:06:28.650 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.648+0000 7fdcd77fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:28.650 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.648+0000 7fdcd77fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.59:3301/0 2023-09-30T02:06:28.651 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.648+0000 7fdcd77fe700 1 -- 172.21.15.59:0/2800463978 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcd8142500 con 0x7fdcd80a5f40 2023-09-30T02:06:28.651 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.650+0000 7fdcd77fe700 1 --2- 172.21.15.59:0/2800463978 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdcd80a5f40 0x7fdcd8141b80 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fdccc0098e0 tx=0x7fdccc0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a6ebb11d5bc5c1ca server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:28.655 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.655+0000 7fdcd6ffd700 1 -- 172.21.15.59:0/2800463978 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdccc00b640 con 0x7fdcd80a5f40 2023-09-30T02:06:28.655 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.655+0000 7fdcd6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:28.655 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.655+0000 7fdcd6ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:06:28.656 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.655+0000 7fdcd6ffd700 10 monclient: dump: 2023-09-30T02:06:28.656 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:28.656 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:28.656 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:28.656 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:28.656 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:28.656 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:28.656 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:28.656 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:28.656 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:28.656 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:28.656 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.655+0000 7fdcd6ffd700 10 monclient: _finish_auth 0 2023-09-30T02:06:28.657 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.655+0000 7fdcd6ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:28.657 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.655+0000 7fdcd6ffd700 1 -- 172.21.15.59:0/2800463978 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdccc00b7a0 con 0x7fdcd80a5f40 2023-09-30T02:06:28.657 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.655+0000 7fdcd6ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:28.657 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.655+0000 7fdcd6ffd700 1 -- 172.21.15.59:0/2800463978 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdccc00bb40 con 0x7fdcd80a5f40 2023-09-30T02:06:28.657 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.655+0000 7fdcd6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:28.657 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.655+0000 7fdcd6ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:06:28.657 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.655+0000 7fdcd6ffd700 10 monclient: dump: 2023-09-30T02:06:28.657 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:28.657 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:28.658 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:28.658 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:28.658 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:28.658 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:28.658 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:28.658 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:28.658 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:28.658 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:28.658 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.655+0000 7fdcde614700 5 monclient: authenticate success, global_id 4349 2023-09-30T02:06:28.658 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.655+0000 7fdcde614700 10 monclient: get_monmap_and_config success 2023-09-30T02:06:28.659 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.655+0000 7fdcde614700 4 set_mon_vals no callback set 2023-09-30T02:06:28.661 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.656+0000 7fdcde614700 10 monclient: shutdown 2023-09-30T02:06:28.662 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.656+0000 7fdcde614700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:28.662 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.656+0000 7fdcde614700 1 -- 172.21.15.59:0/2800463978 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdcd80a5f40 msgr2=0x7fdcd8141b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:28.662 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.656+0000 7fdcde614700 1 --2- 172.21.15.59:0/2800463978 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdcd80a5f40 0x7fdcd8141b80 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fdccc0098e0 tx=0x7fdccc0091a0 comp rx=0 tx=0).stop 2023-09-30T02:06:28.662 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.656+0000 7fdcde614700 1 -- 172.21.15.59:0/2800463978 shutdown_connections 2023-09-30T02:06:28.662 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.656+0000 7fdcde614700 1 --2- 172.21.15.59:0/2800463978 >> v2:172.21.15.59:3300/0 conn(0x7fdcd80a5b70 0x7fdcd80b1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:28.662 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.656+0000 7fdcde614700 1 --2- 172.21.15.59:0/2800463978 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdcd80a5f40 0x7fdcd8141b80 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:28.662 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.656+0000 7fdcde614700 1 -- 172.21.15.59:0/2800463978 >> 172.21.15.59:0/2800463978 conn(0x7fdcd80631f0 msgr2=0x7fdcd80635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:28.662 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.656+0000 7fdcde614700 1 -- 172.21.15.59:0/2800463978 shutdown_connections 2023-09-30T02:06:28.663 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.656+0000 7fdcde614700 1 -- 172.21.15.59:0/2800463978 wait complete. 2023-09-30T02:06:28.663 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.656+0000 7fdcde614700 10 monclient: build_initial_monmap 2023-09-30T02:06:28.663 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.656+0000 7fdcde614700 10 monclient: monmap: 2023-09-30T02:06:28.663 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:28.663 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:06:28.663 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:06:28.663 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:06:28.663 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:28.663 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:06:28.663 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:06:28.663 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:06:28.664 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:06:28.664 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:28.664 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.657+0000 7fdcde614700 1 librados: starting msgr at 2023-09-30T02:06:28.664 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.657+0000 7fdcde614700 1 librados: starting objecter 2023-09-30T02:06:28.664 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.657+0000 7fdcde614700 1 Processor -- start 2023-09-30T02:06:28.664 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.657+0000 7fdcde614700 1 -- start start 2023-09-30T02:06:28.664 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.657+0000 7fdcde614700 1 librados: setting wanted keys 2023-09-30T02:06:28.664 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.657+0000 7fdcde614700 1 librados: calling monclient init 2023-09-30T02:06:28.664 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.657+0000 7fdcde614700 10 monclient: init 2023-09-30T02:06:28.664 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.658+0000 7fdcde614700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:28.664 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.658+0000 7fdcde614700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T02:06:28.664 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.658+0000 7fdcde614700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdcd80a5b70 0x7fdcd813a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:28.665 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.658+0000 7fdcde614700 10 monclient(hunting): picked mon.noname-a con 0x7fdcd80a5b70 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:06:28.665 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.658+0000 7fdcde614700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdcd80a5f40 0x7fdcd8151560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:28.665 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.658+0000 7fdcde614700 10 monclient(hunting): picked mon.noname-c con 0x7fdcd80a5f40 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:28.665 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.658+0000 7fdcde614700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdcd81420c0 0x7fdcd8151aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:28.665 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.658+0000 7fdcde614700 10 monclient(hunting): picked mon.noname-b con 0x7fdcd81420c0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:06:28.665 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.658+0000 7fdcde614700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:28.665 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.658+0000 7fdcde614700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd80aa490 con 0x7fdcd80a5b70 2023-09-30T02:06:28.665 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.658+0000 7fdcde614700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:28.665 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.658+0000 7fdcde614700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd80aa920 con 0x7fdcd81420c0 2023-09-30T02:06:28.665 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.658+0000 7fdcde614700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:28.665 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.658+0000 7fdcde614700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd80aa090 con 0x7fdcd80a5f40 2023-09-30T02:06:28.665 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.658+0000 7fdcde614700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:28.665 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.658+0000 7fdcde614700 10 monclient(hunting): authenticate will time out at 1833.961189s 2023-09-30T02:06:28.666 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.658+0000 7fdcd7fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdcd80a5b70 0x7fdcd813a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:28.666 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.658+0000 7fdcdcbb1700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdcd81420c0 0x7fdcd8151aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:28.666 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.659+0000 7fdcd77fe700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdcd80a5f40 0x7fdcd8151560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:28.666 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.659+0000 7fdcd7fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdcd80a5b70 0x7fdcd813a680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34014/0 (socket says 172.21.15.59:34014) 2023-09-30T02:06:28.666 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.659+0000 7fdcdcbb1700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdcd81420c0 0x7fdcd8151aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:38620/0 (socket says 172.21.15.59:38620) 2023-09-30T02:06:28.666 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.659+0000 7fdcd7fff700 1 -- 172.21.15.59:0/2795987598 learned_addr learned my addr 172.21.15.59:0/2795987598 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:06:28.666 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.659+0000 7fdcd7fff700 10 monclient(hunting): get_auth_request con 0x7fdcd80a5b70 auth_method 0 2023-09-30T02:06:28.666 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.659+0000 7fdcd7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:28.666 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.659+0000 7fdcd7fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:28.666 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.659+0000 7fdcd7fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:28.666 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.659+0000 7fdcdcbb1700 10 monclient(hunting): get_auth_request con 0x7fdcd81420c0 auth_method 0 2023-09-30T02:06:28.666 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.659+0000 7fdcdcbb1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:28.667 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.659+0000 7fdcdcbb1700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:28.667 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.659+0000 7fdcdcbb1700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:28.667 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.659+0000 7fdcd77fe700 10 monclient(hunting): get_auth_request con 0x7fdcd80a5f40 auth_method 0 2023-09-30T02:06:28.667 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.659+0000 7fdcd77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:28.667 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.659+0000 7fdcd77fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:28.667 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.659+0000 7fdcd77fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:28.667 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.659+0000 7fdcdcbb1700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:28.667 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.659+0000 7fdcdcbb1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:28.667 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.659+0000 7fdcdcbb1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:28.667 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.659+0000 7fdcd7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:28.667 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.659+0000 7fdcd7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:28.667 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.659+0000 7fdcd7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:28.668 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.659+0000 7fdcd77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:28.668 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.659+0000 7fdcd77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:28.668 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.659+0000 7fdcd77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:28.668 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.663+0000 7fdcdcbb1700 10 monclient(hunting): handle_auth_done global_id 4402 payload 824 2023-09-30T02:06:28.669 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.663+0000 7fdcdcbb1700 1 -- 172.21.15.59:0/2795987598 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdcd80a5f40 msgr2=0x7fdcd8151560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:28.669 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.663+0000 7fdcdcbb1700 1 --2- 172.21.15.59:0/2795987598 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdcd80a5f40 0x7fdcd8151560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:28.669 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.663+0000 7fdcdcbb1700 1 -- 172.21.15.59:0/2795987598 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdcd80a5b70 msgr2=0x7fdcd813a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:28.669 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.663+0000 7fdcdcbb1700 1 --2- 172.21.15.59:0/2795987598 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdcd80a5b70 0x7fdcd813a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:28.669 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.663+0000 7fdcdcbb1700 10 monclient: _finish_hunting 0 2023-09-30T02:06:28.669 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.663+0000 7fdcdcbb1700 1 monclient: found mon.noname-b 2023-09-30T02:06:28.669 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.663+0000 7fdcdcbb1700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:28.669 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.663+0000 7fdcdcbb1700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:06:28.669 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.663+0000 7fdcdcbb1700 1 -- 172.21.15.59:0/2795987598 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdccc009420 con 0x7fdcd81420c0 2023-09-30T02:06:28.669 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.666+0000 7fdcdcbb1700 1 --2- 172.21.15.59:0/2795987598 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdcd81420c0 0x7fdcd8151aa0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fdcd000cb40 tx=0x7fdcd0001900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:28.669 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.666+0000 7fdcbffff700 10 client.?.objecter ms_handle_connect 0x7fdcd81420c0 2023-09-30T02:06:28.669 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.666+0000 7fdcbffff700 10 client.?.objecter resend_mon_ops 2023-09-30T02:06:28.670 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.670+0000 7fdcbffff700 1 -- 172.21.15.59:0/2795987598 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcd000a7e0 con 0x7fdcd81420c0 2023-09-30T02:06:28.671 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.670+0000 7fdcbffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:28.671 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.670+0000 7fdcbffff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:06:28.671 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.670+0000 7fdcbffff700 10 monclient: dump: 2023-09-30T02:06:28.671 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:28.671 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:28.671 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:28.671 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:28.671 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:28.671 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:28.671 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:28.672 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:28.672 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:28.672 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:28.672 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.670+0000 7fdcbffff700 10 monclient: _finish_auth 0 2023-09-30T02:06:28.672 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.670+0000 7fdcbffff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:28.672 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.670+0000 7fdcbffff700 1 -- 172.21.15.59:0/2795987598 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcd000a940 con 0x7fdcd81420c0 2023-09-30T02:06:28.672 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.670+0000 7fdcbffff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:28.672 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.670+0000 7fdcbffff700 1 -- 172.21.15.59:0/2795987598 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcd000ace0 con 0x7fdcd81420c0 2023-09-30T02:06:28.672 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.670+0000 7fdcbffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:28.672 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.670+0000 7fdcbffff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:06:28.672 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.670+0000 7fdcbffff700 10 monclient: dump: 2023-09-30T02:06:28.672 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:28.673 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:28.673 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:28.673 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:28.673 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:28.673 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:28.673 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:28.673 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:28.673 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:28.673 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:28.673 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.670+0000 7fdcde614700 5 monclient: authenticate success, global_id 4402 2023-09-30T02:06:28.673 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.670+0000 7fdcde614700 10 monclient: _renew_subs 2023-09-30T02:06:28.673 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.670+0000 7fdcde614700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:28.674 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.670+0000 7fdcde614700 1 -- 172.21.15.59:0/2795987598 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcd80a8990 con 0x7fdcd81420c0 2023-09-30T02:06:28.674 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.670+0000 7fdcd67fc700 4 set_mon_vals no callback set 2023-09-30T02:06:28.674 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.670+0000 7fdcde614700 10 client.4402.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:06:28.674 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.670+0000 7fdcde614700 10 monclient: _renew_subs 2023-09-30T02:06:28.674 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.670+0000 7fdcde614700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:28.674 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.670+0000 7fdcde614700 1 -- 172.21.15.59:0/2795987598 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcd80a9790 con 0x7fdcd81420c0 2023-09-30T02:06:28.674 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.670+0000 7fdcde614700 1 librados: init done 2023-09-30T02:06:28.674 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.671+0000 7fdcde614700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:28.675 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.671+0000 7fdcde614700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:28.675 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.671+0000 7fdcde614700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:28.675 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.671+0000 7fdcde614700 1 -- 172.21.15.59:0/2795987598 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcd80abd80 con 0x7fdcd81420c0 2023-09-30T02:06:28.698 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.698+0000 7fdcbffff700 1 -- 172.21.15.59:0/2795987598 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fdcd0007300 con 0x7fdcd81420c0 2023-09-30T02:06:28.700 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.699+0000 7fdcbffff700 1 --2- 172.21.15.59:0/2795987598 >> v2:172.21.15.110:6800/115870 conn(0x7fdcc4068910 0x7fdcc406adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:28.700 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.699+0000 7fdcbffff700 1 -- 172.21.15.59:0/2795987598 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fdcd0084c00 con 0x7fdcd81420c0 2023-09-30T02:06:28.700 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.699+0000 7fdcbffff700 10 client.4402.objecter ms_dispatch 0x7fdcd8001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:06:28.700 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.699+0000 7fdcbffff700 3 client.4402.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:06:28.700 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.699+0000 7fdcbffff700 3 client.4402.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:06:28.700 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.699+0000 7fdcbffff700 20 client.4402.objecter dump_active .. 0 homeless 2023-09-30T02:06:28.700 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.699+0000 7fdcd7fff700 1 --2- 172.21.15.59:0/2795987598 >> v2:172.21.15.110:6800/115870 conn(0x7fdcc4068910 0x7fdcc406adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:28.700 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.699+0000 7fdcd7fff700 10 monclient: get_auth_request con 0x7fdcc4068910 auth_method 0 2023-09-30T02:06:28.700 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.699+0000 7fdcbffff700 10 client.4402.objecter ms_handle_connect 0x7fdcc4068910 2023-09-30T02:06:28.701 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.699+0000 7fdcd7fff700 1 --2- 172.21.15.59:0/2795987598 >> v2:172.21.15.110:6800/115870 conn(0x7fdcc4068910 0x7fdcc406adc0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fdcc8009e70 tx=0x7fdcc80091a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:28.930 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.929+0000 7fdcbffff700 1 -- 172.21.15.59:0/2795987598 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fdcd0052490 con 0x7fdcd81420c0 2023-09-30T02:06:28.930 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.929+0000 7fdcbffff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:28.930 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:28.929+0000 7fdcbffff700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:06:29.133 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.132+0000 7fdcde614700 1 -- 172.21.15.59:0/2795987598 --> v2:172.21.15.110:6800/115870 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fdcd80b2880 con 0x7fdcc4068910 2023-09-30T02:06:29.135 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.135+0000 7fdcbffff700 1 -- 172.21.15.59:0/2795987598 <== mgr.4108 v2:172.21.15.110:6800/115870 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+47789 (secure 0 0 0) 0x7fdcd80b2880 con 0x7fdcc4068910 2023-09-30T02:06:29.136 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:06:29.138 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.137+0000 7fdcde614700 10 librados: watch_flush enter 2023-09-30T02:06:29.139 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.137+0000 7fdcde614700 10 librados: watch_flush exit 2023-09-30T02:06:29.139 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.137+0000 7fdcde614700 20 client.4402.objecter shutdown clearing up homeless session... 2023-09-30T02:06:29.139 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.137+0000 7fdcde614700 10 client.4402.objecter successfully canceled tick 2023-09-30T02:06:29.139 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.137+0000 7fdcde614700 1 -- 172.21.15.59:0/2795987598 >> v2:172.21.15.110:6800/115870 conn(0x7fdcc4068910 msgr2=0x7fdcc406adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:29.139 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.137+0000 7fdcde614700 1 --2- 172.21.15.59:0/2795987598 >> v2:172.21.15.110:6800/115870 conn(0x7fdcc4068910 0x7fdcc406adc0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fdcc8009e70 tx=0x7fdcc80091a0 comp rx=0 tx=0).stop 2023-09-30T02:06:29.139 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.137+0000 7fdcde614700 10 monclient: shutdown 2023-09-30T02:06:29.139 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.137+0000 7fdcde614700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:29.139 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.137+0000 7fdcde614700 1 -- 172.21.15.59:0/2795987598 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdcd81420c0 msgr2=0x7fdcd8151aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:29.139 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.137+0000 7fdcde614700 1 --2- 172.21.15.59:0/2795987598 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdcd81420c0 0x7fdcd8151aa0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fdcd000cb40 tx=0x7fdcd0001900 comp rx=0 tx=0).stop 2023-09-30T02:06:29.139 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.137+0000 7fdcde614700 1 -- 172.21.15.59:0/2795987598 shutdown_connections 2023-09-30T02:06:29.139 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.137+0000 7fdcde614700 1 --2- 172.21.15.59:0/2795987598 >> v2:172.21.15.110:6800/115870 conn(0x7fdcc4068910 0x7fdcc406adc0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:29.139 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.137+0000 7fdcde614700 1 --2- 172.21.15.59:0/2795987598 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdcd81420c0 0x7fdcd8151aa0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:29.140 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.137+0000 7fdcde614700 1 --2- 172.21.15.59:0/2795987598 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdcd80a5b70 0x7fdcd813a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:29.140 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.137+0000 7fdcde614700 1 --2- 172.21.15.59:0/2795987598 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdcd80a5f40 0x7fdcd8151560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:29.140 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.137+0000 7fdcde614700 1 -- 172.21.15.59:0/2795987598 >> 172.21.15.59:0/2795987598 conn(0x7fdcd80631f0 msgr2=0x7fdcd8137a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:29.140 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.138+0000 7fdcde614700 1 -- 172.21.15.59:0/2795987598 shutdown_connections 2023-09-30T02:06:29.140 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.138+0000 7fdcde614700 1 -- 172.21.15.59:0/2795987598 wait complete. 2023-09-30T02:06:29.140 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.138+0000 7fdcde614700 1 librados: shutdown 2023-09-30T02:06:29.140 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2023-09-30T02:06:29.151 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":42,"stamp":"2023-09-30T02:06:27.585062+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":43464,"kb_used_data":2448,"kb_used_omap":0,"kb_used_meta":40576,"kb_avail":754931256,"statfs":{"total":773094113280,"available":773049606144,"internally_reserved":0,"allocated":2506752,"data_stored":1396000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":41549824},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":9,"num_write":14,"num_write_kb":262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.355335"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T02:06:18.350818+0000","last_change":"2023-09-30T02:06:18.210390+0000","last_active":"2023-09-30T02:06:18.350818+0000","last_peered":"2023-09-30T02:06:18.350818+0000","last_clean":"2023-09-30T02:06:18.350818+0000","last_became_active":"2023-09-30T02:06:18.198153+0000","last_became_peered":"2023-09-30T02:06:18.198153+0000","last_unstale":"2023-09-30T02:06:18.350818+0000","last_undegraded":"2023-09-30T02:06:18.350818+0000","last_fullsized":"2023-09-30T02:06:18.350818+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_clean_scrub_stamp":"2023-09-30T02:06:15.800912+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:42:32.739379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T02:06:18.437839+0000","last_change":"2023-09-30T02:06:18.262708+0000","last_active":"2023-09-30T02:06:18.437839+0000","last_peered":"2023-09-30T02:06:18.437839+0000","last_clean":"2023-09-30T02:06:18.437839+0000","last_became_active":"2023-09-30T02:06:18.224464+0000","last_became_peered":"2023-09-30T02:06:18.224464+0000","last_unstale":"2023-09-30T02:06:18.437839+0000","last_undegraded":"2023-09-30T02:06:18.437839+0000","last_fullsized":"2023-09-30T02:06:18.437839+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_clean_scrub_stamp":"2023-09-30T02:06:15.800912+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:13:39.084627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T02:06:18.364779+0000","last_change":"2023-09-30T02:06:17.910244+0000","last_active":"2023-09-30T02:06:18.364779+0000","last_peered":"2023-09-30T02:06:18.364779+0000","last_clean":"2023-09-30T02:06:18.364779+0000","last_became_active":"2023-09-30T02:06:17.855689+0000","last_became_peered":"2023-09-30T02:06:17.855689+0000","last_unstale":"2023-09-30T02:06:18.364779+0000","last_undegraded":"2023-09-30T02:06:18.364779+0000","last_fullsized":"2023-09-30T02:06:18.364779+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_clean_scrub_stamp":"2023-09-30T02:06:15.800912+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:18:08.185701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T02:06:18.359472+0000","last_change":"2023-09-30T02:06:17.806354+0000","last_active":"2023-09-30T02:06:18.359472+0000","last_peered":"2023-09-30T02:06:18.359472+0000","last_clean":"2023-09-30T02:06:18.359472+0000","last_became_active":"2023-09-30T02:06:17.706615+0000","last_became_peered":"2023-09-30T02:06:17.706615+0000","last_unstale":"2023-09-30T02:06:18.359472+0000","last_undegraded":"2023-09-30T02:06:18.359472+0000","last_fullsized":"2023-09-30T02:06:18.359472+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_clean_scrub_stamp":"2023-09-30T02:06:15.800912+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:08:20.786129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"25'76","reported_seq":117,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-09-30T02:06:21.647351+0000","last_change":"2023-09-30T02:06:11.645599+0000","last_active":"2023-09-30T02:06:21.647351+0000","last_peered":"2023-09-30T02:06:21.647351+0000","last_clean":"2023-09-30T02:06:21.647351+0000","last_became_active":"2023-09-30T02:06:11.592631+0000","last_became_peered":"2023-09-30T02:06:11.592631+0000","last_unstale":"2023-09-30T02:06:21.647351+0000","last_undegraded":"2023-09-30T02:06:21.647351+0000","last_fullsized":"2023-09-30T02:06:21.647351+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:06:06.666529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:06:06.666529+0000","last_clean_scrub_stamp":"2023-09-30T02:06:06.666529+0000","objects_scrubbed":0,"log_size":76,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:16:35.859435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T02:06:18.343393+0000","last_change":"2023-09-30T02:06:18.102959+0000","last_active":"2023-09-30T02:06:18.343393+0000","last_peered":"2023-09-30T02:06:18.343393+0000","last_clean":"2023-09-30T02:06:18.343393+0000","last_became_active":"2023-09-30T02:06:18.091407+0000","last_became_peered":"2023-09-30T02:06:18.091407+0000","last_unstale":"2023-09-30T02:06:18.343393+0000","last_undegraded":"2023-09-30T02:06:18.343393+0000","last_fullsized":"2023-09-30T02:06:18.343393+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_clean_scrub_stamp":"2023-09-30T02:06:15.800912+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:34:28.390487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T02:06:18.250861+0000","last_change":"2023-09-30T02:06:17.465018+0000","last_active":"2023-09-30T02:06:18.250861+0000","last_peered":"2023-09-30T02:06:18.250861+0000","last_clean":"2023-09-30T02:06:18.250861+0000","last_became_active":"2023-09-30T02:06:17.420574+0000","last_became_peered":"2023-09-30T02:06:17.420574+0000","last_unstale":"2023-09-30T02:06:18.250861+0000","last_undegraded":"2023-09-30T02:06:18.250861+0000","last_fullsized":"2023-09-30T02:06:18.250861+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_clean_scrub_stamp":"2023-09-30T02:06:15.800912+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:46:28.414555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T02:06:18.354965+0000","last_change":"2023-09-30T02:06:17.896913+0000","last_active":"2023-09-30T02:06:18.354965+0000","last_peered":"2023-09-30T02:06:18.354965+0000","last_clean":"2023-09-30T02:06:18.354965+0000","last_became_active":"2023-09-30T02:06:17.890282+0000","last_became_peered":"2023-09-30T02:06:17.890282+0000","last_unstale":"2023-09-30T02:06:18.354965+0000","last_undegraded":"2023-09-30T02:06:18.354965+0000","last_fullsized":"2023-09-30T02:06:18.354965+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_clean_scrub_stamp":"2023-09-30T02:06:15.800912+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:35:55.671531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-09-30T02:06:17.900193+0000","last_change":"2023-09-30T02:06:17.877329+0000","last_active":"2023-09-30T02:06:17.900193+0000","last_peered":"2023-09-30T02:06:17.900193+0000","last_clean":"2023-09-30T02:06:17.900193+0000","last_became_active":"2023-09-30T02:06:17.836124+0000","last_became_peered":"2023-09-30T02:06:17.836124+0000","last_unstale":"2023-09-30T02:06:17.900193+0000","last_undegraded":"2023-09-30T02:06:17.900193+0000","last_fullsized":"2023-09-30T02:06:17.900193+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_clean_scrub_stamp":"2023-09-30T02:06:15.800912+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T14:04:06.678156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":19,"seq":81604378631,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5208,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366632,"statfs":{"total":96636764160,"available":96631431168,"internally_reserved":0,"allocated":212992,"data_stored":75040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.199999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.113}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.157}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.887}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.225}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.835000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.414999999999999}]}]},{"osd":1,"up_from":18,"seq":77309411334,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6044,"kb_used_data":600,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":94365796,"statfs":{"total":96636764160,"available":96630575104,"internally_reserved":0,"allocated":614400,"data_stored":472880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.931000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":27.437999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":82.474999999999994}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":21.207999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.454000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":30.315000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":31.234000000000002}]}]},{"osd":7,"up_from":14,"seq":60129542152,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5272,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":94366568,"statfs":{"total":96636764160,"available":96631365632,"internally_reserved":0,"allocated":212992,"data_stored":75040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.6859999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.766}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.614000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.484}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.791}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.928000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":22.553000000000001}]}]},{"osd":0,"up_from":18,"seq":77309411335,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5208,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366632,"statfs":{"total":96636764160,"available":96631431168,"internally_reserved":0,"allocated":212992,"data_stored":75040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.007}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.223000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.823}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.632999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.632999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.202000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":21.349}]}]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5208,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366632,"statfs":{"total":96636764160,"available":96631431168,"internally_reserved":0,"allocated":212992,"data_stored":75040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.972}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.997999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.7479999999999993}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.41}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.875}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.504999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":22.341999999999999}]}]},{"osd":4,"up_from":15,"seq":64424509448,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5208,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366632,"statfs":{"total":96636764160,"available":96631431168,"internally_reserved":0,"allocated":212992,"data_stored":75040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":5.6760000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.0050000000000008}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.378}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.938000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.898}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.773}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":20.459}]}]},{"osd":5,"up_from":15,"seq":64424509447,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6108,"kb_used_data":600,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":94365732,"statfs":{"total":96636764160,"available":96630509568,"internally_reserved":0,"allocated":614400,"data_stored":472880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.0980000000000008}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.9079999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.776999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.827999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.786999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.885999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.873999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411334,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5208,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366632,"statfs":{"total":96636764160,"available":96631431168,"internally_reserved":0,"allocated":212992,"data_stored":75040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.484999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.294}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":19.588999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.709}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":22.763000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":21.433}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":27.452999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T02:06:29.152 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-09-30T02:06:29.152 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-09-30T02:06:29.153 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-09-30T02:06:29.153 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:06:29.284 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.284+0000 7f0186c56700 10 monclient: get_monmap_and_config 2023-09-30T02:06:29.284 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.284+0000 7f0186c56700 10 monclient: build_initial_monmap 2023-09-30T02:06:29.284 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.284+0000 7f0186c56700 10 monclient: monmap: 2023-09-30T02:06:29.285 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:29.285 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:29.285 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:06:29.285451+0000 2023-09-30T02:06:29.285 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:06:29.285451+0000 2023-09-30T02:06:29.285 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:29.285 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:29.285 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:06:29.285 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:06:29.285 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:06:29.285 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:06:29.285 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:06:29.285 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:29.287 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.284+0000 7f0186c56700 1 Processor -- start 2023-09-30T02:06:29.288 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.285+0000 7f0186c56700 1 -- start start 2023-09-30T02:06:29.288 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.285+0000 7f0186c56700 10 monclient: init 2023-09-30T02:06:29.288 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.286+0000 7f0186c56700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:29.288 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.286+0000 7f0186c56700 10 monclient: _add_conns ranks=[4,0,2,1,3] 2023-09-30T02:06:29.288 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.286+0000 7f0186c56700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f01800a5b70 0x7f01800b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:29.288 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.286+0000 7f0186c56700 10 monclient(hunting): picked mon.noname-c con 0x7f01800a5b70 addr v2:172.21.15.110:3300/0 2023-09-30T02:06:29.288 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.286+0000 7f0186c56700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f01800a5f40 addr v1:172.21.15.59:6789/0 2023-09-30T02:06:29.288 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.286+0000 7f0186c56700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0180142290 0x7f018013af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:29.289 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.286+0000 7f0186c56700 10 monclient(hunting): picked mon.noname-b con 0x7f0180142290 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:29.289 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.286+0000 7f0186c56700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01800a75a0 con 0x7f01800a5f40 2023-09-30T02:06:29.289 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.286+0000 7f0186c56700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:29.289 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.286+0000 7f0186c56700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01801426d0 con 0x7f01800a5b70 2023-09-30T02:06:29.290 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.286+0000 7f0186c56700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:29.290 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.286+0000 7f0186c56700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01800a63e0 con 0x7f0180142290 2023-09-30T02:06:29.290 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.286+0000 7f0186c56700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:29.290 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.286+0000 7f0186c56700 10 monclient(hunting): authenticate will time out at 1834.589166s 2023-09-30T02:06:29.290 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.286+0000 7f01851f3700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0180142290 0x7f018013af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:29.290 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.286+0000 7f01849f2700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f01800a5b70 0x7f01800b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:29.290 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.286+0000 7f017ffff700 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f01800a5f40 0x7f0180141b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:48142/0 (socket says 172.21.15.59:48142) 2023-09-30T02:06:29.290 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.286+0000 7f017ffff700 1 -- 172.21.15.59:0/858753969 learned_addr learned my addr 172.21.15.59:0/858753969 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:06:29.290 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.287+0000 7f01849f2700 10 monclient(hunting): get_auth_request con 0x7f01800a5b70 auth_method 0 2023-09-30T02:06:29.290 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.287+0000 7f01849f2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:29.290 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.287+0000 7f01849f2700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:29.290 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.287+0000 7f01849f2700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:29.291 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.287+0000 7f01851f3700 10 monclient(hunting): get_auth_request con 0x7f0180142290 auth_method 0 2023-09-30T02:06:29.291 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.287+0000 7f01851f3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:29.291 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.287+0000 7f01851f3700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:29.291 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.287+0000 7f01851f3700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:29.291 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.287+0000 7f01849f2700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:29.291 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.287+0000 7f01849f2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:29.291 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.287+0000 7f01849f2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:29.291 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.287+0000 7f01851f3700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:29.291 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.287+0000 7f01851f3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:29.291 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.288+0000 7f01851f3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:29.292 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.289+0000 7f01849f2700 10 monclient(hunting): handle_auth_done global_id 4405 payload 274 2023-09-30T02:06:29.292 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.289+0000 7f01849f2700 1 -- 172.21.15.59:0/858753969 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0180142290 msgr2=0x7f018013af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:06:29.292 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.289+0000 7f01849f2700 1 --2- 172.21.15.59:0/858753969 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0180142290 0x7f018013af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:29.292 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.289+0000 7f01849f2700 1 -- 172.21.15.59:0/858753969 >> v1:172.21.15.59:6789/0 conn(0x7f01800a5f40 legacy=0x7f0180141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:29.292 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.289+0000 7f01849f2700 10 monclient: _finish_hunting 0 2023-09-30T02:06:29.293 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.289+0000 7f01849f2700 1 monclient: found mon.noname-c 2023-09-30T02:06:29.293 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.289+0000 7f01849f2700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:29.293 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.289+0000 7f01849f2700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3300/0 2023-09-30T02:06:29.293 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.289+0000 7f01849f2700 1 -- 172.21.15.59:0/858753969 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01800af3f0 con 0x7f01800a5b70 2023-09-30T02:06:29.293 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.291+0000 7f01849f2700 1 --2- 172.21.15.59:0/858753969 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01800a5b70 0x7f01800b1500 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f0170009910 tx=0x7f01700091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=53f2b96af90cb6a3 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:29.296 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.295+0000 7f017f7fe700 1 -- 172.21.15.59:0/858753969 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f017000b640 con 0x7f01800a5b70 2023-09-30T02:06:29.296 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.295+0000 7f017f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:29.296 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.295+0000 7f017f7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:06:29.296 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.295+0000 7f017f7fe700 10 monclient: dump: 2023-09-30T02:06:29.296 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:29.296 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:29.296 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:29.296 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:29.297 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:29.297 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:29.297 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:29.297 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:29.297 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:29.298 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:29.298 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.295+0000 7f017f7fe700 10 monclient: _finish_auth 0 2023-09-30T02:06:29.298 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.295+0000 7f017f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:29.298 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.295+0000 7f017f7fe700 1 -- 172.21.15.59:0/858753969 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f017000b7a0 con 0x7f01800a5b70 2023-09-30T02:06:29.298 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.295+0000 7f017f7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:29.298 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.295+0000 7f017f7fe700 1 -- 172.21.15.59:0/858753969 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f017000bb60 con 0x7f01800a5b70 2023-09-30T02:06:29.299 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.295+0000 7f0186c56700 5 monclient: authenticate success, global_id 4405 2023-09-30T02:06:29.299 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.295+0000 7f0186c56700 10 monclient: get_monmap_and_config success 2023-09-30T02:06:29.299 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.296+0000 7f0186c56700 4 set_mon_vals no callback set 2023-09-30T02:06:29.299 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.296+0000 7f017f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:29.299 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.296+0000 7f017f7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:06:29.299 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.296+0000 7f017f7fe700 10 monclient: dump: 2023-09-30T02:06:29.300 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:29.300 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:29.300 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:29.300 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:29.300 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:29.300 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:29.300 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:29.301 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:29.301 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:29.301 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:29.304 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.296+0000 7f0186c56700 10 monclient: shutdown 2023-09-30T02:06:29.304 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.296+0000 7f0186c56700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:29.305 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.296+0000 7f0186c56700 1 -- 172.21.15.59:0/858753969 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01800a5b70 msgr2=0x7f01800b1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:29.305 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.296+0000 7f0186c56700 1 --2- 172.21.15.59:0/858753969 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01800a5b70 0x7f01800b1500 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f0170009910 tx=0x7f01700091a0 comp rx=0 tx=0).stop 2023-09-30T02:06:29.305 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.296+0000 7f0186c56700 1 -- 172.21.15.59:0/858753969 shutdown_connections 2023-09-30T02:06:29.305 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.296+0000 7f0186c56700 1 --2- 172.21.15.59:0/858753969 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01800a5b70 0x7f01800b1500 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:29.305 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.296+0000 7f0186c56700 1 --2- 172.21.15.59:0/858753969 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0180142290 0x7f018013af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:29.305 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.296+0000 7f0186c56700 1 -- 172.21.15.59:0/858753969 >> 172.21.15.59:0/858753969 conn(0x7f01800631f0 msgr2=0x7f01800635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:29.306 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.296+0000 7f0186c56700 1 -- 172.21.15.59:0/858753969 shutdown_connections 2023-09-30T02:06:29.306 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.296+0000 7f0186c56700 1 -- 172.21.15.59:0/858753969 wait complete. 2023-09-30T02:06:29.306 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.297+0000 7f0186c56700 10 monclient: build_initial_monmap 2023-09-30T02:06:29.306 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.297+0000 7f0186c56700 10 monclient: monmap: 2023-09-30T02:06:29.306 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:29.306 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:06:29.306 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:06:29.306 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:06:29.307 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:29.307 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:06:29.307 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:06:29.307 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:06:29.310 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:06:29.310 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:29.310 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.298+0000 7f0186c56700 1 librados: starting msgr at 2023-09-30T02:06:29.310 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.298+0000 7f0186c56700 1 librados: starting objecter 2023-09-30T02:06:29.310 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.298+0000 7f0186c56700 1 Processor -- start 2023-09-30T02:06:29.311 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.298+0000 7f0186c56700 1 -- start start 2023-09-30T02:06:29.311 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.298+0000 7f0186c56700 1 librados: setting wanted keys 2023-09-30T02:06:29.311 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.298+0000 7f0186c56700 1 librados: calling monclient init 2023-09-30T02:06:29.311 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.298+0000 7f0186c56700 10 monclient: init 2023-09-30T02:06:29.311 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.299+0000 7f0186c56700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:29.311 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.299+0000 7f0186c56700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T02:06:29.311 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.299+0000 7f0186c56700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01800a5b70 0x7f01801513d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:29.311 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.299+0000 7f0186c56700 10 monclient(hunting): picked mon.noname-b con 0x7f01800a5b70 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:06:29.312 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.299+0000 7f0186c56700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01800a5f40 0x7f0180151910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:29.312 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.299+0000 7f0186c56700 10 monclient(hunting): picked mon.noname-a con 0x7f01800a5f40 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:06:29.312 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.299+0000 7f0186c56700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0180142290 0x7f01800b3b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:29.312 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.299+0000 7f0186c56700 10 monclient(hunting): picked mon.noname-c con 0x7f0180142290 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:29.312 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.299+0000 7f0186c56700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:29.312 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.299+0000 7f0186c56700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0180151ff0 con 0x7f01800a5f40 2023-09-30T02:06:29.312 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.299+0000 7f0186c56700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:29.312 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.299+0000 7f0186c56700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f018013f000 con 0x7f01800a5b70 2023-09-30T02:06:29.313 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.299+0000 7f0186c56700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:29.313 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.299+0000 7f0186c56700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f018014b720 con 0x7f0180142290 2023-09-30T02:06:29.313 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.299+0000 7f0186c56700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:29.313 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.299+0000 7f0186c56700 10 monclient(hunting): authenticate will time out at 1834.602673s 2023-09-30T02:06:29.313 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.300+0000 7f01849f2700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01800a5b70 0x7f01801513d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:29.313 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.300+0000 7f01851f3700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0180142290 0x7f01800b3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:29.313 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.300+0000 7f017ffff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01800a5f40 0x7f0180151910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:29.313 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.300+0000 7f01849f2700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01800a5b70 0x7f01801513d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:38634/0 (socket says 172.21.15.59:38634) 2023-09-30T02:06:29.314 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.300+0000 7f01849f2700 1 -- 172.21.15.59:0/1309190621 learned_addr learned my addr 172.21.15.59:0/1309190621 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:06:29.314 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.300+0000 7f01849f2700 10 monclient(hunting): get_auth_request con 0x7f01800a5b70 auth_method 0 2023-09-30T02:06:29.314 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.300+0000 7f01849f2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:29.314 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.300+0000 7f01849f2700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:29.314 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.300+0000 7f01849f2700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:29.314 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.300+0000 7f01851f3700 10 monclient(hunting): get_auth_request con 0x7f0180142290 auth_method 0 2023-09-30T02:06:29.314 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.300+0000 7f01851f3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:29.315 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.300+0000 7f01851f3700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:29.315 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.300+0000 7f01851f3700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:29.315 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.300+0000 7f017ffff700 10 monclient(hunting): get_auth_request con 0x7f01800a5f40 auth_method 0 2023-09-30T02:06:29.315 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.300+0000 7f017ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:29.315 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.300+0000 7f017ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:29.315 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.300+0000 7f017ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:29.315 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.301+0000 7f01849f2700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:29.316 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.301+0000 7f01849f2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:29.316 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.301+0000 7f01849f2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:29.316 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.301+0000 7f01851f3700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:29.316 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.301+0000 7f01851f3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:29.316 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.301+0000 7f01851f3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:29.316 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.301+0000 7f017ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:29.316 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.301+0000 7f017ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:29.316 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.301+0000 7f017ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:29.319 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.305+0000 7f01849f2700 10 monclient(hunting): handle_auth_done global_id 4408 payload 824 2023-09-30T02:06:29.319 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.305+0000 7f01849f2700 1 -- 172.21.15.59:0/1309190621 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0180142290 msgr2=0x7f01800b3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:29.319 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.305+0000 7f01849f2700 1 --2- 172.21.15.59:0/1309190621 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0180142290 0x7f01800b3b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:29.319 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.305+0000 7f01849f2700 1 -- 172.21.15.59:0/1309190621 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01800a5f40 msgr2=0x7f0180151910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:29.319 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.305+0000 7f01849f2700 1 --2- 172.21.15.59:0/1309190621 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01800a5f40 0x7f0180151910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:29.319 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.305+0000 7f01849f2700 10 monclient: _finish_hunting 0 2023-09-30T02:06:29.320 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.305+0000 7f01849f2700 1 monclient: found mon.noname-b 2023-09-30T02:06:29.320 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.305+0000 7f01849f2700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:29.320 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.305+0000 7f01849f2700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:06:29.320 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.305+0000 7f01849f2700 1 -- 172.21.15.59:0/1309190621 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0170009420 con 0x7f01800a5b70 2023-09-30T02:06:29.320 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.308+0000 7f01849f2700 1 --2- 172.21.15.59:0/1309190621 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01800a5b70 0x7f01801513d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f01800b1ff0 tx=0x7f0170017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:29.320 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.308+0000 7f017cff9700 10 client.?.objecter ms_handle_connect 0x7f01800a5b70 2023-09-30T02:06:29.320 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.308+0000 7f017cff9700 10 client.?.objecter resend_mon_ops 2023-09-30T02:06:29.321 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.310+0000 7f017cff9700 1 -- 172.21.15.59:0/1309190621 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f017001f070 con 0x7f01800a5b70 2023-09-30T02:06:29.321 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.310+0000 7f017cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:29.321 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.310+0000 7f017cff9700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:06:29.321 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.310+0000 7f017cff9700 10 monclient: dump: 2023-09-30T02:06:29.321 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:29.321 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:29.321 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:29.321 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:29.322 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:29.322 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:29.322 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:29.322 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:29.322 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:29.322 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:29.322 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.310+0000 7f017cff9700 10 monclient: _finish_auth 0 2023-09-30T02:06:29.323 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.310+0000 7f017cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:29.323 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.310+0000 7f0186c56700 5 monclient: authenticate success, global_id 4408 2023-09-30T02:06:29.323 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.310+0000 7f0186c56700 10 monclient: _renew_subs 2023-09-30T02:06:29.323 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.310+0000 7f0186c56700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:29.323 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.310+0000 7f0186c56700 1 -- 172.21.15.59:0/1309190621 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01800b40c0 con 0x7f01800a5b70 2023-09-30T02:06:29.323 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.310+0000 7f0186c56700 10 client.4408.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:06:29.323 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.310+0000 7f0186c56700 10 monclient: _renew_subs 2023-09-30T02:06:29.323 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.310+0000 7f0186c56700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:29.324 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.310+0000 7f0186c56700 1 -- 172.21.15.59:0/1309190621 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01800b4550 con 0x7f01800a5b70 2023-09-30T02:06:29.324 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.310+0000 7f0186c56700 1 librados: init done 2023-09-30T02:06:29.324 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.311+0000 7f0186c56700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:29.324 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.311+0000 7f0186c56700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:29.324 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.311+0000 7f0186c56700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:29.324 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.311+0000 7f0186c56700 1 -- 172.21.15.59:0/1309190621 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01800b39d0 con 0x7f01800a5b70 2023-09-30T02:06:29.324 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.312+0000 7f017cff9700 1 -- 172.21.15.59:0/1309190621 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0170002f30 con 0x7f01800a5b70 2023-09-30T02:06:29.324 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.312+0000 7f017cff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:29.325 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.312+0000 7f017cff9700 1 -- 172.21.15.59:0/1309190621 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f017000bb20 con 0x7f01800a5b70 2023-09-30T02:06:29.325 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.312+0000 7f017cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:29.325 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.312+0000 7f017cff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:06:29.325 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.312+0000 7f017cff9700 10 monclient: dump: 2023-09-30T02:06:29.325 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:29.325 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:29.325 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:29.325 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:29.326 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:29.326 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:29.326 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:29.326 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:29.326 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:29.326 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:29.326 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.312+0000 7f017effd700 4 set_mon_vals no callback set 2023-09-30T02:06:29.560 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.559+0000 7f017cff9700 1 -- 172.21.15.59:0/1309190621 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f0170011420 con 0x7f01800a5b70 2023-09-30T02:06:29.560 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.560+0000 7f017cff9700 1 --2- 172.21.15.59:0/1309190621 >> v2:172.21.15.110:6800/115870 conn(0x7f0160068890 0x7f016006ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:29.560 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.560+0000 7f017ffff700 1 --2- 172.21.15.59:0/1309190621 >> v2:172.21.15.110:6800/115870 conn(0x7f0160068890 0x7f016006ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:29.561 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.560+0000 7f017ffff700 10 monclient: get_auth_request con 0x7f0160068890 auth_method 0 2023-09-30T02:06:29.561 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.560+0000 7f017cff9700 10 client.4408.objecter ms_handle_connect 0x7f0160068890 2023-09-30T02:06:29.561 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.560+0000 7f017ffff700 1 --2- 172.21.15.59:0/1309190621 >> v2:172.21.15.110:6800/115870 conn(0x7f0160068890 0x7f016006ad40 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0180144d80 tx=0x7f0168009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:29.564 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.563+0000 7f017cff9700 1 -- 172.21.15.59:0/1309190621 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f01700570b0 con 0x7f01800a5b70 2023-09-30T02:06:29.564 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.563+0000 7f017cff9700 10 client.4408.objecter ms_dispatch 0x7f0180001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:06:29.564 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.563+0000 7f017cff9700 3 client.4408.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:06:29.564 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.563+0000 7f017cff9700 3 client.4408.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:06:29.564 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.564+0000 7f017cff9700 20 client.4408.objecter dump_active .. 0 homeless 2023-09-30T02:06:29.573 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.573+0000 7f017cff9700 1 -- 172.21.15.59:0/1309190621 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0170052cf0 con 0x7f01800a5b70 2023-09-30T02:06:29.574 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.573+0000 7f017cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:29.574 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.573+0000 7f017cff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:06:29.831 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.830+0000 7f0186c56700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:06:29.831 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.830+0000 7f0186c56700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:06:29.831 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.830+0000 7f0186c56700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:29.831 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.830+0000 7f0186c56700 1 -- 172.21.15.59:0/1309190621 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f0180064a00 con 0x7f01800a5b70 2023-09-30T02:06:29.839 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.838+0000 7f017cff9700 1 -- 172.21.15.59:0/1309190621 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f0170056310 con 0x7f01800a5b70 2023-09-30T02:06:29.839 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.838+0000 7f017cff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:06:29.839 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.838+0000 7f017cff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:06:29.839 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:06:29.839 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:06:29.842 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.841+0000 7f0186c56700 10 librados: watch_flush enter 2023-09-30T02:06:29.842 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.841+0000 7f0186c56700 10 librados: watch_flush exit 2023-09-30T02:06:29.842 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.841+0000 7f0186c56700 20 client.4408.objecter shutdown clearing up homeless session... 2023-09-30T02:06:29.842 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.841+0000 7f0186c56700 10 client.4408.objecter successfully canceled tick 2023-09-30T02:06:29.842 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.841+0000 7f0186c56700 1 -- 172.21.15.59:0/1309190621 >> v2:172.21.15.110:6800/115870 conn(0x7f0160068890 msgr2=0x7f016006ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:29.842 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.841+0000 7f0186c56700 1 --2- 172.21.15.59:0/1309190621 >> v2:172.21.15.110:6800/115870 conn(0x7f0160068890 0x7f016006ad40 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0180144d80 tx=0x7f0168009040 comp rx=0 tx=0).stop 2023-09-30T02:06:29.842 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.841+0000 7f0186c56700 10 monclient: shutdown 2023-09-30T02:06:29.842 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.841+0000 7f0186c56700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:29.843 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.841+0000 7f0186c56700 1 -- 172.21.15.59:0/1309190621 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01800a5b70 msgr2=0x7f01801513d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:29.843 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.841+0000 7f0186c56700 1 --2- 172.21.15.59:0/1309190621 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01800a5b70 0x7f01801513d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f01800b1ff0 tx=0x7f0170017000 comp rx=0 tx=0).stop 2023-09-30T02:06:29.843 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.841+0000 7f0186c56700 1 -- 172.21.15.59:0/1309190621 shutdown_connections 2023-09-30T02:06:29.843 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.841+0000 7f0186c56700 1 --2- 172.21.15.59:0/1309190621 >> v2:172.21.15.110:6800/115870 conn(0x7f0160068890 0x7f016006ad40 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:29.843 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.841+0000 7f0186c56700 1 --2- 172.21.15.59:0/1309190621 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01800a5b70 0x7f01801513d0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:29.843 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.841+0000 7f0186c56700 1 --2- 172.21.15.59:0/1309190621 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0180142290 0x7f01800b3b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:29.843 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.841+0000 7f0186c56700 1 --2- 172.21.15.59:0/1309190621 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01800a5f40 0x7f0180151910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:29.843 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.841+0000 7f0186c56700 1 -- 172.21.15.59:0/1309190621 >> 172.21.15.59:0/1309190621 conn(0x7f01800631f0 msgr2=0x7f0180144db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:29.843 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.841+0000 7f0186c56700 1 -- 172.21.15.59:0/1309190621 shutdown_connections 2023-09-30T02:06:29.843 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.841+0000 7f0186c56700 1 -- 172.21.15.59:0/1309190621 wait complete. 2023-09-30T02:06:29.843 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:29.841+0000 7f0186c56700 1 librados: shutdown 2023-09-30T02:06:32.856 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:06:32.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:32.988+0000 7fca4e141700 10 monclient: get_monmap_and_config 2023-09-30T02:06:32.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:32.988+0000 7fca4e141700 10 monclient: build_initial_monmap 2023-09-30T02:06:32.989 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:32.988+0000 7fca4e141700 10 monclient: monmap: 2023-09-30T02:06:32.989 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:32.989 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:32.989 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:06:32.989641+0000 2023-09-30T02:06:32.989 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:06:32.989641+0000 2023-09-30T02:06:32.989 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:32.989 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:32.989 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:06:32.989 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:06:32.989 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:06:32.989 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:06:32.990 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:06:32.990 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:32.991 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:32.989+0000 7fca4e141700 1 Processor -- start 2023-09-30T02:06:32.991 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:32.989+0000 7fca4e141700 1 -- start start 2023-09-30T02:06:32.991 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:32.989+0000 7fca4e141700 10 monclient: init 2023-09-30T02:06:32.991 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:32.989+0000 7fca4e141700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:32.991 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:32.989+0000 7fca4e141700 10 monclient: _add_conns ranks=[0,3,1,2,4] 2023-09-30T02:06:32.991 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:32.989+0000 7fca4e141700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fca480a5b70 addr v1:172.21.15.59:6789/0 2023-09-30T02:06:32.992 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:32.989+0000 7fca4e141700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fca48142290 0x7fca480b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:32.992 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:32.989+0000 7fca4e141700 10 monclient(hunting): picked mon.noname-a con 0x7fca48142290 addr v2:172.21.15.59:3300/0 2023-09-30T02:06:32.992 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:32.989+0000 7fca4e141700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fca480a5f40 addr v1:172.21.15.110:6789/0 2023-09-30T02:06:32.992 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:32.989+0000 7fca4e141700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca480a75a0 con 0x7fca480a5b70 2023-09-30T02:06:32.992 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:32.989+0000 7fca4e141700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca480a6360 con 0x7fca480a5f40 2023-09-30T02:06:32.992 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:32.989+0000 7fca4e141700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:32.992 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:32.989+0000 7fca4e141700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca480af3f0 con 0x7fca48142290 2023-09-30T02:06:32.992 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:32.989+0000 7fca4e141700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:32.992 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:32.989+0000 7fca4e141700 10 monclient(hunting): authenticate will time out at 1838.292845s 2023-09-30T02:06:32.992 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:32.990+0000 7fca46ffd700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fca48142290 0x7fca480b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:32.992 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:32.990+0000 7fca47fff700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fca480a5f40 0x7fca4813af30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.59:55060/0 (socket says 172.21.15.59:55060) 2023-09-30T02:06:32.992 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:32.990+0000 7fca47fff700 1 -- 172.21.15.59:0/2480855119 learned_addr learned my addr 172.21.15.59:0/2480855119 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:06:32.993 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:32.991+0000 7fca46ffd700 10 monclient(hunting): get_auth_request con 0x7fca48142290 auth_method 0 2023-09-30T02:06:32.993 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:32.991+0000 7fca46ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:32.993 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:32.991+0000 7fca46ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:32.993 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:32.991+0000 7fca46ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:32.993 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:32.992+0000 7fca46ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:32.993 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:32.992+0000 7fca46ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:32.993 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:32.992+0000 7fca46ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:32.994 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:32.994+0000 7fca467fc700 1 -- 172.21.15.59:0/2480855119 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 120095596 0 0) 0x7fca480a6360 con 0x7fca480a5f40 2023-09-30T02:06:32.994 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:32.994+0000 7fca467fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:32.994 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:32.994+0000 7fca467fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:32.994 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:32.994+0000 7fca467fc700 10 monclient(hunting): my global_id is 4411 2023-09-30T02:06:32.994 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:32.994+0000 7fca467fc700 1 -- 172.21.15.59:0/2480855119 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca30003570 con 0x7fca480a5f40 2023-09-30T02:06:32.995 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:32.995+0000 7fca46ffd700 10 monclient(hunting): handle_auth_done global_id 4392 payload 274 2023-09-30T02:06:32.995 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:32.995+0000 7fca46ffd700 1 -- 172.21.15.59:0/2480855119 >> v1:172.21.15.110:6789/0 conn(0x7fca480a5f40 legacy=0x7fca4813af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:32.995 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:32.995+0000 7fca46ffd700 1 -- 172.21.15.59:0/2480855119 >> v1:172.21.15.59:6789/0 conn(0x7fca480a5b70 legacy=0x7fca48141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:32.995 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:32.995+0000 7fca46ffd700 10 monclient: _finish_hunting 0 2023-09-30T02:06:32.996 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:32.995+0000 7fca46ffd700 1 monclient: found mon.noname-a 2023-09-30T02:06:32.996 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:32.995+0000 7fca46ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:32.996 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:32.995+0000 7fca46ffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.59:3300/0 2023-09-30T02:06:32.996 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:32.995+0000 7fca46ffd700 1 -- 172.21.15.59:0/2480855119 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca4813b640 con 0x7fca48142290 2023-09-30T02:06:32.998 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:32.997+0000 7fca46ffd700 1 --2- 172.21.15.59:0/2480855119 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca48142290 0x7fca480b1500 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fca3c0098e0 tx=0x7fca3c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d968c0acf2e3a881 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:33.006 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.005+0000 7fca467fc700 1 -- 172.21.15.59:0/2480855119 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca3c00b640 con 0x7fca48142290 2023-09-30T02:06:33.006 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.005+0000 7fca467fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:33.006 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.005+0000 7fca467fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T02:06:33.006 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.005+0000 7fca467fc700 10 monclient: dump: 2023-09-30T02:06:33.006 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:33.006 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:33.006 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:33.006 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:33.006 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:33.007 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:33.007 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:33.007 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:33.007 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:33.007 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:33.007 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.005+0000 7fca467fc700 10 monclient: _finish_auth 0 2023-09-30T02:06:33.007 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.005+0000 7fca467fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:33.007 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.005+0000 7fca467fc700 1 -- 172.21.15.59:0/2480855119 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca3c00b7a0 con 0x7fca48142290 2023-09-30T02:06:33.007 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.005+0000 7fca4e141700 5 monclient: authenticate success, global_id 4392 2023-09-30T02:06:33.007 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.005+0000 7fca4e141700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:06:33.007 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.005+0000 7fca467fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:33.007 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.005+0000 7fca4e141700 10 monclient: get_monmap_and_config success 2023-09-30T02:06:33.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.005+0000 7fca467fc700 1 -- 172.21.15.59:0/2480855119 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca3c00bb40 con 0x7fca48142290 2023-09-30T02:06:33.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.005+0000 7fca4e141700 4 set_mon_vals no callback set 2023-09-30T02:06:33.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.005+0000 7fca467fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:33.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.005+0000 7fca467fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T02:06:33.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.005+0000 7fca467fc700 10 monclient: dump: 2023-09-30T02:06:33.008 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:33.008 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:33.008 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:33.008 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:33.008 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:33.008 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:33.008 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:33.008 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:33.009 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:33.009 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:33.012 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.006+0000 7fca4e141700 10 monclient: shutdown 2023-09-30T02:06:33.012 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.006+0000 7fca4e141700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:33.012 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.006+0000 7fca4e141700 1 -- 172.21.15.59:0/2480855119 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca48142290 msgr2=0x7fca480b1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:33.012 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.006+0000 7fca4e141700 1 --2- 172.21.15.59:0/2480855119 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca48142290 0x7fca480b1500 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fca3c0098e0 tx=0x7fca3c0091a0 comp rx=0 tx=0).stop 2023-09-30T02:06:33.013 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.006+0000 7fca4e141700 1 -- 172.21.15.59:0/2480855119 shutdown_connections 2023-09-30T02:06:33.013 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.006+0000 7fca4e141700 1 --2- 172.21.15.59:0/2480855119 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca48142290 0x7fca480b1500 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:33.013 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.006+0000 7fca4e141700 1 -- 172.21.15.59:0/2480855119 >> 172.21.15.59:0/2480855119 conn(0x7fca480631f0 msgr2=0x7fca480635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:33.013 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.006+0000 7fca4e141700 1 -- 172.21.15.59:0/2480855119 shutdown_connections 2023-09-30T02:06:33.013 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.006+0000 7fca4e141700 1 -- 172.21.15.59:0/2480855119 wait complete. 2023-09-30T02:06:33.013 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.007+0000 7fca4e141700 10 monclient: build_initial_monmap 2023-09-30T02:06:33.013 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.007+0000 7fca4e141700 10 monclient: monmap: 2023-09-30T02:06:33.013 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:33.013 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:06:33.013 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:06:33.013 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:06:33.013 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:33.014 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:06:33.014 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:06:33.014 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:06:33.014 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:06:33.014 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:33.014 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.007+0000 7fca4e141700 1 librados: starting msgr at 2023-09-30T02:06:33.014 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.007+0000 7fca4e141700 1 librados: starting objecter 2023-09-30T02:06:33.014 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.008+0000 7fca4e141700 1 Processor -- start 2023-09-30T02:06:33.014 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.008+0000 7fca4e141700 1 -- start start 2023-09-30T02:06:33.014 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.008+0000 7fca4e141700 1 librados: setting wanted keys 2023-09-30T02:06:33.014 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.008+0000 7fca4e141700 1 librados: calling monclient init 2023-09-30T02:06:33.014 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.008+0000 7fca4e141700 10 monclient: init 2023-09-30T02:06:33.015 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.009+0000 7fca4e141700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:33.015 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.009+0000 7fca4e141700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T02:06:33.015 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.009+0000 7fca4e141700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fca480a5b70 0x7fca48140ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:33.015 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.009+0000 7fca4e141700 10 monclient(hunting): picked mon.noname-b con 0x7fca480a5b70 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:06:33.015 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.009+0000 7fca4e141700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fca480a5f40 0x7fca4813e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:33.015 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.009+0000 7fca4e141700 10 monclient(hunting): picked mon.noname-c con 0x7fca480a5f40 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:33.015 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.009+0000 7fca4e141700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca48142290 0x7fca4814de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:33.015 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.009+0000 7fca4e141700 10 monclient(hunting): picked mon.noname-a con 0x7fca48142290 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:06:33.015 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.009+0000 7fca4e141700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:33.015 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.009+0000 7fca4e141700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca4813f100 con 0x7fca48142290 2023-09-30T02:06:33.015 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.009+0000 7fca4e141700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:33.015 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.009+0000 7fca4e141700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca4813eca0 con 0x7fca480a5b70 2023-09-30T02:06:33.016 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.009+0000 7fca4e141700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:33.016 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.009+0000 7fca4e141700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca4813ee10 con 0x7fca480a5f40 2023-09-30T02:06:33.016 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.009+0000 7fca4e141700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:33.016 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.009+0000 7fca4e141700 10 monclient(hunting): authenticate will time out at 1838.312163s 2023-09-30T02:06:33.016 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.009+0000 7fca47fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca48142290 0x7fca4814de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:33.016 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.009+0000 7fca477fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fca480a5b70 0x7fca48140ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:33.016 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.009+0000 7fca46ffd700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fca480a5f40 0x7fca4813e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:33.016 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.010+0000 7fca47fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca48142290 0x7fca4814de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34036/0 (socket says 172.21.15.59:34036) 2023-09-30T02:06:33.016 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.010+0000 7fca47fff700 1 -- 172.21.15.59:0/2002996627 learned_addr learned my addr 172.21.15.59:0/2002996627 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:06:33.016 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.010+0000 7fca47fff700 10 monclient(hunting): get_auth_request con 0x7fca48142290 auth_method 0 2023-09-30T02:06:33.016 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.010+0000 7fca47fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:33.016 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.010+0000 7fca47fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:33.017 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.010+0000 7fca47fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:33.017 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.010+0000 7fca477fe700 10 monclient(hunting): get_auth_request con 0x7fca480a5b70 auth_method 0 2023-09-30T02:06:33.017 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.010+0000 7fca477fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:33.017 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.010+0000 7fca477fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:33.017 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.010+0000 7fca477fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:33.017 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.010+0000 7fca46ffd700 10 monclient(hunting): get_auth_request con 0x7fca480a5f40 auth_method 0 2023-09-30T02:06:33.017 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.010+0000 7fca46ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:33.017 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.010+0000 7fca46ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:33.017 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.010+0000 7fca46ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:33.017 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.010+0000 7fca47fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:33.017 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.010+0000 7fca47fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:33.017 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.010+0000 7fca47fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:33.017 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.010+0000 7fca477fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:33.018 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.010+0000 7fca477fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:33.018 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.010+0000 7fca477fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:33.018 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.011+0000 7fca46ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:33.018 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.011+0000 7fca46ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:33.018 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.011+0000 7fca46ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:33.019 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.014+0000 7fca477fe700 10 monclient(hunting): handle_auth_done global_id 4414 payload 824 2023-09-30T02:06:33.019 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.014+0000 7fca477fe700 1 -- 172.21.15.59:0/2002996627 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fca480a5f40 msgr2=0x7fca4813e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:33.019 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.014+0000 7fca477fe700 1 --2- 172.21.15.59:0/2002996627 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fca480a5f40 0x7fca4813e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:33.019 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.014+0000 7fca477fe700 1 -- 172.21.15.59:0/2002996627 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca48142290 msgr2=0x7fca4814de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:33.019 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.014+0000 7fca477fe700 1 --2- 172.21.15.59:0/2002996627 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca48142290 0x7fca4814de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:33.019 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.014+0000 7fca477fe700 10 monclient: _finish_hunting 0 2023-09-30T02:06:33.019 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.014+0000 7fca477fe700 1 monclient: found mon.noname-b 2023-09-30T02:06:33.019 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.014+0000 7fca477fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:33.019 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.014+0000 7fca477fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:06:33.019 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.014+0000 7fca477fe700 1 -- 172.21.15.59:0/2002996627 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca3c009420 con 0x7fca480a5b70 2023-09-30T02:06:33.020 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.017+0000 7fca477fe700 1 --2- 172.21.15.59:0/2002996627 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fca480a5b70 0x7fca48140ca0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fca3800c590 tx=0x7fca38007e80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:33.020 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.017+0000 7fca2f7fe700 10 client.?.objecter ms_handle_connect 0x7fca480a5b70 2023-09-30T02:06:33.020 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.017+0000 7fca2f7fe700 10 client.?.objecter resend_mon_ops 2023-09-30T02:06:33.022 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.021+0000 7fca2f7fe700 1 -- 172.21.15.59:0/2002996627 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca38011590 con 0x7fca480a5b70 2023-09-30T02:06:33.022 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.021+0000 7fca2f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:33.022 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.021+0000 7fca2f7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:06:33.022 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.021+0000 7fca2f7fe700 10 monclient: dump: 2023-09-30T02:06:33.022 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:33.023 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:33.023 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:33.023 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:33.023 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:33.023 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:33.023 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:33.023 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:33.023 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:33.023 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:33.023 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.021+0000 7fca2f7fe700 10 monclient: _finish_auth 0 2023-09-30T02:06:33.023 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.021+0000 7fca2f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:33.023 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.021+0000 7fca2f7fe700 1 -- 172.21.15.59:0/2002996627 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca38011710 con 0x7fca480a5b70 2023-09-30T02:06:33.024 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.021+0000 7fca2f7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:33.024 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.021+0000 7fca2f7fe700 1 -- 172.21.15.59:0/2002996627 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca38011ad0 con 0x7fca480a5b70 2023-09-30T02:06:33.024 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.021+0000 7fca4e141700 5 monclient: authenticate success, global_id 4414 2023-09-30T02:06:33.024 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.021+0000 7fca2f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:33.024 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.021+0000 7fca2f7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:06:33.024 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.021+0000 7fca2f7fe700 10 monclient: dump: 2023-09-30T02:06:33.024 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:33.024 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:33.024 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:33.024 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:33.024 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:33.024 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:33.024 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:33.025 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:33.025 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:33.025 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:33.025 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.021+0000 7fca4e141700 10 monclient: _renew_subs 2023-09-30T02:06:33.025 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.021+0000 7fca4e141700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:33.025 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.021+0000 7fca457fa700 4 set_mon_vals no callback set 2023-09-30T02:06:33.025 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.021+0000 7fca4e141700 1 -- 172.21.15.59:0/2002996627 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca4813e350 con 0x7fca480a5b70 2023-09-30T02:06:33.025 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.021+0000 7fca4e141700 10 client.4414.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:06:33.025 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.021+0000 7fca4e141700 10 monclient: _renew_subs 2023-09-30T02:06:33.025 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.021+0000 7fca4e141700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:33.025 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.021+0000 7fca4e141700 1 -- 172.21.15.59:0/2002996627 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca4814e560 con 0x7fca480a5b70 2023-09-30T02:06:33.025 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.021+0000 7fca4e141700 1 librados: init done 2023-09-30T02:06:33.026 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.022+0000 7fca4e141700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:33.026 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.022+0000 7fca4e141700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:33.026 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.022+0000 7fca4e141700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:33.026 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.022+0000 7fca4e141700 1 -- 172.21.15.59:0/2002996627 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca48064a00 con 0x7fca480a5b70 2023-09-30T02:06:33.252 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.252+0000 7fca2f7fe700 1 -- 172.21.15.59:0/2002996627 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fca3801b3b0 con 0x7fca480a5b70 2023-09-30T02:06:33.253 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.252+0000 7fca2f7fe700 1 --2- 172.21.15.59:0/2002996627 >> v2:172.21.15.110:6800/115870 conn(0x7fca340689e0 0x7fca3406ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:33.253 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.252+0000 7fca2f7fe700 1 -- 172.21.15.59:0/2002996627 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fca38083610 con 0x7fca480a5b70 2023-09-30T02:06:33.253 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.252+0000 7fca2f7fe700 10 client.4414.objecter ms_dispatch 0x7fca48001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:06:33.253 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.252+0000 7fca2f7fe700 3 client.4414.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:06:33.253 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.252+0000 7fca2f7fe700 3 client.4414.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:06:33.253 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.252+0000 7fca2f7fe700 20 client.4414.objecter dump_active .. 0 homeless 2023-09-30T02:06:33.254 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.252+0000 7fca46ffd700 1 --2- 172.21.15.59:0/2002996627 >> v2:172.21.15.110:6800/115870 conn(0x7fca340689e0 0x7fca3406ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:33.254 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.253+0000 7fca46ffd700 10 monclient: get_auth_request con 0x7fca340689e0 auth_method 0 2023-09-30T02:06:33.254 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.253+0000 7fca2f7fe700 10 client.4414.objecter ms_handle_connect 0x7fca340689e0 2023-09-30T02:06:33.254 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.253+0000 7fca46ffd700 1 --2- 172.21.15.59:0/2002996627 >> v2:172.21.15.110:6800/115870 conn(0x7fca340689e0 0x7fca3406ae90 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fca3c0093f0 tx=0x7fca3c017000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:33.282 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.282+0000 7fca2f7fe700 1 -- 172.21.15.59:0/2002996627 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fca38050f30 con 0x7fca480a5b70 2023-09-30T02:06:33.282 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.282+0000 7fca2f7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:33.282 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.282+0000 7fca2f7fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:06:33.335 INFO:tasks.ceph.mon.a.smithi059.stderr:==00:00:01:43.200 116256== Warning: unimplemented fcntl command: 1036 2023-09-30T02:06:33.342 INFO:tasks.ceph.mon.b.smithi110.stderr:==00:00:01:43.189 115869== Warning: unimplemented fcntl command: 1036 2023-09-30T02:06:33.344 INFO:tasks.ceph.mon.b.smithi110.stderr:==00:00:01:43.191 115869== Warning: unimplemented fcntl command: 1036 2023-09-30T02:06:33.365 INFO:tasks.ceph.mon.a.smithi059.stderr:==00:00:01:43.230 116256== Warning: unimplemented fcntl command: 1036 2023-09-30T02:06:33.403 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:01:43.268 116257== Warning: unimplemented fcntl command: 1036 2023-09-30T02:06:33.406 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:01:43.272 116257== Warning: unimplemented fcntl command: 1036 2023-09-30T02:06:33.406 INFO:tasks.ceph.mon.b.smithi110.stderr:==00:00:01:43.253 115869== Warning: unimplemented fcntl command: 1036 2023-09-30T02:06:33.475 INFO:tasks.ceph.mon.a.smithi059.stderr:==00:00:01:43.340 116256== Warning: unimplemented fcntl command: 1036 2023-09-30T02:06:33.476 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:01:43.341 116257== Warning: unimplemented fcntl command: 1036 2023-09-30T02:06:33.615 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.615+0000 7fca4e141700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:06:33.660 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.615+0000 7fca4e141700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:06:33.660 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.615+0000 7fca4e141700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:33.660 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.615+0000 7fca4e141700 1 -- 172.21.15.59:0/2002996627 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fca4806a720 con 0x7fca480a5b70 2023-09-30T02:06:33.661 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.620+0000 7fca2f7fe700 1 -- 172.21.15.59:0/2002996627 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fca38017100 con 0x7fca480a5b70 2023-09-30T02:06:33.661 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.620+0000 7fca2f7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:06:33.661 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.620+0000 7fca2f7fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:06:33.661 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:06:33.662 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:06:33.664 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.622+0000 7fca4e141700 10 librados: watch_flush enter 2023-09-30T02:06:33.664 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.622+0000 7fca4e141700 10 librados: watch_flush exit 2023-09-30T02:06:33.664 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.622+0000 7fca4e141700 20 client.4414.objecter shutdown clearing up homeless session... 2023-09-30T02:06:33.664 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.622+0000 7fca4e141700 10 client.4414.objecter successfully canceled tick 2023-09-30T02:06:33.664 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.622+0000 7fca4e141700 1 -- 172.21.15.59:0/2002996627 >> v2:172.21.15.110:6800/115870 conn(0x7fca340689e0 msgr2=0x7fca3406ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:33.664 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.622+0000 7fca4e141700 1 --2- 172.21.15.59:0/2002996627 >> v2:172.21.15.110:6800/115870 conn(0x7fca340689e0 0x7fca3406ae90 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fca3c0093f0 tx=0x7fca3c017000 comp rx=0 tx=0).stop 2023-09-30T02:06:33.664 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.623+0000 7fca4e141700 10 monclient: shutdown 2023-09-30T02:06:33.664 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.623+0000 7fca4e141700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:33.664 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.623+0000 7fca4e141700 1 -- 172.21.15.59:0/2002996627 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fca480a5b70 msgr2=0x7fca48140ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:33.664 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.623+0000 7fca4e141700 1 --2- 172.21.15.59:0/2002996627 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fca480a5b70 0x7fca48140ca0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fca3800c590 tx=0x7fca38007e80 comp rx=0 tx=0).stop 2023-09-30T02:06:33.665 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.623+0000 7fca4e141700 1 -- 172.21.15.59:0/2002996627 shutdown_connections 2023-09-30T02:06:33.665 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.623+0000 7fca4e141700 1 --2- 172.21.15.59:0/2002996627 >> v2:172.21.15.110:6800/115870 conn(0x7fca340689e0 0x7fca3406ae90 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:33.665 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.623+0000 7fca4e141700 1 --2- 172.21.15.59:0/2002996627 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fca480a5b70 0x7fca48140ca0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:33.665 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.623+0000 7fca4e141700 1 --2- 172.21.15.59:0/2002996627 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca48142290 0x7fca4814de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:33.665 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.623+0000 7fca4e141700 1 --2- 172.21.15.59:0/2002996627 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fca480a5f40 0x7fca4813e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:33.665 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.623+0000 7fca4e141700 1 -- 172.21.15.59:0/2002996627 >> 172.21.15.59:0/2002996627 conn(0x7fca480631f0 msgr2=0x7fca48070f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:33.665 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.623+0000 7fca4e141700 1 -- 172.21.15.59:0/2002996627 shutdown_connections 2023-09-30T02:06:33.665 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.623+0000 7fca4e141700 1 -- 172.21.15.59:0/2002996627 wait complete. 2023-09-30T02:06:33.665 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:33.623+0000 7fca4e141700 1 librados: shutdown 2023-09-30T02:06:36.666 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:06:36.811 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.811+0000 7f1e6bcfe700 10 monclient: get_monmap_and_config 2023-09-30T02:06:36.812 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.811+0000 7f1e6bcfe700 10 monclient: build_initial_monmap 2023-09-30T02:06:36.812 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.811+0000 7f1e6bcfe700 10 monclient: monmap: 2023-09-30T02:06:36.812 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:36.812 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:36.812 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:06:36.812681+0000 2023-09-30T02:06:36.812 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:06:36.812681+0000 2023-09-30T02:06:36.812 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:36.812 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:36.812 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:06:36.812 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:06:36.812 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:06:36.812 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:06:36.813 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:06:36.813 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:36.814 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.811+0000 7f1e6bcfe700 1 Processor -- start 2023-09-30T02:06:36.814 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.812+0000 7f1e6bcfe700 1 -- start start 2023-09-30T02:06:36.814 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.812+0000 7f1e6bcfe700 10 monclient: init 2023-09-30T02:06:36.815 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.812+0000 7f1e6bcfe700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:36.815 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.812+0000 7f1e6bcfe700 10 monclient: _add_conns ranks=[1,0,3,4,2] 2023-09-30T02:06:36.815 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.812+0000 7f1e6bcfe700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1e640a5b70 addr v1:172.21.15.110:6789/0 2023-09-30T02:06:36.815 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.812+0000 7f1e6bcfe700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1e64142290 addr v1:172.21.15.59:6789/0 2023-09-30T02:06:36.815 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.812+0000 7f1e6bcfe700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f1e640b1500 0x7f1e640a5f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:36.815 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.812+0000 7f1e6bcfe700 10 monclient(hunting): picked mon.noname-a con 0x7f1e640b1500 addr v2:172.21.15.59:3300/0 2023-09-30T02:06:36.815 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.812+0000 7f1e6bcfe700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e640b1920 con 0x7f1e64142290 2023-09-30T02:06:36.815 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.812+0000 7f1e6bcfe700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e640a75a0 con 0x7f1e640a5b70 2023-09-30T02:06:36.815 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.812+0000 7f1e6bcfe700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:36.815 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.812+0000 7f1e6bcfe700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e640af3f0 con 0x7f1e640b1500 2023-09-30T02:06:36.815 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.812+0000 7f1e6bcfe700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:36.815 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.812+0000 7f1e6bcfe700 10 monclient(hunting): authenticate will time out at 1842.115954s 2023-09-30T02:06:36.815 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.813+0000 7f1e6a29b700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f1e640b1500 0x7f1e640a5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:36.816 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.813+0000 7f1e69a9a700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f1e640a5b70 0x7f1e64141b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.59:55076/0 (socket says 172.21.15.59:55076) 2023-09-30T02:06:36.816 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.813+0000 7f1e69a9a700 1 -- 172.21.15.59:0/2228226269 learned_addr learned my addr 172.21.15.59:0/2228226269 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:06:36.816 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.814+0000 7f1e6a29b700 10 monclient(hunting): get_auth_request con 0x7f1e640b1500 auth_method 0 2023-09-30T02:06:36.816 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.814+0000 7f1e6a29b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:36.816 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.814+0000 7f1e6a29b700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:36.816 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.814+0000 7f1e6a29b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:36.816 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.815+0000 7f1e6a29b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:36.817 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.815+0000 7f1e6a29b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:36.817 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.815+0000 7f1e6a29b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:36.817 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.817+0000 7f1e68a98700 1 -- 172.21.15.59:0/2228226269 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 557817971 0 0) 0x7f1e640a75a0 con 0x7f1e640a5b70 2023-09-30T02:06:36.817 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.817+0000 7f1e68a98700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:36.817 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.817+0000 7f1e68a98700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:36.817 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.817+0000 7f1e68a98700 10 monclient(hunting): my global_id is 4417 2023-09-30T02:06:36.817 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.817+0000 7f1e68a98700 1 -- 172.21.15.59:0/2228226269 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e50003570 con 0x7f1e640a5b70 2023-09-30T02:06:36.819 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.818+0000 7f1e6a29b700 10 monclient(hunting): handle_auth_done global_id 4401 payload 274 2023-09-30T02:06:36.819 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.818+0000 7f1e6a29b700 1 -- 172.21.15.59:0/2228226269 >> v1:172.21.15.110:6789/0 conn(0x7f1e640a5b70 legacy=0x7f1e64141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:36.819 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.818+0000 7f1e6a29b700 1 -- 172.21.15.59:0/2228226269 >> v1:172.21.15.59:6789/0 conn(0x7f1e64142290 legacy=0x7f1e6413af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:36.819 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.818+0000 7f1e6a29b700 10 monclient: _finish_hunting 0 2023-09-30T02:06:36.819 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.818+0000 7f1e6a29b700 1 monclient: found mon.noname-a 2023-09-30T02:06:36.819 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.818+0000 7f1e6a29b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:36.819 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.818+0000 7f1e6a29b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.59:3300/0 2023-09-30T02:06:36.819 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.818+0000 7f1e6a29b700 1 -- 172.21.15.59:0/2228226269 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e6413b640 con 0x7f1e640b1500 2023-09-30T02:06:36.821 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.821+0000 7f1e6a29b700 1 --2- 172.21.15.59:0/2228226269 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1e640b1500 0x7f1e640a5f40 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f1e5c009e60 tx=0x7f1e5c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc181f7037137cbe server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:36.831 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.831+0000 7f1e68a98700 1 -- 172.21.15.59:0/2228226269 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e5c00e070 con 0x7f1e640b1500 2023-09-30T02:06:36.831 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.831+0000 7f1e68a98700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:36.831 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.831+0000 7f1e68a98700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T02:06:36.831 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.831+0000 7f1e68a98700 10 monclient: dump: 2023-09-30T02:06:36.831 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:36.832 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:36.832 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:36.832 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:36.832 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:36.832 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:36.832 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:36.832 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:36.832 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:36.832 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:36.832 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.831+0000 7f1e68a98700 10 monclient: _finish_auth 0 2023-09-30T02:06:36.832 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.831+0000 7f1e68a98700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:36.832 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.831+0000 7f1e68a98700 1 -- 172.21.15.59:0/2228226269 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e5c00bb60 con 0x7f1e640b1500 2023-09-30T02:06:36.833 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.831+0000 7f1e6bcfe700 5 monclient: authenticate success, global_id 4401 2023-09-30T02:06:36.833 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.831+0000 7f1e6bcfe700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:06:36.833 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.831+0000 7f1e68a98700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:36.833 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.831+0000 7f1e6bcfe700 10 monclient: get_monmap_and_config success 2023-09-30T02:06:36.833 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.831+0000 7f1e6bcfe700 4 set_mon_vals no callback set 2023-09-30T02:06:36.833 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.831+0000 7f1e68a98700 1 -- 172.21.15.59:0/2228226269 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e5c012430 con 0x7f1e640b1500 2023-09-30T02:06:36.833 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.831+0000 7f1e68a98700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:36.833 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.831+0000 7f1e68a98700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T02:06:36.833 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.831+0000 7f1e68a98700 10 monclient: dump: 2023-09-30T02:06:36.833 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:36.833 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:36.833 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:36.833 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:36.834 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:36.834 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:36.834 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:36.834 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:36.834 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:36.834 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:36.837 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.831+0000 7f1e6bcfe700 10 monclient: shutdown 2023-09-30T02:06:36.837 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.831+0000 7f1e6bcfe700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:36.837 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.831+0000 7f1e6bcfe700 1 -- 172.21.15.59:0/2228226269 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1e640b1500 msgr2=0x7f1e640a5f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:36.837 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.831+0000 7f1e6bcfe700 1 --2- 172.21.15.59:0/2228226269 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1e640b1500 0x7f1e640a5f40 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f1e5c009e60 tx=0x7f1e5c009780 comp rx=0 tx=0).stop 2023-09-30T02:06:36.837 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.831+0000 7f1e6bcfe700 1 -- 172.21.15.59:0/2228226269 shutdown_connections 2023-09-30T02:06:36.837 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.831+0000 7f1e6bcfe700 1 --2- 172.21.15.59:0/2228226269 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1e640b1500 0x7f1e640a5f40 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:36.837 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.831+0000 7f1e6bcfe700 1 -- 172.21.15.59:0/2228226269 >> 172.21.15.59:0/2228226269 conn(0x7f1e640631f0 msgr2=0x7f1e640635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:36.837 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.832+0000 7f1e6bcfe700 1 -- 172.21.15.59:0/2228226269 shutdown_connections 2023-09-30T02:06:36.838 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.832+0000 7f1e6bcfe700 1 -- 172.21.15.59:0/2228226269 wait complete. 2023-09-30T02:06:36.838 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.832+0000 7f1e6bcfe700 10 monclient: build_initial_monmap 2023-09-30T02:06:36.838 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.832+0000 7f1e6bcfe700 10 monclient: monmap: 2023-09-30T02:06:36.838 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:36.838 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:06:36.838 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:06:36.838 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:06:36.838 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:36.838 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:06:36.838 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:06:36.838 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:06:36.839 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:06:36.839 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:36.839 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.832+0000 7f1e6bcfe700 1 librados: starting msgr at 2023-09-30T02:06:36.839 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.832+0000 7f1e6bcfe700 1 librados: starting objecter 2023-09-30T02:06:36.839 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.833+0000 7f1e6bcfe700 1 Processor -- start 2023-09-30T02:06:36.839 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.833+0000 7f1e6bcfe700 1 -- start start 2023-09-30T02:06:36.839 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.833+0000 7f1e6bcfe700 1 librados: setting wanted keys 2023-09-30T02:06:36.839 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.833+0000 7f1e6bcfe700 1 librados: calling monclient init 2023-09-30T02:06:36.839 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.833+0000 7f1e6bcfe700 10 monclient: init 2023-09-30T02:06:36.839 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.833+0000 7f1e6bcfe700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:36.839 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.833+0000 7f1e6bcfe700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T02:06:36.839 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.833+0000 7f1e6bcfe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e640a5b70 0x7f1e641450b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:36.840 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.833+0000 7f1e6bcfe700 10 monclient(hunting): picked mon.noname-b con 0x7f1e640a5b70 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:06:36.840 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.833+0000 7f1e6bcfe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1e640b1500 0x7f1e64151470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:36.840 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.833+0000 7f1e6bcfe700 10 monclient(hunting): picked mon.noname-a con 0x7f1e640b1500 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:06:36.840 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.833+0000 7f1e6bcfe700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1e64142290 0x7f1e641519b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:36.840 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.833+0000 7f1e6bcfe700 10 monclient(hunting): picked mon.noname-c con 0x7f1e64142290 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:36.840 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.833+0000 7f1e6bcfe700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:36.840 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.833+0000 7f1e6bcfe700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e6413d3f0 con 0x7f1e640b1500 2023-09-30T02:06:36.840 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.833+0000 7f1e6bcfe700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:36.840 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.833+0000 7f1e6bcfe700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e6413e2e0 con 0x7f1e640a5b70 2023-09-30T02:06:36.840 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.833+0000 7f1e6bcfe700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:36.840 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.833+0000 7f1e6bcfe700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e64151ef0 con 0x7f1e64142290 2023-09-30T02:06:36.840 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.833+0000 7f1e6bcfe700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:36.841 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.833+0000 7f1e6bcfe700 10 monclient(hunting): authenticate will time out at 1842.136970s 2023-09-30T02:06:36.841 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.834+0000 7f1e69a9a700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e640a5b70 0x7f1e641450b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:36.841 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.834+0000 7f1e6a29b700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1e64142290 0x7f1e641519b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:36.841 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.834+0000 7f1e69299700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1e640b1500 0x7f1e64151470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:36.841 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.835+0000 7f1e69a9a700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e640a5b70 0x7f1e641450b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:38654/0 (socket says 172.21.15.59:38654) 2023-09-30T02:06:36.841 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.835+0000 7f1e69a9a700 1 -- 172.21.15.59:0/1239317004 learned_addr learned my addr 172.21.15.59:0/1239317004 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:06:36.841 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.835+0000 7f1e69a9a700 10 monclient(hunting): get_auth_request con 0x7f1e640a5b70 auth_method 0 2023-09-30T02:06:36.841 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.835+0000 7f1e69a9a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:36.841 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.835+0000 7f1e69a9a700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:36.841 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.835+0000 7f1e69a9a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:36.841 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.835+0000 7f1e6a29b700 10 monclient(hunting): get_auth_request con 0x7f1e64142290 auth_method 0 2023-09-30T02:06:36.841 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.835+0000 7f1e6a29b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:36.841 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.835+0000 7f1e6a29b700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:36.842 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.835+0000 7f1e6a29b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:36.842 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.835+0000 7f1e69299700 10 monclient(hunting): get_auth_request con 0x7f1e640b1500 auth_method 0 2023-09-30T02:06:36.842 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.835+0000 7f1e69299700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:36.842 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.835+0000 7f1e69299700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:36.842 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.835+0000 7f1e69299700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:36.842 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.835+0000 7f1e69a9a700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:36.842 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.835+0000 7f1e69a9a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:36.842 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.835+0000 7f1e69a9a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:36.842 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.835+0000 7f1e69299700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:36.842 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.835+0000 7f1e69299700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:36.842 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.835+0000 7f1e69299700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:36.842 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.836+0000 7f1e6a29b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:36.843 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.836+0000 7f1e6a29b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:36.843 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.836+0000 7f1e6a29b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:36.843 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.840+0000 7f1e69299700 10 monclient(hunting): handle_auth_done global_id 4407 payload 824 2023-09-30T02:06:36.844 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.840+0000 7f1e69299700 1 -- 172.21.15.59:0/1239317004 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1e64142290 msgr2=0x7f1e641519b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:36.844 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.840+0000 7f1e69299700 1 --2- 172.21.15.59:0/1239317004 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1e64142290 0x7f1e641519b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:36.844 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.840+0000 7f1e69299700 1 -- 172.21.15.59:0/1239317004 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e640a5b70 msgr2=0x7f1e641450b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:36.844 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.840+0000 7f1e69299700 1 --2- 172.21.15.59:0/1239317004 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e640a5b70 0x7f1e641450b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:36.844 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.840+0000 7f1e69299700 10 monclient: _finish_hunting 0 2023-09-30T02:06:36.844 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.840+0000 7f1e69299700 1 monclient: found mon.noname-a 2023-09-30T02:06:36.844 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.840+0000 7f1e69299700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:36.844 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.840+0000 7f1e69299700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.59:3300/0 2023-09-30T02:06:36.844 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.840+0000 7f1e69299700 1 -- 172.21.15.59:0/1239317004 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e5c009a20 con 0x7f1e640b1500 2023-09-30T02:06:36.844 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.842+0000 7f1e69299700 1 --2- 172.21.15.59:0/1239317004 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1e640b1500 0x7f1e64151470 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1e5400c4c0 tx=0x7f1e54007ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:36.844 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.842+0000 7f1e59ffb700 10 client.?.objecter ms_handle_connect 0x7f1e640b1500 2023-09-30T02:06:36.845 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.842+0000 7f1e59ffb700 10 client.?.objecter resend_mon_ops 2023-09-30T02:06:36.849 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.848+0000 7f1e59ffb700 1 -- 172.21.15.59:0/1239317004 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e5400ce30 con 0x7f1e640b1500 2023-09-30T02:06:36.849 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.848+0000 7f1e59ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:36.849 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.848+0000 7f1e59ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T02:06:36.849 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.848+0000 7f1e59ffb700 10 monclient: dump: 2023-09-30T02:06:36.849 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:36.849 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:36.849 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:36.849 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:36.849 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:36.849 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:36.850 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:36.850 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:36.850 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:36.850 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:36.850 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.848+0000 7f1e59ffb700 10 monclient: _finish_auth 0 2023-09-30T02:06:36.850 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.848+0000 7f1e59ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:36.850 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.848+0000 7f1e59ffb700 1 -- 172.21.15.59:0/1239317004 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e54011590 con 0x7f1e640b1500 2023-09-30T02:06:36.850 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.848+0000 7f1e6bcfe700 5 monclient: authenticate success, global_id 4407 2023-09-30T02:06:36.850 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.848+0000 7f1e6bcfe700 10 monclient: _renew_subs 2023-09-30T02:06:36.850 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.848+0000 7f1e6bcfe700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:06:36.850 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.848+0000 7f1e6bcfe700 1 -- 172.21.15.59:0/1239317004 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e6413dee0 con 0x7f1e640b1500 2023-09-30T02:06:36.850 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.848+0000 7f1e59ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:36.851 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.848+0000 7f1e59ffb700 1 -- 172.21.15.59:0/1239317004 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e54011950 con 0x7f1e640b1500 2023-09-30T02:06:36.851 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.848+0000 7f1e59ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:36.851 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.848+0000 7f1e59ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T02:06:36.851 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.848+0000 7f1e59ffb700 10 monclient: dump: 2023-09-30T02:06:36.851 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:36.851 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:36.851 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:36.851 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:36.851 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:36.851 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:36.851 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:36.851 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:36.852 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:36.852 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:36.852 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.848+0000 7f1e5bfff700 4 set_mon_vals no callback set 2023-09-30T02:06:36.852 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.848+0000 7f1e6bcfe700 10 client.4407.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:06:36.852 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.848+0000 7f1e6bcfe700 10 monclient: _renew_subs 2023-09-30T02:06:36.852 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.848+0000 7f1e6bcfe700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:06:36.852 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.848+0000 7f1e6bcfe700 1 -- 172.21.15.59:0/1239317004 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e6413eef0 con 0x7f1e640b1500 2023-09-30T02:06:36.852 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.848+0000 7f1e6bcfe700 1 librados: init done 2023-09-30T02:06:36.852 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.850+0000 7f1e6bcfe700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:36.853 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.850+0000 7f1e6bcfe700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:36.853 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.850+0000 7f1e6bcfe700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:06:36.853 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:36.850+0000 7f1e6bcfe700 1 -- 172.21.15.59:0/1239317004 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e6413e470 con 0x7f1e640b1500 2023-09-30T02:06:37.149 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:37.149+0000 7f1e59ffb700 1 -- 172.21.15.59:0/1239317004 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f1e54011cc0 con 0x7f1e640b1500 2023-09-30T02:06:37.150 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:37.149+0000 7f1e59ffb700 1 --2- 172.21.15.59:0/1239317004 >> v2:172.21.15.110:6800/115870 conn(0x7f1e48068960 0x7f1e4806ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:37.150 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:37.149+0000 7f1e59ffb700 1 -- 172.21.15.59:0/1239317004 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f1e54083410 con 0x7f1e640b1500 2023-09-30T02:06:37.150 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:37.149+0000 7f1e59ffb700 10 client.4407.objecter ms_dispatch 0x7f1e64001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:06:37.150 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:37.149+0000 7f1e59ffb700 3 client.4407.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:06:37.151 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:37.149+0000 7f1e59ffb700 3 client.4407.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:06:37.151 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:37.149+0000 7f1e59ffb700 20 client.4407.objecter dump_active .. 0 homeless 2023-09-30T02:06:37.151 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:37.149+0000 7f1e59ffb700 1 -- 172.21.15.59:0/1239317004 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1e54083860 con 0x7f1e640b1500 2023-09-30T02:06:37.151 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:37.149+0000 7f1e59ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:37.151 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:37.149+0000 7f1e59ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:06:37.151 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:37.150+0000 7f1e69a9a700 1 --2- 172.21.15.59:0/1239317004 >> v2:172.21.15.110:6800/115870 conn(0x7f1e48068960 0x7f1e4806ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:37.151 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:37.150+0000 7f1e69a9a700 10 monclient: get_auth_request con 0x7f1e48068960 auth_method 0 2023-09-30T02:06:37.151 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:37.150+0000 7f1e59ffb700 10 client.4407.objecter ms_handle_connect 0x7f1e48068960 2023-09-30T02:06:37.151 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:37.150+0000 7f1e69a9a700 1 --2- 172.21.15.59:0/1239317004 >> v2:172.21.15.110:6800/115870 conn(0x7f1e48068960 0x7f1e4806ae10 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1e60001ad0 tx=0x7f1e60009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:37.401 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:37.401+0000 7f1e6bcfe700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:06:37.401 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:37.401+0000 7f1e6bcfe700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:06:37.401 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:37.401+0000 7f1e6bcfe700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:06:37.402 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:37.401+0000 7f1e6bcfe700 1 -- 172.21.15.59:0/1239317004 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f1e640b1920 con 0x7f1e640b1500 2023-09-30T02:06:37.414 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:37.413+0000 7f1e59ffb700 1 -- 172.21.15.59:0/1239317004 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f1e54017100 con 0x7f1e640b1500 2023-09-30T02:06:37.414 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:37.413+0000 7f1e59ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:06:37.414 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:37.413+0000 7f1e59ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:06:37.414 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:06:37.414 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:06:37.417 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:37.415+0000 7f1e6bcfe700 10 librados: watch_flush enter 2023-09-30T02:06:37.417 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:37.416+0000 7f1e6bcfe700 10 librados: watch_flush exit 2023-09-30T02:06:37.417 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:37.416+0000 7f1e6bcfe700 20 client.4407.objecter shutdown clearing up homeless session... 2023-09-30T02:06:37.417 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:37.416+0000 7f1e6bcfe700 10 client.4407.objecter successfully canceled tick 2023-09-30T02:06:37.417 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:37.416+0000 7f1e6bcfe700 1 -- 172.21.15.59:0/1239317004 >> v2:172.21.15.110:6800/115870 conn(0x7f1e48068960 msgr2=0x7f1e4806ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:37.417 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:37.416+0000 7f1e6bcfe700 1 --2- 172.21.15.59:0/1239317004 >> v2:172.21.15.110:6800/115870 conn(0x7f1e48068960 0x7f1e4806ae10 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1e60001ad0 tx=0x7f1e60009040 comp rx=0 tx=0).stop 2023-09-30T02:06:37.417 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:37.416+0000 7f1e6bcfe700 10 monclient: shutdown 2023-09-30T02:06:37.417 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:37.416+0000 7f1e6bcfe700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:37.417 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:37.416+0000 7f1e6bcfe700 1 -- 172.21.15.59:0/1239317004 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1e640b1500 msgr2=0x7f1e64151470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:37.418 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:37.416+0000 7f1e6bcfe700 1 --2- 172.21.15.59:0/1239317004 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1e640b1500 0x7f1e64151470 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1e5400c4c0 tx=0x7f1e54007ea0 comp rx=0 tx=0).stop 2023-09-30T02:06:37.418 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:37.416+0000 7f1e6bcfe700 1 -- 172.21.15.59:0/1239317004 shutdown_connections 2023-09-30T02:06:37.418 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:37.416+0000 7f1e6bcfe700 1 --2- 172.21.15.59:0/1239317004 >> v2:172.21.15.110:6800/115870 conn(0x7f1e48068960 0x7f1e4806ae10 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:37.418 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:37.416+0000 7f1e6bcfe700 1 --2- 172.21.15.59:0/1239317004 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1e640a5b70 0x7f1e641450b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:37.418 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:37.416+0000 7f1e6bcfe700 1 --2- 172.21.15.59:0/1239317004 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1e64142290 0x7f1e641519b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:37.418 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:37.416+0000 7f1e6bcfe700 1 --2- 172.21.15.59:0/1239317004 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1e640b1500 0x7f1e64151470 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:37.418 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:37.416+0000 7f1e6bcfe700 1 -- 172.21.15.59:0/1239317004 >> 172.21.15.59:0/1239317004 conn(0x7f1e640631f0 msgr2=0x7f1e64138190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:37.418 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:37.416+0000 7f1e6bcfe700 1 -- 172.21.15.59:0/1239317004 shutdown_connections 2023-09-30T02:06:37.418 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:37.416+0000 7f1e6bcfe700 1 -- 172.21.15.59:0/1239317004 wait complete. 2023-09-30T02:06:37.418 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:37.416+0000 7f1e6bcfe700 1 librados: shutdown 2023-09-30T02:06:40.431 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:06:40.576 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.576+0000 7f5b4f92c700 10 monclient: get_monmap_and_config 2023-09-30T02:06:40.576 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.576+0000 7f5b4f92c700 10 monclient: build_initial_monmap 2023-09-30T02:06:40.576 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.576+0000 7f5b4f92c700 10 monclient: monmap: 2023-09-30T02:06:40.577 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:40.577 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:40.577 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:06:40.577505+0000 2023-09-30T02:06:40.577 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:06:40.577505+0000 2023-09-30T02:06:40.577 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:40.577 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:40.577 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:06:40.577 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:06:40.577 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:06:40.578 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:06:40.578 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:06:40.578 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:40.580 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.576+0000 7f5b4f92c700 1 Processor -- start 2023-09-30T02:06:40.580 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.577+0000 7f5b4f92c700 1 -- start start 2023-09-30T02:06:40.580 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.577+0000 7f5b4f92c700 10 monclient: init 2023-09-30T02:06:40.580 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.577+0000 7f5b4f92c700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:40.580 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.577+0000 7f5b4f92c700 10 monclient: _add_conns ranks=[1,3,4,0,2] 2023-09-30T02:06:40.580 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.577+0000 7f5b4f92c700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5b480a5b70 addr v1:172.21.15.110:6789/0 2023-09-30T02:06:40.580 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.578+0000 7f5b4f92c700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f5b48142290 0x7f5b480b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:40.581 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.578+0000 7f5b4f92c700 10 monclient(hunting): picked mon.noname-a con 0x7f5b48142290 addr v2:172.21.15.59:3300/0 2023-09-30T02:06:40.581 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.578+0000 7f5b4f92c700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5b480a5f40 0x7f5b4813af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:40.581 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.578+0000 7f5b4f92c700 10 monclient(hunting): picked mon.noname-c con 0x7f5b480a5f40 addr v2:172.21.15.110:3300/0 2023-09-30T02:06:40.581 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.578+0000 7f5b4f92c700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b480a75a0 con 0x7f5b480a5b70 2023-09-30T02:06:40.581 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.578+0000 7f5b4f92c700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:40.581 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.578+0000 7f5b4f92c700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b480a6360 con 0x7f5b48142290 2023-09-30T02:06:40.581 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.578+0000 7f5b4f92c700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:40.581 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.578+0000 7f5b4f92c700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b480af3f0 con 0x7f5b480a5f40 2023-09-30T02:06:40.581 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.578+0000 7f5b4f92c700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:40.581 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.578+0000 7f5b4f92c700 10 monclient(hunting): authenticate will time out at 1845.881103s 2023-09-30T02:06:40.581 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.578+0000 7f5b4cec7700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f5b48142290 0x7f5b480b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:40.582 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.579+0000 7f5b4dec9700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5b480a5f40 0x7f5b4813af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:40.582 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.579+0000 7f5b4cec7700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f5b48142290 0x7f5b480b1500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51742/0 (socket says 172.21.15.59:51742) 2023-09-30T02:06:40.582 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.579+0000 7f5b4cec7700 1 -- 172.21.15.59:0/1494525510 learned_addr learned my addr 172.21.15.59:0/1494525510 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:06:40.582 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.579+0000 7f5b4cec7700 10 monclient(hunting): get_auth_request con 0x7f5b48142290 auth_method 0 2023-09-30T02:06:40.582 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.579+0000 7f5b4cec7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:40.582 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.579+0000 7f5b4cec7700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:40.582 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.579+0000 7f5b4cec7700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:40.582 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.579+0000 7f5b4dec9700 10 monclient(hunting): get_auth_request con 0x7f5b480a5f40 auth_method 0 2023-09-30T02:06:40.582 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.579+0000 7f5b4dec9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:40.583 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.579+0000 7f5b4dec9700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:40.583 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.579+0000 7f5b4dec9700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:40.583 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.579+0000 7f5b4cec7700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:40.583 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.579+0000 7f5b4cec7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:40.583 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.579+0000 7f5b4cec7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:40.584 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.580+0000 7f5b4dec9700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:40.584 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.580+0000 7f5b4dec9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:40.584 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.580+0000 7f5b4dec9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:40.584 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.582+0000 7f5b4cec7700 10 monclient(hunting): handle_auth_done global_id 4410 payload 274 2023-09-30T02:06:40.584 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.582+0000 7f5b4cec7700 1 -- 172.21.15.59:0/1494525510 >> v2:172.21.15.110:3300/0 conn(0x7f5b480a5f40 msgr2=0x7f5b4813af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:06:40.584 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.582+0000 7f5b4cec7700 1 --2- 172.21.15.59:0/1494525510 >> v2:172.21.15.110:3300/0 conn(0x7f5b480a5f40 0x7f5b4813af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:40.585 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.582+0000 7f5b4cec7700 1 -- 172.21.15.59:0/1494525510 >> v1:172.21.15.110:6789/0 conn(0x7f5b480a5b70 legacy=0x7f5b48141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:40.585 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.582+0000 7f5b4cec7700 10 monclient: _finish_hunting 0 2023-09-30T02:06:40.585 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.582+0000 7f5b4cec7700 1 monclient: found mon.noname-a 2023-09-30T02:06:40.585 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.582+0000 7f5b4cec7700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:40.585 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.582+0000 7f5b4cec7700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.59:3300/0 2023-09-30T02:06:40.585 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.582+0000 7f5b4cec7700 1 -- 172.21.15.59:0/1494525510 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b4813b470 con 0x7f5b48142290 2023-09-30T02:06:40.585 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.584+0000 7f5b4cec7700 1 --2- 172.21.15.59:0/1494525510 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5b48142290 0x7f5b480b1500 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5b480b0640 tx=0x7f5b38009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7be3f6c1f72c6167 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:40.589 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.588+0000 7f5b3ffff700 1 -- 172.21.15.59:0/1494525510 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b38009e70 con 0x7f5b48142290 2023-09-30T02:06:40.589 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.588+0000 7f5b3ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:40.590 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.588+0000 7f5b3ffff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T02:06:40.590 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.588+0000 7f5b3ffff700 10 monclient: dump: 2023-09-30T02:06:40.590 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:40.590 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:40.590 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:40.590 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:40.590 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:40.590 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:40.590 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:40.590 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:40.590 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:40.591 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:40.591 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.588+0000 7f5b3ffff700 10 monclient: _finish_auth 0 2023-09-30T02:06:40.591 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.588+0000 7f5b3ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:40.591 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.588+0000 7f5b4f92c700 5 monclient: authenticate success, global_id 4410 2023-09-30T02:06:40.591 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.588+0000 7f5b4f92c700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:06:40.591 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.588+0000 7f5b3ffff700 1 -- 172.21.15.59:0/1494525510 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b3800b860 con 0x7f5b48142290 2023-09-30T02:06:40.593 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.588+0000 7f5b3ffff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:40.593 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.588+0000 7f5b4f92c700 10 monclient: get_monmap_and_config success 2023-09-30T02:06:40.593 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.588+0000 7f5b3ffff700 1 -- 172.21.15.59:0/1494525510 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b3800bc00 con 0x7f5b48142290 2023-09-30T02:06:40.593 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.588+0000 7f5b4f92c700 4 set_mon_vals no callback set 2023-09-30T02:06:40.593 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.588+0000 7f5b3ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:40.593 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.588+0000 7f5b3ffff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T02:06:40.593 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.588+0000 7f5b3ffff700 10 monclient: dump: 2023-09-30T02:06:40.593 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:40.594 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:40.594 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:40.594 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:40.594 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:40.594 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:40.594 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:40.594 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:40.594 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:40.594 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:40.594 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.588+0000 7f5b4f92c700 10 monclient: shutdown 2023-09-30T02:06:40.595 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.588+0000 7f5b4f92c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:40.595 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.588+0000 7f5b4f92c700 1 -- 172.21.15.59:0/1494525510 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5b48142290 msgr2=0x7f5b480b1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:40.595 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.588+0000 7f5b4f92c700 1 --2- 172.21.15.59:0/1494525510 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5b48142290 0x7f5b480b1500 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5b480b0640 tx=0x7f5b38009250 comp rx=0 tx=0).stop 2023-09-30T02:06:40.595 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.588+0000 7f5b4f92c700 1 -- 172.21.15.59:0/1494525510 shutdown_connections 2023-09-30T02:06:40.595 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.588+0000 7f5b4f92c700 1 --2- 172.21.15.59:0/1494525510 >> v2:172.21.15.110:3300/0 conn(0x7f5b480a5f40 0x7f5b4813af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:40.595 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.588+0000 7f5b4f92c700 1 --2- 172.21.15.59:0/1494525510 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5b48142290 0x7f5b480b1500 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:40.595 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.588+0000 7f5b4f92c700 1 -- 172.21.15.59:0/1494525510 >> 172.21.15.59:0/1494525510 conn(0x7f5b480631f0 msgr2=0x7f5b480635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:40.595 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.589+0000 7f5b4f92c700 1 -- 172.21.15.59:0/1494525510 shutdown_connections 2023-09-30T02:06:40.595 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.589+0000 7f5b4f92c700 1 -- 172.21.15.59:0/1494525510 wait complete. 2023-09-30T02:06:40.600 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.589+0000 7f5b4f92c700 10 monclient: build_initial_monmap 2023-09-30T02:06:40.600 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.589+0000 7f5b4f92c700 10 monclient: monmap: 2023-09-30T02:06:40.600 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:40.600 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:06:40.600 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:06:40.600 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:06:40.600 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:40.601 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:06:40.601 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:06:40.601 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:06:40.601 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:06:40.601 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:40.601 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.590+0000 7f5b4f92c700 1 librados: starting msgr at 2023-09-30T02:06:40.601 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.590+0000 7f5b4f92c700 1 librados: starting objecter 2023-09-30T02:06:40.601 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.590+0000 7f5b4f92c700 1 Processor -- start 2023-09-30T02:06:40.601 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.590+0000 7f5b4f92c700 1 -- start start 2023-09-30T02:06:40.602 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.590+0000 7f5b4f92c700 1 librados: setting wanted keys 2023-09-30T02:06:40.602 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.590+0000 7f5b4f92c700 1 librados: calling monclient init 2023-09-30T02:06:40.602 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.590+0000 7f5b4f92c700 10 monclient: init 2023-09-30T02:06:40.602 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.591+0000 7f5b4f92c700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:40.602 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.591+0000 7f5b4f92c700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T02:06:40.602 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.591+0000 7f5b4f92c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b480a5b70 0x7f5b4813fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:40.602 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.591+0000 7f5b4f92c700 10 monclient(hunting): picked mon.noname-b con 0x7f5b480a5b70 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:06:40.602 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.591+0000 7f5b4f92c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5b480a5f40 0x7f5b4814e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:40.603 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.591+0000 7f5b4f92c700 10 monclient(hunting): picked mon.noname-a con 0x7f5b480a5f40 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:06:40.603 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.591+0000 7f5b4f92c700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5b48142290 0x7f5b4813ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:40.603 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.591+0000 7f5b4f92c700 10 monclient(hunting): picked mon.noname-c con 0x7f5b48142290 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:40.603 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.591+0000 7f5b4f92c700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:40.603 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.591+0000 7f5b4f92c700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b4813d3d0 con 0x7f5b480a5f40 2023-09-30T02:06:40.603 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.591+0000 7f5b4f92c700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:40.603 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.591+0000 7f5b4f92c700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b4813d680 con 0x7f5b480a5b70 2023-09-30T02:06:40.603 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.591+0000 7f5b4f92c700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:40.603 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.591+0000 7f5b4f92c700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b4814eca0 con 0x7f5b48142290 2023-09-30T02:06:40.603 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.591+0000 7f5b4f92c700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:40.604 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.591+0000 7f5b4f92c700 10 monclient(hunting): authenticate will time out at 1845.894670s 2023-09-30T02:06:40.604 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.592+0000 7f5b4cec7700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5b480a5f40 0x7f5b4814e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:40.604 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.592+0000 7f5b4dec9700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5b48142290 0x7f5b4813ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:40.604 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.592+0000 7f5b4d6c8700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b480a5b70 0x7f5b4813fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:40.604 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.592+0000 7f5b4cec7700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5b480a5f40 0x7f5b4814e200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51756/0 (socket says 172.21.15.59:51756) 2023-09-30T02:06:40.604 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.592+0000 7f5b4cec7700 1 -- 172.21.15.59:0/382609690 learned_addr learned my addr 172.21.15.59:0/382609690 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:06:40.604 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.592+0000 7f5b4cec7700 10 monclient(hunting): get_auth_request con 0x7f5b480a5f40 auth_method 0 2023-09-30T02:06:40.604 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.592+0000 7f5b4cec7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:40.604 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.592+0000 7f5b4cec7700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:40.604 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.592+0000 7f5b4cec7700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:40.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.592+0000 7f5b4d6c8700 10 monclient(hunting): get_auth_request con 0x7f5b480a5b70 auth_method 0 2023-09-30T02:06:40.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.592+0000 7f5b4d6c8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:40.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.592+0000 7f5b4d6c8700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:40.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.592+0000 7f5b4d6c8700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:40.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.592+0000 7f5b4dec9700 10 monclient(hunting): get_auth_request con 0x7f5b48142290 auth_method 0 2023-09-30T02:06:40.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.592+0000 7f5b4dec9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:40.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.592+0000 7f5b4dec9700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:40.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.592+0000 7f5b4dec9700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:40.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.593+0000 7f5b4cec7700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:40.606 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.593+0000 7f5b4cec7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:40.606 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.593+0000 7f5b4cec7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:40.606 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.593+0000 7f5b4d6c8700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:40.606 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.593+0000 7f5b4d6c8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:40.606 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.593+0000 7f5b4d6c8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:40.606 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.593+0000 7f5b4dec9700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:40.606 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.593+0000 7f5b4dec9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:40.606 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.593+0000 7f5b4dec9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:40.606 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.596+0000 7f5b4d6c8700 10 monclient(hunting): handle_auth_done global_id 4429 payload 824 2023-09-30T02:06:40.606 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.596+0000 7f5b4d6c8700 1 -- 172.21.15.59:0/382609690 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5b48142290 msgr2=0x7f5b4813ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:40.607 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.596+0000 7f5b4d6c8700 1 --2- 172.21.15.59:0/382609690 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5b48142290 0x7f5b4813ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:40.607 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.596+0000 7f5b4d6c8700 1 -- 172.21.15.59:0/382609690 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5b480a5f40 msgr2=0x7f5b4814e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:40.607 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.596+0000 7f5b4d6c8700 1 --2- 172.21.15.59:0/382609690 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5b480a5f40 0x7f5b4814e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:40.607 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.596+0000 7f5b4d6c8700 10 monclient: _finish_hunting 0 2023-09-30T02:06:40.607 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.596+0000 7f5b4d6c8700 1 monclient: found mon.noname-b 2023-09-30T02:06:40.607 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.596+0000 7f5b4d6c8700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:40.607 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.596+0000 7f5b4d6c8700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:06:40.608 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.596+0000 7f5b4d6c8700 1 -- 172.21.15.59:0/382609690 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b380094f0 con 0x7f5b480a5b70 2023-09-30T02:06:40.608 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.599+0000 7f5b4d6c8700 1 --2- 172.21.15.59:0/382609690 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b480a5b70 0x7f5b4813fb70 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f5b4400c590 tx=0x7f5b44007ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:40.608 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.599+0000 7f5b3d7fa700 10 client.?.objecter ms_handle_connect 0x7f5b480a5b70 2023-09-30T02:06:40.608 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.599+0000 7f5b3d7fa700 10 client.?.objecter resend_mon_ops 2023-09-30T02:06:40.610 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.601+0000 7f5b3d7fa700 1 -- 172.21.15.59:0/382609690 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b44011590 con 0x7f5b480a5b70 2023-09-30T02:06:40.610 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.601+0000 7f5b3d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:40.610 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.601+0000 7f5b3d7fa700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:06:40.610 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.601+0000 7f5b3d7fa700 10 monclient: dump: 2023-09-30T02:06:40.610 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:40.611 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:40.611 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:40.611 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:40.611 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:40.611 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:40.611 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:40.611 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:40.611 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:40.611 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:40.612 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.601+0000 7f5b3d7fa700 10 monclient: _finish_auth 0 2023-09-30T02:06:40.612 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.601+0000 7f5b3d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:40.612 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.601+0000 7f5b4f92c700 5 monclient: authenticate success, global_id 4429 2023-09-30T02:06:40.612 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.601+0000 7f5b4f92c700 10 monclient: _renew_subs 2023-09-30T02:06:40.612 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.601+0000 7f5b4f92c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:40.612 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.601+0000 7f5b4f92c700 1 -- 172.21.15.59:0/382609690 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b4813dba0 con 0x7f5b480a5b70 2023-09-30T02:06:40.612 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.601+0000 7f5b4f92c700 10 client.4429.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:06:40.612 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.601+0000 7f5b4f92c700 10 monclient: _renew_subs 2023-09-30T02:06:40.612 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.601+0000 7f5b4f92c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:40.613 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.601+0000 7f5b4f92c700 1 -- 172.21.15.59:0/382609690 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b4813e0e0 con 0x7f5b480a5b70 2023-09-30T02:06:40.613 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.601+0000 7f5b4f92c700 1 librados: init done 2023-09-30T02:06:40.613 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.602+0000 7f5b4f92c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:40.613 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.602+0000 7f5b4f92c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:40.613 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.602+0000 7f5b4f92c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:40.613 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.602+0000 7f5b4f92c700 1 -- 172.21.15.59:0/382609690 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b4813d8c0 con 0x7f5b480a5b70 2023-09-30T02:06:40.613 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.603+0000 7f5b3d7fa700 1 -- 172.21.15.59:0/382609690 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b44001120 con 0x7f5b480a5b70 2023-09-30T02:06:40.613 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.603+0000 7f5b3d7fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:40.613 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.603+0000 7f5b3d7fa700 1 -- 172.21.15.59:0/382609690 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b44011900 con 0x7f5b480a5b70 2023-09-30T02:06:40.613 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.603+0000 7f5b3d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:40.613 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.603+0000 7f5b3d7fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:06:40.614 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.603+0000 7f5b3d7fa700 10 monclient: dump: 2023-09-30T02:06:40.614 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:40.614 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:40.614 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:40.614 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:40.614 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:40.614 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:40.614 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:40.615 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:40.615 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:40.615 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:40.615 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.603+0000 7f5b3f7fe700 4 set_mon_vals no callback set 2023-09-30T02:06:40.848 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.848+0000 7f5b3d7fa700 1 -- 172.21.15.59:0/382609690 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f5b44011a60 con 0x7f5b480a5b70 2023-09-30T02:06:40.849 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.848+0000 7f5b3d7fa700 1 --2- 172.21.15.59:0/382609690 >> v2:172.21.15.110:6800/115870 conn(0x7f5b2c068890 0x7f5b2c06ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:40.849 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.849+0000 7f5b4cec7700 1 --2- 172.21.15.59:0/382609690 >> v2:172.21.15.110:6800/115870 conn(0x7f5b2c068890 0x7f5b2c06ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:40.849 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.849+0000 7f5b4cec7700 10 monclient: get_auth_request con 0x7f5b2c068890 auth_method 0 2023-09-30T02:06:40.850 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.849+0000 7f5b4cec7700 1 --2- 172.21.15.59:0/382609690 >> v2:172.21.15.110:6800/115870 conn(0x7f5b2c068890 0x7f5b2c06ad40 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5b4813eb80 tx=0x7f5b38017000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:40.850 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.849+0000 7f5b3d7fa700 10 client.4429.objecter ms_handle_connect 0x7f5b2c068890 2023-09-30T02:06:40.852 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.853+0000 7f5b3d7fa700 1 -- 172.21.15.59:0/382609690 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f5b44054c60 con 0x7f5b480a5b70 2023-09-30T02:06:40.853 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.853+0000 7f5b3d7fa700 10 client.4429.objecter ms_dispatch 0x7f5b48001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:06:40.853 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.853+0000 7f5b3d7fa700 3 client.4429.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:06:40.853 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.853+0000 7f5b3d7fa700 3 client.4429.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:06:40.853 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.853+0000 7f5b3d7fa700 20 client.4429.objecter dump_active .. 0 homeless 2023-09-30T02:06:40.862 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.862+0000 7f5b3d7fa700 1 -- 172.21.15.59:0/382609690 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5b44050fa0 con 0x7f5b480a5b70 2023-09-30T02:06:40.862 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.862+0000 7f5b3d7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:40.862 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:40.862+0000 7f5b3d7fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:06:41.118 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:41.118+0000 7f5b4f92c700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:06:41.118 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:41.118+0000 7f5b4f92c700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:06:41.118 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:41.118+0000 7f5b4f92c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:41.118 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:41.118+0000 7f5b4f92c700 1 -- 172.21.15.59:0/382609690 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f5b4813d8c0 con 0x7f5b480a5b70 2023-09-30T02:06:41.172 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:41.171+0000 7f5b3d7fa700 1 -- 172.21.15.59:0/382609690 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f5b44021100 con 0x7f5b480a5b70 2023-09-30T02:06:41.172 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:41.171+0000 7f5b3d7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:06:41.172 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:41.171+0000 7f5b3d7fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:06:41.172 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:06:41.172 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:06:41.175 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:41.174+0000 7f5b4f92c700 10 librados: watch_flush enter 2023-09-30T02:06:41.175 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:41.174+0000 7f5b4f92c700 10 librados: watch_flush exit 2023-09-30T02:06:41.176 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:41.174+0000 7f5b4f92c700 20 client.4429.objecter shutdown clearing up homeless session... 2023-09-30T02:06:41.176 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:41.174+0000 7f5b4f92c700 10 client.4429.objecter successfully canceled tick 2023-09-30T02:06:41.176 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:41.174+0000 7f5b4f92c700 1 -- 172.21.15.59:0/382609690 >> v2:172.21.15.110:6800/115870 conn(0x7f5b2c068890 msgr2=0x7f5b2c06ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:41.176 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:41.174+0000 7f5b4f92c700 1 --2- 172.21.15.59:0/382609690 >> v2:172.21.15.110:6800/115870 conn(0x7f5b2c068890 0x7f5b2c06ad40 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5b4813eb80 tx=0x7f5b38017000 comp rx=0 tx=0).stop 2023-09-30T02:06:41.176 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:41.174+0000 7f5b4f92c700 10 monclient: shutdown 2023-09-30T02:06:41.176 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:41.174+0000 7f5b4f92c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:41.176 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:41.174+0000 7f5b4f92c700 1 -- 172.21.15.59:0/382609690 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b480a5b70 msgr2=0x7f5b4813fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:41.176 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:41.174+0000 7f5b4f92c700 1 --2- 172.21.15.59:0/382609690 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b480a5b70 0x7f5b4813fb70 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f5b4400c590 tx=0x7f5b44007ec0 comp rx=0 tx=0).stop 2023-09-30T02:06:41.176 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:41.174+0000 7f5b4f92c700 1 -- 172.21.15.59:0/382609690 shutdown_connections 2023-09-30T02:06:41.177 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:41.174+0000 7f5b4f92c700 1 --2- 172.21.15.59:0/382609690 >> v2:172.21.15.110:6800/115870 conn(0x7f5b2c068890 0x7f5b2c06ad40 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:41.177 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:41.174+0000 7f5b4f92c700 1 --2- 172.21.15.59:0/382609690 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5b480a5b70 0x7f5b4813fb70 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:41.177 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:41.174+0000 7f5b4f92c700 1 --2- 172.21.15.59:0/382609690 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5b48142290 0x7f5b4813ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:41.177 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:41.174+0000 7f5b4f92c700 1 --2- 172.21.15.59:0/382609690 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5b480a5f40 0x7f5b4814e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:41.177 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:41.174+0000 7f5b4f92c700 1 -- 172.21.15.59:0/382609690 >> 172.21.15.59:0/382609690 conn(0x7f5b480631f0 msgr2=0x7f5b481383d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:41.177 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:41.175+0000 7f5b4f92c700 1 -- 172.21.15.59:0/382609690 shutdown_connections 2023-09-30T02:06:41.177 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:41.175+0000 7f5b4f92c700 1 -- 172.21.15.59:0/382609690 wait complete. 2023-09-30T02:06:41.177 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:41.175+0000 7f5b4f92c700 1 librados: shutdown 2023-09-30T02:06:44.190 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:06:44.333 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.332+0000 7fc02a697700 10 monclient: get_monmap_and_config 2023-09-30T02:06:44.333 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.332+0000 7fc02a697700 10 monclient: build_initial_monmap 2023-09-30T02:06:44.333 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.332+0000 7fc02a697700 10 monclient: monmap: 2023-09-30T02:06:44.333 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:44.333 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:44.333 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:06:44.334013+0000 2023-09-30T02:06:44.334 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:06:44.334013+0000 2023-09-30T02:06:44.334 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:44.334 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:44.334 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:06:44.334 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:06:44.334 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:06:44.334 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:06:44.334 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:06:44.335 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:44.336 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.333+0000 7fc02a697700 1 Processor -- start 2023-09-30T02:06:44.336 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.333+0000 7fc02a697700 1 -- start start 2023-09-30T02:06:44.337 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.333+0000 7fc02a697700 10 monclient: init 2023-09-30T02:06:44.337 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.334+0000 7fc02a697700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:44.337 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.334+0000 7fc02a697700 10 monclient: _add_conns ranks=[2,3,1,4,0] 2023-09-30T02:06:44.337 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.334+0000 7fc02a697700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc0240a5b70 0x7fc0240b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:44.337 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.334+0000 7fc02a697700 10 monclient(hunting): picked mon.noname-b con 0x7fc0240a5b70 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:44.337 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.334+0000 7fc02a697700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fc0240a5f40 0x7fc024141b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:44.337 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.334+0000 7fc02a697700 10 monclient(hunting): picked mon.noname-a con 0x7fc0240a5f40 addr v2:172.21.15.59:3300/0 2023-09-30T02:06:44.337 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.334+0000 7fc02a697700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc0241420c0 addr v1:172.21.15.110:6789/0 2023-09-30T02:06:44.338 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.334+0000 7fc02a697700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0240a63b0 con 0x7fc0241420c0 2023-09-30T02:06:44.338 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.334+0000 7fc02a697700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:44.338 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.334+0000 7fc02a697700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0240a75a0 con 0x7fc0240a5f40 2023-09-30T02:06:44.338 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.334+0000 7fc02a697700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:44.338 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.334+0000 7fc02a697700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0240af3f0 con 0x7fc0240a5b70 2023-09-30T02:06:44.338 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.334+0000 7fc02a697700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:44.338 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.334+0000 7fc02a697700 10 monclient(hunting): authenticate will time out at 1849.637203s 2023-09-30T02:06:44.338 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.334+0000 7fc023fff700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc0240a5b70 0x7fc0240b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:44.339 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.335+0000 7fc028c34700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fc0241420c0 0x7fc02413e230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.59:60484/0 (socket says 172.21.15.59:60484) 2023-09-30T02:06:44.339 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.335+0000 7fc028c34700 1 -- 172.21.15.59:0/1390399583 learned_addr learned my addr 172.21.15.59:0/1390399583 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:06:44.339 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.335+0000 7fc023fff700 10 monclient(hunting): get_auth_request con 0x7fc0240a5b70 auth_method 0 2023-09-30T02:06:44.339 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.335+0000 7fc023fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:44.339 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.335+0000 7fc023fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:44.339 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.335+0000 7fc023fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:44.339 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.336+0000 7fc023fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:44.339 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.336+0000 7fc023fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:44.339 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.336+0000 7fc023fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:44.340 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.339+0000 7fc023fff700 10 monclient(hunting): handle_auth_done global_id 4370 payload 274 2023-09-30T02:06:44.340 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.339+0000 7fc023fff700 1 -- 172.21.15.59:0/1390399583 >> v2:172.21.15.59:3300/0 conn(0x7fc0240a5f40 msgr2=0x7fc024141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:06:44.340 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.339+0000 7fc023fff700 1 --2- 172.21.15.59:0/1390399583 >> v2:172.21.15.59:3300/0 conn(0x7fc0240a5f40 0x7fc024141b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:44.341 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.339+0000 7fc023fff700 1 -- 172.21.15.59:0/1390399583 >> v1:172.21.15.110:6789/0 conn(0x7fc0241420c0 legacy=0x7fc02413e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:44.341 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.339+0000 7fc023fff700 10 monclient: _finish_hunting 0 2023-09-30T02:06:44.341 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.339+0000 7fc023fff700 1 monclient: found mon.noname-b 2023-09-30T02:06:44.341 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.339+0000 7fc023fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:44.341 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.339+0000 7fc023fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.59:3301/0 2023-09-30T02:06:44.341 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.339+0000 7fc023fff700 1 -- 172.21.15.59:0/1390399583 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc024142500 con 0x7fc0240a5b70 2023-09-30T02:06:44.341 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.341+0000 7fc023fff700 1 --2- 172.21.15.59:0/1390399583 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc0240a5b70 0x7fc0240b1500 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc0140098e0 tx=0x7fc0140091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c6b424b19ee2b21 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:44.347 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.345+0000 7fc022ffd700 1 -- 172.21.15.59:0/1390399583 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc014009e80 con 0x7fc0240a5b70 2023-09-30T02:06:44.347 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.346+0000 7fc022ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:44.347 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.346+0000 7fc022ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:06:44.347 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.346+0000 7fc022ffd700 10 monclient: dump: 2023-09-30T02:06:44.348 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:44.348 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:44.348 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:44.348 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:44.348 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:44.349 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:44.349 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:44.349 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:44.349 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:44.349 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:44.349 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.346+0000 7fc022ffd700 10 monclient: _finish_auth 0 2023-09-30T02:06:44.350 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.346+0000 7fc022ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:44.350 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.346+0000 7fc022ffd700 1 -- 172.21.15.59:0/1390399583 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc01400b860 con 0x7fc0240a5b70 2023-09-30T02:06:44.350 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.346+0000 7fc02a697700 5 monclient: authenticate success, global_id 4370 2023-09-30T02:06:44.350 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.346+0000 7fc02a697700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:06:44.350 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.346+0000 7fc022ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:44.351 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.346+0000 7fc02a697700 10 monclient: get_monmap_and_config success 2023-09-30T02:06:44.351 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.346+0000 7fc022ffd700 1 -- 172.21.15.59:0/1390399583 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc01400bbe0 con 0x7fc0240a5b70 2023-09-30T02:06:44.351 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.346+0000 7fc02a697700 4 set_mon_vals no callback set 2023-09-30T02:06:44.351 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.346+0000 7fc022ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:44.351 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.346+0000 7fc022ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:06:44.351 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.346+0000 7fc022ffd700 10 monclient: dump: 2023-09-30T02:06:44.351 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:44.352 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:44.352 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:44.352 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:44.352 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:44.353 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:44.353 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:44.353 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:44.353 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:44.353 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:44.353 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.346+0000 7fc02a697700 10 monclient: shutdown 2023-09-30T02:06:44.354 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.346+0000 7fc02a697700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:44.354 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.346+0000 7fc02a697700 1 -- 172.21.15.59:0/1390399583 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc0240a5b70 msgr2=0x7fc0240b1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:44.354 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.346+0000 7fc02a697700 1 --2- 172.21.15.59:0/1390399583 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc0240a5b70 0x7fc0240b1500 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc0140098e0 tx=0x7fc0140091a0 comp rx=0 tx=0).stop 2023-09-30T02:06:44.354 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.346+0000 7fc02a697700 1 -- 172.21.15.59:0/1390399583 shutdown_connections 2023-09-30T02:06:44.354 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.346+0000 7fc02a697700 1 --2- 172.21.15.59:0/1390399583 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc0240a5b70 0x7fc0240b1500 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:44.355 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.346+0000 7fc02a697700 1 --2- 172.21.15.59:0/1390399583 >> v2:172.21.15.59:3300/0 conn(0x7fc0240a5f40 0x7fc024141b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:44.355 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.346+0000 7fc02a697700 1 -- 172.21.15.59:0/1390399583 >> 172.21.15.59:0/1390399583 conn(0x7fc0240631f0 msgr2=0x7fc0240635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:44.355 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.346+0000 7fc02a697700 1 -- 172.21.15.59:0/1390399583 shutdown_connections 2023-09-30T02:06:44.355 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.347+0000 7fc02a697700 1 -- 172.21.15.59:0/1390399583 wait complete. 2023-09-30T02:06:44.359 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.347+0000 7fc02a697700 10 monclient: build_initial_monmap 2023-09-30T02:06:44.359 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.347+0000 7fc02a697700 10 monclient: monmap: 2023-09-30T02:06:44.359 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:44.359 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:06:44.359 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:06:44.359 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:06:44.359 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:44.359 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:06:44.359 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:06:44.360 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:06:44.360 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:06:44.360 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:44.360 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.347+0000 7fc02a697700 1 librados: starting msgr at 2023-09-30T02:06:44.360 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.347+0000 7fc02a697700 1 librados: starting objecter 2023-09-30T02:06:44.360 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.348+0000 7fc02a697700 1 Processor -- start 2023-09-30T02:06:44.360 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.348+0000 7fc02a697700 1 -- start start 2023-09-30T02:06:44.360 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.348+0000 7fc02a697700 1 librados: setting wanted keys 2023-09-30T02:06:44.360 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.348+0000 7fc02a697700 1 librados: calling monclient init 2023-09-30T02:06:44.361 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.348+0000 7fc02a697700 10 monclient: init 2023-09-30T02:06:44.361 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.348+0000 7fc02a697700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:44.361 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.348+0000 7fc02a697700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T02:06:44.361 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.348+0000 7fc02a697700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0240a5b70 0x7fc0240b5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:44.361 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.348+0000 7fc02a697700 10 monclient(hunting): picked mon.noname-a con 0x7fc0240a5b70 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:06:44.361 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.348+0000 7fc02a697700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0240a5f40 0x7fc02414dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:44.361 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.348+0000 7fc02a697700 10 monclient(hunting): picked mon.noname-b con 0x7fc0240a5f40 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:06:44.361 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.348+0000 7fc02a697700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc0241420c0 0x7fc02414e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:44.362 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.348+0000 7fc02a697700 10 monclient(hunting): picked mon.noname-c con 0x7fc0241420c0 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:44.362 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.348+0000 7fc02a697700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:44.362 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.348+0000 7fc02a697700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0240aa830 con 0x7fc0240a5b70 2023-09-30T02:06:44.362 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.348+0000 7fc02a697700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:44.362 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.348+0000 7fc02a697700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0240aec50 con 0x7fc0240a5f40 2023-09-30T02:06:44.362 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.348+0000 7fc02a697700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:44.362 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.348+0000 7fc02a697700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0240a8c50 con 0x7fc0241420c0 2023-09-30T02:06:44.362 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.348+0000 7fc02a697700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:44.362 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.348+0000 7fc02a697700 10 monclient(hunting): authenticate will time out at 1849.651993s 2023-09-30T02:06:44.363 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.349+0000 7fc023fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0240a5b70 0x7fc0240b5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:44.363 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.351+0000 7fc023fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0240a5b70 0x7fc0240b5300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:51776/0 (socket says 172.21.15.59:51776) 2023-09-30T02:06:44.363 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.351+0000 7fc023fff700 1 -- 172.21.15.59:0/2870115393 learned_addr learned my addr 172.21.15.59:0/2870115393 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:06:44.363 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.351+0000 7fc023fff700 10 monclient(hunting): get_auth_request con 0x7fc0240a5b70 auth_method 0 2023-09-30T02:06:44.363 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.351+0000 7fc023fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:44.363 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.351+0000 7fc023fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:44.363 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.351+0000 7fc023fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:44.363 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.352+0000 7fc023fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:44.363 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.352+0000 7fc023fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:44.363 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.352+0000 7fc023fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:44.364 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.356+0000 7fc023fff700 10 monclient(hunting): handle_auth_done global_id 4416 payload 824 2023-09-30T02:06:44.364 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.357+0000 7fc023fff700 1 -- 172.21.15.59:0/2870115393 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc0241420c0 msgr2=0x7fc02414e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:44.364 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.357+0000 7fc023fff700 1 --2- 172.21.15.59:0/2870115393 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc0241420c0 0x7fc02414e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:44.364 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.357+0000 7fc023fff700 1 -- 172.21.15.59:0/2870115393 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0240a5f40 msgr2=0x7fc02414dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:44.364 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.357+0000 7fc023fff700 1 --2- 172.21.15.59:0/2870115393 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0240a5f40 0x7fc02414dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:44.364 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.357+0000 7fc023fff700 10 monclient: _finish_hunting 0 2023-09-30T02:06:44.364 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.357+0000 7fc023fff700 1 monclient: found mon.noname-a 2023-09-30T02:06:44.364 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.357+0000 7fc023fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:44.364 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.357+0000 7fc023fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.59:3300/0 2023-09-30T02:06:44.364 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.357+0000 7fc023fff700 1 -- 172.21.15.59:0/2870115393 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc014009420 con 0x7fc0240a5b70 2023-09-30T02:06:44.366 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.360+0000 7fc023fff700 1 --2- 172.21.15.59:0/2870115393 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0240a5b70 0x7fc0240b5300 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc01400b3c0 tx=0x7fc014017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:44.366 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.360+0000 7fc00bfff700 10 client.?.objecter ms_handle_connect 0x7fc0240a5b70 2023-09-30T02:06:44.367 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.360+0000 7fc00bfff700 10 client.?.objecter resend_mon_ops 2023-09-30T02:06:44.367 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.365+0000 7fc00bfff700 1 -- 172.21.15.59:0/2870115393 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc014011420 con 0x7fc0240a5b70 2023-09-30T02:06:44.367 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.365+0000 7fc00bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:44.367 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.365+0000 7fc00bfff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T02:06:44.367 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.365+0000 7fc00bfff700 10 monclient: dump: 2023-09-30T02:06:44.367 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:44.367 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:44.367 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:44.367 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:44.367 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:44.368 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:44.368 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:44.368 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:44.368 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:44.368 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:44.368 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.365+0000 7fc00bfff700 10 monclient: _finish_auth 0 2023-09-30T02:06:44.368 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.365+0000 7fc00bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:44.368 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.365+0000 7fc00bfff700 1 -- 172.21.15.59:0/2870115393 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc014011580 con 0x7fc0240a5b70 2023-09-30T02:06:44.368 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.365+0000 7fc02a697700 5 monclient: authenticate success, global_id 4416 2023-09-30T02:06:44.368 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.365+0000 7fc02a697700 10 monclient: _renew_subs 2023-09-30T02:06:44.369 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.365+0000 7fc02a697700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:06:44.369 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.365+0000 7fc02a697700 1 -- 172.21.15.59:0/2870115393 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0240aa350 con 0x7fc0240a5b70 2023-09-30T02:06:44.369 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.365+0000 7fc00bfff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:44.369 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.365+0000 7fc00bfff700 1 -- 172.21.15.59:0/2870115393 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc014011940 con 0x7fc0240a5b70 2023-09-30T02:06:44.369 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.365+0000 7fc00bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:44.369 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.365+0000 7fc00bfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T02:06:44.369 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.365+0000 7fc00bfff700 10 monclient: dump: 2023-09-30T02:06:44.369 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:44.369 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:44.369 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:44.370 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:44.370 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:44.370 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:44.370 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:44.370 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:44.370 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:44.370 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:44.370 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.365+0000 7fc0227fc700 4 set_mon_vals no callback set 2023-09-30T02:06:44.370 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.365+0000 7fc02a697700 10 client.4416.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:06:44.370 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.365+0000 7fc02a697700 10 monclient: _renew_subs 2023-09-30T02:06:44.371 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.365+0000 7fc02a697700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:06:44.371 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.365+0000 7fc02a697700 1 -- 172.21.15.59:0/2870115393 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0240aaf00 con 0x7fc0240a5b70 2023-09-30T02:06:44.371 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.365+0000 7fc02a697700 1 librados: init done 2023-09-30T02:06:44.371 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.366+0000 7fc02a697700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:44.371 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.366+0000 7fc02a697700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:44.371 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.366+0000 7fc02a697700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:06:44.371 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.366+0000 7fc02a697700 1 -- 172.21.15.59:0/2870115393 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc024141c80 con 0x7fc0240a5b70 2023-09-30T02:06:44.437 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.436+0000 7fc00bfff700 1 -- 172.21.15.59:0/2870115393 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fc01402c3b0 con 0x7fc0240a5b70 2023-09-30T02:06:44.437 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.436+0000 7fc00bfff700 1 --2- 172.21.15.59:0/2870115393 >> v2:172.21.15.110:6800/115870 conn(0x7fc010068960 0x7fc01006ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:44.437 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.436+0000 7fc00bfff700 1 -- 172.21.15.59:0/2870115393 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fc014085a10 con 0x7fc0240a5b70 2023-09-30T02:06:44.438 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.436+0000 7fc00bfff700 10 client.4416.objecter ms_dispatch 0x7fc024001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:06:44.438 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.436+0000 7fc00bfff700 3 client.4416.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:06:44.438 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.436+0000 7fc00bfff700 3 client.4416.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:06:44.438 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.436+0000 7fc00bfff700 20 client.4416.objecter dump_active .. 0 homeless 2023-09-30T02:06:44.438 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.437+0000 7fc0237fe700 1 --2- 172.21.15.59:0/2870115393 >> v2:172.21.15.110:6800/115870 conn(0x7fc010068960 0x7fc01006ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:44.438 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.437+0000 7fc0237fe700 10 monclient: get_auth_request con 0x7fc010068960 auth_method 0 2023-09-30T02:06:44.439 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.437+0000 7fc0237fe700 1 --2- 172.21.15.59:0/2870115393 >> v2:172.21.15.110:6800/115870 conn(0x7fc010068960 0x7fc01006ae10 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc018006fd0 tx=0x7fc018006db0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:44.439 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.437+0000 7fc00bfff700 10 client.4416.objecter ms_handle_connect 0x7fc010068960 2023-09-30T02:06:44.689 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.688+0000 7fc00bfff700 1 -- 172.21.15.59:0/2870115393 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc014053330 con 0x7fc0240a5b70 2023-09-30T02:06:44.689 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.688+0000 7fc00bfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:44.689 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.688+0000 7fc00bfff700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:06:44.959 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.959+0000 7fc02a697700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:06:44.959 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.959+0000 7fc02a697700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:06:44.959 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.959+0000 7fc02a697700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:06:44.959 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.959+0000 7fc02a697700 1 -- 172.21.15.59:0/2870115393 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fc024064a00 con 0x7fc0240a5b70 2023-09-30T02:06:44.964 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.964+0000 7fc00bfff700 1 -- 172.21.15.59:0/2870115393 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fc014056950 con 0x7fc0240a5b70 2023-09-30T02:06:44.964 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.964+0000 7fc00bfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:06:44.964 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.964+0000 7fc00bfff700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:06:44.964 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:06:44.964 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:06:44.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.966+0000 7fc02a697700 10 librados: watch_flush enter 2023-09-30T02:06:44.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.966+0000 7fc02a697700 10 librados: watch_flush exit 2023-09-30T02:06:44.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.966+0000 7fc02a697700 20 client.4416.objecter shutdown clearing up homeless session... 2023-09-30T02:06:44.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.966+0000 7fc02a697700 10 client.4416.objecter successfully canceled tick 2023-09-30T02:06:44.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.966+0000 7fc02a697700 1 -- 172.21.15.59:0/2870115393 >> v2:172.21.15.110:6800/115870 conn(0x7fc010068960 msgr2=0x7fc01006ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:44.968 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.966+0000 7fc02a697700 1 --2- 172.21.15.59:0/2870115393 >> v2:172.21.15.110:6800/115870 conn(0x7fc010068960 0x7fc01006ae10 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc018006fd0 tx=0x7fc018006db0 comp rx=0 tx=0).stop 2023-09-30T02:06:44.968 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.966+0000 7fc02a697700 10 monclient: shutdown 2023-09-30T02:06:44.968 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.966+0000 7fc02a697700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:44.968 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.966+0000 7fc02a697700 1 -- 172.21.15.59:0/2870115393 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0240a5b70 msgr2=0x7fc0240b5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:44.968 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.966+0000 7fc02a697700 1 --2- 172.21.15.59:0/2870115393 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0240a5b70 0x7fc0240b5300 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc01400b3c0 tx=0x7fc014017000 comp rx=0 tx=0).stop 2023-09-30T02:06:44.968 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.966+0000 7fc02a697700 1 -- 172.21.15.59:0/2870115393 shutdown_connections 2023-09-30T02:06:44.968 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.966+0000 7fc02a697700 1 --2- 172.21.15.59:0/2870115393 >> v2:172.21.15.110:6800/115870 conn(0x7fc010068960 0x7fc01006ae10 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:44.968 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.966+0000 7fc02a697700 1 --2- 172.21.15.59:0/2870115393 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc0241420c0 0x7fc02414e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:44.968 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.966+0000 7fc02a697700 1 --2- 172.21.15.59:0/2870115393 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0240a5b70 0x7fc0240b5300 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:44.968 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.966+0000 7fc02a697700 1 --2- 172.21.15.59:0/2870115393 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0240a5f40 0x7fc02414dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:44.968 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.966+0000 7fc02a697700 1 -- 172.21.15.59:0/2870115393 >> 172.21.15.59:0/2870115393 conn(0x7fc0240631f0 msgr2=0x7fc0240635c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:44.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.966+0000 7fc02a697700 1 -- 172.21.15.59:0/2870115393 shutdown_connections 2023-09-30T02:06:44.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.966+0000 7fc02a697700 1 -- 172.21.15.59:0/2870115393 wait complete. 2023-09-30T02:06:44.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:44.966+0000 7fc02a697700 1 librados: shutdown 2023-09-30T02:06:47.982 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:06:48.121 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.120+0000 7f992f86d700 10 monclient: get_monmap_and_config 2023-09-30T02:06:48.121 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.120+0000 7f992f86d700 10 monclient: build_initial_monmap 2023-09-30T02:06:48.121 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.121+0000 7f992f86d700 10 monclient: monmap: 2023-09-30T02:06:48.121 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:48.121 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:48.122 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:06:48.122377+0000 2023-09-30T02:06:48.122 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:06:48.122377+0000 2023-09-30T02:06:48.122 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:48.122 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:48.122 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:06:48.122 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:06:48.122 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:06:48.122 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:06:48.122 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:06:48.122 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:48.123 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.121+0000 7f992f86d700 1 Processor -- start 2023-09-30T02:06:48.123 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.122+0000 7f992f86d700 1 -- start start 2023-09-30T02:06:48.123 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.122+0000 7f992f86d700 10 monclient: init 2023-09-30T02:06:48.123 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.123+0000 7f992f86d700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:48.124 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.123+0000 7f992f86d700 10 monclient: _add_conns ranks=[0,2,1,4,3] 2023-09-30T02:06:48.124 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.123+0000 7f992f86d700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f99280a5b70 addr v1:172.21.15.59:6789/0 2023-09-30T02:06:48.124 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.123+0000 7f992f86d700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9928142290 0x7f99280b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:48.124 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.123+0000 7f992f86d700 10 monclient(hunting): picked mon.noname-b con 0x7f9928142290 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:48.124 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.123+0000 7f992f86d700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f99280a5f40 addr v1:172.21.15.110:6789/0 2023-09-30T02:06:48.124 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.123+0000 7f992f86d700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99280a6380 con 0x7f99280a5b70 2023-09-30T02:06:48.124 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.123+0000 7f992f86d700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99280a75a0 con 0x7f99280a5f40 2023-09-30T02:06:48.124 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.123+0000 7f992f86d700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:48.124 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.123+0000 7f992f86d700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99280af3f0 con 0x7f9928142290 2023-09-30T02:06:48.124 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.123+0000 7f992f86d700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:48.124 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.123+0000 7f992f86d700 10 monclient(hunting): authenticate will time out at 1853.426335s 2023-09-30T02:06:48.125 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.124+0000 7f992de0a700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f99280a5f40 0x7f992813af30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.59:42236/0 (socket says 172.21.15.59:42236) 2023-09-30T02:06:48.125 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.124+0000 7f992de0a700 1 -- 172.21.15.59:0/272628662 learned_addr learned my addr 172.21.15.59:0/272628662 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:06:48.125 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.125+0000 7f992ce08700 1 --2- 172.21.15.59:0/272628662 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9928142290 0x7f99280b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:48.125 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.125+0000 7f992ce08700 10 monclient(hunting): get_auth_request con 0x7f9928142290 auth_method 0 2023-09-30T02:06:48.126 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.125+0000 7f992ce08700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:48.126 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.125+0000 7f992ce08700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:48.126 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.125+0000 7f992ce08700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:48.126 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.126+0000 7f992ce08700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:48.126 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.126+0000 7f992ce08700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:48.126 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.126+0000 7f992ce08700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:48.129 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.129+0000 7f992ce08700 10 monclient(hunting): handle_auth_done global_id 4373 payload 274 2023-09-30T02:06:48.129 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.129+0000 7f992ce08700 1 -- 172.21.15.59:0/272628662 >> v1:172.21.15.110:6789/0 conn(0x7f99280a5f40 legacy=0x7f992813af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:48.130 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.129+0000 7f992ce08700 1 -- 172.21.15.59:0/272628662 >> v1:172.21.15.59:6789/0 conn(0x7f99280a5b70 legacy=0x7f9928141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:06:48.130 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.129+0000 7f992ce08700 10 monclient: _finish_hunting 0 2023-09-30T02:06:48.130 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.129+0000 7f992ce08700 1 monclient: found mon.noname-b 2023-09-30T02:06:48.130 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.129+0000 7f992ce08700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:48.130 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.129+0000 7f992ce08700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.59:3301/0 2023-09-30T02:06:48.130 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.129+0000 7f992ce08700 1 -- 172.21.15.59:0/272628662 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f992813b640 con 0x7f9928142290 2023-09-30T02:06:48.132 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.132+0000 7f992ce08700 1 --2- 172.21.15.59:0/272628662 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9928142290 0x7f99280b1500 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9924009910 tx=0x7f99240091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=84db5fc2a96d903 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:48.136 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.136+0000 7f991bfff700 1 -- 172.21.15.59:0/272628662 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f992400b820 con 0x7f9928142290 2023-09-30T02:06:48.136 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.136+0000 7f991bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:48.137 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.136+0000 7f991bfff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:06:48.137 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.136+0000 7f991bfff700 10 monclient: dump: 2023-09-30T02:06:48.137 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:48.137 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:48.137 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:48.137 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:48.137 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:48.137 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:48.137 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:48.137 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:48.137 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:48.137 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:48.138 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.136+0000 7f991bfff700 10 monclient: _finish_auth 0 2023-09-30T02:06:48.138 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.136+0000 7f991bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:48.138 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.136+0000 7f992f86d700 5 monclient: authenticate success, global_id 4373 2023-09-30T02:06:48.138 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.136+0000 7f992f86d700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:06:48.194 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.193+0000 7f991bfff700 1 -- 172.21.15.59:0/272628662 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f992400b980 con 0x7f9928142290 2023-09-30T02:06:48.194 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.193+0000 7f991bfff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:48.194 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.193+0000 7f991bfff700 1 -- 172.21.15.59:0/272628662 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f992400bd20 con 0x7f9928142290 2023-09-30T02:06:48.194 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.193+0000 7f991bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:48.194 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.193+0000 7f991bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:06:48.194 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.193+0000 7f991bfff700 10 monclient: dump: 2023-09-30T02:06:48.194 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:48.195 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:48.195 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:48.195 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:48.195 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:48.195 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:48.195 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:48.195 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:48.195 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:48.195 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:48.195 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.193+0000 7f992f86d700 10 monclient: get_monmap_and_config success 2023-09-30T02:06:48.195 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.193+0000 7f992f86d700 4 set_mon_vals no callback set 2023-09-30T02:06:48.197 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.194+0000 7f992f86d700 10 monclient: shutdown 2023-09-30T02:06:48.197 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.194+0000 7f992f86d700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:48.197 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.194+0000 7f992f86d700 1 -- 172.21.15.59:0/272628662 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9928142290 msgr2=0x7f99280b1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:48.198 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.194+0000 7f992f86d700 1 --2- 172.21.15.59:0/272628662 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9928142290 0x7f99280b1500 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9924009910 tx=0x7f99240091a0 comp rx=0 tx=0).stop 2023-09-30T02:06:48.198 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.194+0000 7f992f86d700 1 -- 172.21.15.59:0/272628662 shutdown_connections 2023-09-30T02:06:48.198 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.194+0000 7f992f86d700 1 --2- 172.21.15.59:0/272628662 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9928142290 0x7f99280b1500 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:48.198 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.194+0000 7f992f86d700 1 -- 172.21.15.59:0/272628662 >> 172.21.15.59:0/272628662 conn(0x7f99280631f0 msgr2=0x7f99280635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:48.198 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.195+0000 7f992f86d700 1 -- 172.21.15.59:0/272628662 shutdown_connections 2023-09-30T02:06:48.198 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.195+0000 7f992f86d700 1 -- 172.21.15.59:0/272628662 wait complete. 2023-09-30T02:06:48.198 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.195+0000 7f992f86d700 10 monclient: build_initial_monmap 2023-09-30T02:06:48.198 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.195+0000 7f992f86d700 10 monclient: monmap: 2023-09-30T02:06:48.198 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:48.198 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:06:48.198 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:06:48.199 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:06:48.199 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:48.199 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:06:48.199 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:06:48.199 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:06:48.199 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:06:48.199 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:48.199 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.195+0000 7f992f86d700 1 librados: starting msgr at 2023-09-30T02:06:48.199 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.195+0000 7f992f86d700 1 librados: starting objecter 2023-09-30T02:06:48.199 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.196+0000 7f992f86d700 1 Processor -- start 2023-09-30T02:06:48.199 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.196+0000 7f992f86d700 1 -- start start 2023-09-30T02:06:48.199 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.196+0000 7f992f86d700 1 librados: setting wanted keys 2023-09-30T02:06:48.199 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.196+0000 7f992f86d700 1 librados: calling monclient init 2023-09-30T02:06:48.200 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.196+0000 7f992f86d700 10 monclient: init 2023-09-30T02:06:48.200 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.197+0000 7f992f86d700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:48.200 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.197+0000 7f992f86d700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T02:06:48.200 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.197+0000 7f992f86d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f99280a5b70 0x7f99281407c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:48.200 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.197+0000 7f992f86d700 10 monclient(hunting): picked mon.noname-b con 0x7f99280a5b70 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:06:48.200 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.197+0000 7f992f86d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f99280a5f40 0x7f992813dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:48.200 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.197+0000 7f992f86d700 10 monclient(hunting): picked mon.noname-a con 0x7f99280a5f40 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:06:48.200 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.197+0000 7f992f86d700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9928142290 0x7f992814e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:48.200 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.197+0000 7f992f86d700 10 monclient(hunting): picked mon.noname-c con 0x7f9928142290 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:48.200 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.197+0000 7f992f86d700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:48.201 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.197+0000 7f992f86d700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f992813e4d0 con 0x7f99280a5f40 2023-09-30T02:06:48.201 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.197+0000 7f992f86d700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:48.201 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.197+0000 7f992f86d700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99280a8230 con 0x7f99280a5b70 2023-09-30T02:06:48.201 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.197+0000 7f992f86d700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:48.201 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.197+0000 7f992f86d700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99281410f0 con 0x7f9928142290 2023-09-30T02:06:48.201 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.197+0000 7f992f86d700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:48.201 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.197+0000 7f992f86d700 10 monclient(hunting): authenticate will time out at 1853.500142s 2023-09-30T02:06:48.201 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.197+0000 7f992ce08700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f99280a5f40 0x7f992813dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:48.202 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.198+0000 7f992ce08700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f99280a5f40 0x7f992813dd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:48954/0 (socket says 172.21.15.59:48954) 2023-09-30T02:06:48.202 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.198+0000 7f992ce08700 1 -- 172.21.15.59:0/932131582 learned_addr learned my addr 172.21.15.59:0/932131582 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:06:48.202 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.198+0000 7f992ce08700 10 monclient(hunting): get_auth_request con 0x7f99280a5f40 auth_method 0 2023-09-30T02:06:48.202 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.198+0000 7f992ce08700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:48.203 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.198+0000 7f992ce08700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:48.203 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.198+0000 7f992ce08700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:48.203 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.199+0000 7f992de0a700 1 --2- 172.21.15.59:0/932131582 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9928142290 0x7f992814e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:48.203 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.199+0000 7f992ce08700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:48.203 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.199+0000 7f992ce08700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:48.203 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.199+0000 7f992ce08700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:48.203 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.199+0000 7f992de0a700 10 monclient(hunting): get_auth_request con 0x7f9928142290 auth_method 0 2023-09-30T02:06:48.203 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.199+0000 7f992de0a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:48.203 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.199+0000 7f992de0a700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:48.203 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.199+0000 7f992de0a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:48.203 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.200+0000 7f992de0a700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:48.204 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.200+0000 7f992de0a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:48.204 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.200+0000 7f992de0a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:48.204 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.203+0000 7f992ce08700 10 monclient(hunting): handle_auth_done global_id 4419 payload 824 2023-09-30T02:06:48.204 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.203+0000 7f992ce08700 1 -- 172.21.15.59:0/932131582 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9928142290 msgr2=0x7f992814e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:48.204 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.204+0000 7f992ce08700 1 --2- 172.21.15.59:0/932131582 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9928142290 0x7f992814e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:48.205 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.204+0000 7f992ce08700 1 -- 172.21.15.59:0/932131582 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f99280a5b70 msgr2=0x7f99281407c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:48.205 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.204+0000 7f992ce08700 1 --2- 172.21.15.59:0/932131582 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f99280a5b70 0x7f99281407c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:48.205 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.204+0000 7f992ce08700 10 monclient: _finish_hunting 0 2023-09-30T02:06:48.205 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.204+0000 7f992ce08700 1 monclient: found mon.noname-a 2023-09-30T02:06:48.205 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.204+0000 7f992ce08700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:48.205 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.204+0000 7f992ce08700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.59:3300/0 2023-09-30T02:06:48.205 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.204+0000 7f992ce08700 1 -- 172.21.15.59:0/932131582 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9924009420 con 0x7f99280a5f40 2023-09-30T02:06:48.207 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.206+0000 7f992ce08700 1 --2- 172.21.15.59:0/932131582 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f99280a5f40 0x7f992813dd50 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f99240030c0 tx=0x7f9924016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:48.208 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.206+0000 7f99197fa700 10 client.?.objecter ms_handle_connect 0x7f99280a5f40 2023-09-30T02:06:48.208 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.206+0000 7f99197fa700 10 client.?.objecter resend_mon_ops 2023-09-30T02:06:48.213 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.212+0000 7f99197fa700 1 -- 172.21.15.59:0/932131582 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9924011e40 con 0x7f99280a5f40 2023-09-30T02:06:48.213 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.213+0000 7f99197fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:48.214 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.213+0000 7f99197fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T02:06:48.214 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.213+0000 7f99197fa700 10 monclient: dump: 2023-09-30T02:06:48.214 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:48.214 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:48.214 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:48.214 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:48.214 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:48.214 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:48.214 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:48.214 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:48.214 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:48.214 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:48.215 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.213+0000 7f99197fa700 10 monclient: _finish_auth 0 2023-09-30T02:06:48.215 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.213+0000 7f99197fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:48.215 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.213+0000 7f99197fa700 1 -- 172.21.15.59:0/932131582 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f992401c460 con 0x7f99280a5f40 2023-09-30T02:06:48.215 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.213+0000 7f992f86d700 5 monclient: authenticate success, global_id 4419 2023-09-30T02:06:48.215 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.213+0000 7f992f86d700 10 monclient: _renew_subs 2023-09-30T02:06:48.215 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.213+0000 7f992f86d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:06:48.215 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.213+0000 7f992f86d700 1 -- 172.21.15.59:0/932131582 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f992813e960 con 0x7f99280a5f40 2023-09-30T02:06:48.215 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.213+0000 7f99197fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:48.215 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.213+0000 7f99197fa700 1 -- 172.21.15.59:0/932131582 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f992401c820 con 0x7f99280a5f40 2023-09-30T02:06:48.215 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.213+0000 7f99197fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:48.215 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.213+0000 7f99197fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T02:06:48.215 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.213+0000 7f99197fa700 10 monclient: dump: 2023-09-30T02:06:48.216 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:48.216 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:48.216 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:48.216 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:48.216 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:48.216 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:48.216 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:48.216 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:48.216 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:48.216 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:48.216 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.213+0000 7f991b7fe700 4 set_mon_vals no callback set 2023-09-30T02:06:48.216 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.213+0000 7f992f86d700 10 client.4419.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:06:48.217 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.213+0000 7f992f86d700 10 monclient: _renew_subs 2023-09-30T02:06:48.217 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.213+0000 7f992f86d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:06:48.217 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.213+0000 7f992f86d700 1 -- 172.21.15.59:0/932131582 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f992814ea60 con 0x7f99280a5f40 2023-09-30T02:06:48.217 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.213+0000 7f992f86d700 1 librados: init done 2023-09-30T02:06:48.217 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.214+0000 7f992f86d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:48.217 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.214+0000 7f992f86d700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:48.217 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.214+0000 7f992f86d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:06:48.218 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.214+0000 7f992f86d700 1 -- 172.21.15.59:0/932131582 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f992813e290 con 0x7f99280a5f40 2023-09-30T02:06:48.518 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.516+0000 7f99197fa700 1 -- 172.21.15.59:0/932131582 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f992401cc10 con 0x7f99280a5f40 2023-09-30T02:06:48.518 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.517+0000 7f99197fa700 1 --2- 172.21.15.59:0/932131582 >> v2:172.21.15.110:6800/115870 conn(0x7f990c0689e0 0x7f990c06ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:48.518 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.517+0000 7f99197fa700 1 -- 172.21.15.59:0/932131582 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f99240867e0 con 0x7f99280a5f40 2023-09-30T02:06:48.518 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.517+0000 7f99197fa700 10 client.4419.objecter ms_dispatch 0x7f9928001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:06:48.518 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.517+0000 7f99197fa700 3 client.4419.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:06:48.518 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.517+0000 7f99197fa700 3 client.4419.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:06:48.518 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.517+0000 7f99197fa700 20 client.4419.objecter dump_active .. 0 homeless 2023-09-30T02:06:48.519 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.517+0000 7f99197fa700 1 -- 172.21.15.59:0/932131582 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9924086c70 con 0x7f99280a5f40 2023-09-30T02:06:48.519 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.517+0000 7f99197fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:48.519 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.517+0000 7f99197fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:06:48.519 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.517+0000 7f992d609700 1 --2- 172.21.15.59:0/932131582 >> v2:172.21.15.110:6800/115870 conn(0x7f990c0689e0 0x7f990c06ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:48.519 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.517+0000 7f992d609700 10 monclient: get_auth_request con 0x7f990c0689e0 auth_method 0 2023-09-30T02:06:48.519 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.518+0000 7f99197fa700 10 client.4419.objecter ms_handle_connect 0x7f990c0689e0 2023-09-30T02:06:48.519 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.518+0000 7f992d609700 1 --2- 172.21.15.59:0/932131582 >> v2:172.21.15.110:6800/115870 conn(0x7f990c0689e0 0x7f990c06ae90 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f991c0097c0 tx=0x7f991c006ce0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:48.769 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.768+0000 7f992f86d700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:06:48.769 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.768+0000 7f992f86d700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:06:48.769 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.768+0000 7f992f86d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:06:48.769 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.768+0000 7f992f86d700 1 -- 172.21.15.59:0/932131582 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f992813e7b0 con 0x7f99280a5f40 2023-09-30T02:06:48.775 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.775+0000 7f99197fa700 1 -- 172.21.15.59:0/932131582 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f9924054100 con 0x7f99280a5f40 2023-09-30T02:06:48.775 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.775+0000 7f99197fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:06:48.775 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.775+0000 7f99197fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:06:48.776 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:06:48.776 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:06:48.779 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.777+0000 7f992f86d700 10 librados: watch_flush enter 2023-09-30T02:06:48.779 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.777+0000 7f992f86d700 10 librados: watch_flush exit 2023-09-30T02:06:48.779 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.777+0000 7f992f86d700 20 client.4419.objecter shutdown clearing up homeless session... 2023-09-30T02:06:48.779 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.778+0000 7f992f86d700 10 client.4419.objecter successfully canceled tick 2023-09-30T02:06:48.779 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.778+0000 7f992f86d700 1 -- 172.21.15.59:0/932131582 >> v2:172.21.15.110:6800/115870 conn(0x7f990c0689e0 msgr2=0x7f990c06ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:48.779 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.778+0000 7f992f86d700 1 --2- 172.21.15.59:0/932131582 >> v2:172.21.15.110:6800/115870 conn(0x7f990c0689e0 0x7f990c06ae90 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f991c0097c0 tx=0x7f991c006ce0 comp rx=0 tx=0).stop 2023-09-30T02:06:48.779 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.778+0000 7f992f86d700 10 monclient: shutdown 2023-09-30T02:06:48.779 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.778+0000 7f992f86d700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:48.779 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.778+0000 7f992f86d700 1 -- 172.21.15.59:0/932131582 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f99280a5f40 msgr2=0x7f992813dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:48.779 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.778+0000 7f992f86d700 1 --2- 172.21.15.59:0/932131582 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f99280a5f40 0x7f992813dd50 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f99240030c0 tx=0x7f9924016000 comp rx=0 tx=0).stop 2023-09-30T02:06:48.780 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.778+0000 7f992f86d700 1 -- 172.21.15.59:0/932131582 shutdown_connections 2023-09-30T02:06:48.780 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.778+0000 7f992f86d700 1 --2- 172.21.15.59:0/932131582 >> v2:172.21.15.110:6800/115870 conn(0x7f990c0689e0 0x7f990c06ae90 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:48.780 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.778+0000 7f992f86d700 1 --2- 172.21.15.59:0/932131582 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f99280a5b70 0x7f99281407c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:48.780 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.778+0000 7f992f86d700 1 --2- 172.21.15.59:0/932131582 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9928142290 0x7f992814e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:48.780 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.778+0000 7f992f86d700 1 --2- 172.21.15.59:0/932131582 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f99280a5f40 0x7f992813dd50 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:48.780 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.778+0000 7f992f86d700 1 -- 172.21.15.59:0/932131582 >> 172.21.15.59:0/932131582 conn(0x7f99280631f0 msgr2=0x7f99280b0be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:48.780 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.778+0000 7f992f86d700 1 -- 172.21.15.59:0/932131582 shutdown_connections 2023-09-30T02:06:48.780 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.778+0000 7f992f86d700 1 -- 172.21.15.59:0/932131582 wait complete. 2023-09-30T02:06:48.780 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:48.778+0000 7f992f86d700 1 librados: shutdown 2023-09-30T02:06:51.794 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:06:51.936 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.935+0000 7f2c7c854700 10 monclient: get_monmap_and_config 2023-09-30T02:06:51.936 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.935+0000 7f2c7c854700 10 monclient: build_initial_monmap 2023-09-30T02:06:51.936 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.935+0000 7f2c7c854700 10 monclient: monmap: 2023-09-30T02:06:51.936 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:51.936 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:51.936 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:06:51.936965+0000 2023-09-30T02:06:51.936 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:06:51.936965+0000 2023-09-30T02:06:51.936 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:51.936 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:51.937 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:06:51.937 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:06:51.937 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:06:51.937 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:06:51.937 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:06:51.937 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:51.938 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.936+0000 7f2c7c854700 1 Processor -- start 2023-09-30T02:06:51.938 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.936+0000 7f2c7c854700 1 -- start start 2023-09-30T02:06:51.938 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.936+0000 7f2c7c854700 10 monclient: init 2023-09-30T02:06:51.939 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.937+0000 7f2c7c854700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:51.939 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.937+0000 7f2c7c854700 10 monclient: _add_conns ranks=[0,3,1,4,2] 2023-09-30T02:06:51.939 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.937+0000 7f2c7c854700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2c780a5b70 addr v1:172.21.15.59:6789/0 2023-09-30T02:06:51.939 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.937+0000 7f2c7c854700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f2c78142290 0x7f2c780b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:51.939 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.937+0000 7f2c7c854700 10 monclient(hunting): picked mon.noname-a con 0x7f2c78142290 addr v2:172.21.15.59:3300/0 2023-09-30T02:06:51.939 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.937+0000 7f2c7c854700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2c780a5f40 addr v1:172.21.15.110:6789/0 2023-09-30T02:06:51.939 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.937+0000 7f2c7c854700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c780a75a0 con 0x7f2c780a5b70 2023-09-30T02:06:51.939 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.937+0000 7f2c7c854700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c780a6360 con 0x7f2c780a5f40 2023-09-30T02:06:51.939 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.937+0000 7f2c7c854700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:51.939 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.937+0000 7f2c7c854700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c780af3f0 con 0x7f2c78142290 2023-09-30T02:06:51.939 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.937+0000 7f2c7c854700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:51.940 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.937+0000 7f2c7c854700 10 monclient(hunting): authenticate will time out at 1857.240131s 2023-09-30T02:06:51.940 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.937+0000 7f2c75d9b700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f2c78142290 0x7f2c780b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:51.940 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.938+0000 7f2c76d9d700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f2c780a5f40 0x7f2c7813af30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.59:42252/0 (socket says 172.21.15.59:42252) 2023-09-30T02:06:51.940 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.938+0000 7f2c76d9d700 1 -- 172.21.15.59:0/2299713226 learned_addr learned my addr 172.21.15.59:0/2299713226 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:06:51.940 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.938+0000 7f2c75d9b700 10 monclient(hunting): get_auth_request con 0x7f2c78142290 auth_method 0 2023-09-30T02:06:51.940 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.938+0000 7f2c75d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:51.940 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.938+0000 7f2c75d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:51.940 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.938+0000 7f2c75d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:51.941 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.939+0000 7f2c75d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:51.941 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.939+0000 7f2c75d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:51.941 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.939+0000 7f2c75d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:51.941 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.941+0000 7f2c7559a700 1 -- 172.21.15.59:0/2299713226 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 524849869 0 0) 0x7f2c780a6360 con 0x7f2c780a5f40 2023-09-30T02:06:51.942 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.941+0000 7f2c7559a700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:51.942 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.941+0000 7f2c7559a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:51.942 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.941+0000 7f2c7559a700 10 monclient(hunting): my global_id is 4432 2023-09-30T02:06:51.942 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.941+0000 7f2c7559a700 1 -- 172.21.15.59:0/2299713226 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c60003570 con 0x7f2c780a5f40 2023-09-30T02:06:51.942 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.942+0000 7f2c75d9b700 10 monclient(hunting): handle_auth_done global_id 4422 payload 274 2023-09-30T02:06:51.942 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.942+0000 7f2c75d9b700 1 -- 172.21.15.59:0/2299713226 >> v1:172.21.15.110:6789/0 conn(0x7f2c780a5f40 legacy=0x7f2c7813af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:51.943 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.942+0000 7f2c75d9b700 1 -- 172.21.15.59:0/2299713226 >> v1:172.21.15.59:6789/0 conn(0x7f2c780a5b70 legacy=0x7f2c78141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:51.943 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.942+0000 7f2c75d9b700 10 monclient: _finish_hunting 0 2023-09-30T02:06:51.943 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.942+0000 7f2c75d9b700 1 monclient: found mon.noname-a 2023-09-30T02:06:51.943 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.942+0000 7f2c75d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:51.943 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.942+0000 7f2c75d9b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.59:3300/0 2023-09-30T02:06:51.943 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.942+0000 7f2c75d9b700 1 -- 172.21.15.59:0/2299713226 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c7813b640 con 0x7f2c78142290 2023-09-30T02:06:51.945 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.944+0000 7f2c75d9b700 1 --2- 172.21.15.59:0/2299713226 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c78142290 0x7f2c780b1500 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f2c6c0098e0 tx=0x7f2c6c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13c3df0571be9e04 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:51.951 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.951+0000 7f2c7559a700 1 -- 172.21.15.59:0/2299713226 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c6c00b640 con 0x7f2c78142290 2023-09-30T02:06:51.952 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.951+0000 7f2c7559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:51.952 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.951+0000 7f2c7559a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T02:06:51.952 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.951+0000 7f2c7559a700 10 monclient: dump: 2023-09-30T02:06:51.952 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:51.952 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:51.952 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:51.952 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:51.952 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:51.952 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:51.952 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:51.953 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:51.953 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:51.953 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:51.953 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.951+0000 7f2c7559a700 10 monclient: _finish_auth 0 2023-09-30T02:06:51.953 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.951+0000 7f2c7559a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:51.953 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.951+0000 7f2c7559a700 1 -- 172.21.15.59:0/2299713226 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c6c00b7a0 con 0x7f2c78142290 2023-09-30T02:06:51.953 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.951+0000 7f2c7559a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:51.953 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.951+0000 7f2c7559a700 1 -- 172.21.15.59:0/2299713226 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c6c00bb40 con 0x7f2c78142290 2023-09-30T02:06:51.953 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.951+0000 7f2c7559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:51.953 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.951+0000 7f2c7559a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T02:06:51.953 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.951+0000 7f2c7559a700 10 monclient: dump: 2023-09-30T02:06:51.954 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:51.954 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:51.954 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:51.954 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:51.954 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:51.954 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:51.954 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:51.954 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:51.954 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:51.954 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:51.954 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.951+0000 7f2c7c854700 5 monclient: authenticate success, global_id 4422 2023-09-30T02:06:51.955 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.951+0000 7f2c7c854700 10 monclient: get_monmap_and_config success 2023-09-30T02:06:51.955 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.951+0000 7f2c7c854700 4 set_mon_vals no callback set 2023-09-30T02:06:51.958 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.952+0000 7f2c7c854700 10 monclient: shutdown 2023-09-30T02:06:51.958 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.952+0000 7f2c7c854700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:51.958 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.952+0000 7f2c7c854700 1 -- 172.21.15.59:0/2299713226 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c78142290 msgr2=0x7f2c780b1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:51.958 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.952+0000 7f2c7c854700 1 --2- 172.21.15.59:0/2299713226 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c78142290 0x7f2c780b1500 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f2c6c0098e0 tx=0x7f2c6c0091a0 comp rx=0 tx=0).stop 2023-09-30T02:06:51.958 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.952+0000 7f2c7c854700 1 -- 172.21.15.59:0/2299713226 shutdown_connections 2023-09-30T02:06:51.958 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.952+0000 7f2c7c854700 1 --2- 172.21.15.59:0/2299713226 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c78142290 0x7f2c780b1500 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:51.958 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.952+0000 7f2c7c854700 1 -- 172.21.15.59:0/2299713226 >> 172.21.15.59:0/2299713226 conn(0x7f2c780631f0 msgr2=0x7f2c780635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:51.958 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.952+0000 7f2c7c854700 1 -- 172.21.15.59:0/2299713226 shutdown_connections 2023-09-30T02:06:51.959 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.952+0000 7f2c7c854700 1 -- 172.21.15.59:0/2299713226 wait complete. 2023-09-30T02:06:51.959 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.953+0000 7f2c7c854700 10 monclient: build_initial_monmap 2023-09-30T02:06:51.959 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.953+0000 7f2c7c854700 10 monclient: monmap: 2023-09-30T02:06:51.959 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:51.959 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:06:51.959 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:06:51.959 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:06:51.959 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:51.959 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:06:51.959 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:06:51.959 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:06:51.959 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:06:51.960 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:51.960 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.953+0000 7f2c7c854700 1 librados: starting msgr at 2023-09-30T02:06:51.960 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.953+0000 7f2c7c854700 1 librados: starting objecter 2023-09-30T02:06:51.960 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.953+0000 7f2c7c854700 1 Processor -- start 2023-09-30T02:06:51.960 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.953+0000 7f2c7c854700 1 -- start start 2023-09-30T02:06:51.960 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.953+0000 7f2c7c854700 1 librados: setting wanted keys 2023-09-30T02:06:51.960 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.953+0000 7f2c7c854700 1 librados: calling monclient init 2023-09-30T02:06:51.960 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.953+0000 7f2c7c854700 10 monclient: init 2023-09-30T02:06:51.960 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.954+0000 7f2c7c854700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:51.960 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.954+0000 7f2c7c854700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T02:06:51.960 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.954+0000 7f2c7c854700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2c780a5b70 0x7f2c78140ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:51.961 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.954+0000 7f2c7c854700 10 monclient(hunting): picked mon.noname-c con 0x7f2c780a5b70 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:51.961 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.954+0000 7f2c7c854700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c780a5f40 0x7f2c7813e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:51.961 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.954+0000 7f2c7c854700 10 monclient(hunting): picked mon.noname-a con 0x7f2c780a5f40 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:06:51.961 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.954+0000 7f2c7c854700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2c78142290 0x7f2c7814de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:51.961 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.954+0000 7f2c7c854700 10 monclient(hunting): picked mon.noname-b con 0x7f2c78142290 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:06:51.961 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.954+0000 7f2c7c854700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:51.961 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.954+0000 7f2c7c854700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c7813f100 con 0x7f2c780a5f40 2023-09-30T02:06:51.961 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.954+0000 7f2c7c854700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:51.961 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.954+0000 7f2c7c854700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c7813eca0 con 0x7f2c78142290 2023-09-30T02:06:51.961 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.954+0000 7f2c7c854700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:51.961 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.954+0000 7f2c7c854700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c7813ee10 con 0x7f2c780a5b70 2023-09-30T02:06:51.962 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.954+0000 7f2c7c854700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:51.962 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.954+0000 7f2c7c854700 10 monclient(hunting): authenticate will time out at 1857.257289s 2023-09-30T02:06:51.962 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.954+0000 7f2c7659c700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2c780a5b70 0x7f2c78140ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:51.962 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.955+0000 7f2c76d9d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2c78142290 0x7f2c7814de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:51.962 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.955+0000 7f2c7659c700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2c780a5b70 0x7f2c78140ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:51650/0 (socket says 172.21.15.59:51650) 2023-09-30T02:06:51.962 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.955+0000 7f2c7659c700 1 -- 172.21.15.59:0/1238450866 learned_addr learned my addr 172.21.15.59:0/1238450866 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:06:51.962 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.955+0000 7f2c7659c700 10 monclient(hunting): get_auth_request con 0x7f2c780a5b70 auth_method 0 2023-09-30T02:06:51.962 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.955+0000 7f2c7659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:51.962 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.955+0000 7f2c7659c700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:51.962 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.955+0000 7f2c7659c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:51.962 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.955+0000 7f2c76d9d700 10 monclient(hunting): get_auth_request con 0x7f2c78142290 auth_method 0 2023-09-30T02:06:51.963 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.955+0000 7f2c76d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:51.963 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.955+0000 7f2c76d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:51.963 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.955+0000 7f2c76d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:51.963 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.955+0000 7f2c75d9b700 1 --2- 172.21.15.59:0/1238450866 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c780a5f40 0x7f2c7813e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:51.963 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.956+0000 7f2c76d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:51.963 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.956+0000 7f2c76d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:51.963 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.956+0000 7f2c76d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:51.963 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.956+0000 7f2c75d9b700 10 monclient(hunting): get_auth_request con 0x7f2c780a5f40 auth_method 0 2023-09-30T02:06:51.963 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.956+0000 7f2c75d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:51.963 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.956+0000 7f2c75d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:51.963 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.956+0000 7f2c75d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:51.964 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.956+0000 7f2c7659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:51.964 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.956+0000 7f2c7659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:51.964 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.956+0000 7f2c7659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:51.964 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.957+0000 7f2c75d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:51.964 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.957+0000 7f2c75d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:51.964 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.957+0000 7f2c75d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:51.965 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.960+0000 7f2c76d9d700 10 monclient(hunting): handle_auth_done global_id 4435 payload 824 2023-09-30T02:06:51.965 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.960+0000 7f2c76d9d700 1 -- 172.21.15.59:0/1238450866 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2c780a5b70 msgr2=0x7f2c78140ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:51.965 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.960+0000 7f2c76d9d700 1 --2- 172.21.15.59:0/1238450866 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2c780a5b70 0x7f2c78140ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:51.965 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.960+0000 7f2c76d9d700 1 -- 172.21.15.59:0/1238450866 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c780a5f40 msgr2=0x7f2c7813e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:51.965 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.960+0000 7f2c76d9d700 1 --2- 172.21.15.59:0/1238450866 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c780a5f40 0x7f2c7813e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:51.965 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.960+0000 7f2c76d9d700 10 monclient: _finish_hunting 0 2023-09-30T02:06:51.965 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.960+0000 7f2c76d9d700 1 monclient: found mon.noname-b 2023-09-30T02:06:51.965 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.960+0000 7f2c76d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:51.966 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.960+0000 7f2c76d9d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:06:51.966 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.960+0000 7f2c76d9d700 1 -- 172.21.15.59:0/1238450866 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c6c009420 con 0x7f2c78142290 2023-09-30T02:06:51.966 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.963+0000 7f2c76d9d700 1 --2- 172.21.15.59:0/1238450866 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2c78142290 0x7f2c7814de00 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f2c7000cbc0 tx=0x7f2c70001a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:51.966 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.963+0000 7f2c667fc700 10 client.?.objecter ms_handle_connect 0x7f2c78142290 2023-09-30T02:06:51.966 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.963+0000 7f2c667fc700 10 client.?.objecter resend_mon_ops 2023-09-30T02:06:51.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.965+0000 7f2c667fc700 1 -- 172.21.15.59:0/1238450866 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c7000a7e0 con 0x7f2c78142290 2023-09-30T02:06:51.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.965+0000 7f2c667fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:51.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.965+0000 7f2c667fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:06:51.968 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.965+0000 7f2c667fc700 10 monclient: dump: 2023-09-30T02:06:51.968 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:51.968 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:51.968 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:51.968 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:51.968 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:51.968 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:51.968 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:51.968 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:51.968 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:51.968 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:51.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.965+0000 7f2c667fc700 10 monclient: _finish_auth 0 2023-09-30T02:06:51.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.965+0000 7f2c667fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:51.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.965+0000 7f2c7c854700 5 monclient: authenticate success, global_id 4435 2023-09-30T02:06:51.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.965+0000 7f2c7c854700 10 monclient: _renew_subs 2023-09-30T02:06:51.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.965+0000 7f2c7c854700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:51.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.965+0000 7f2c7c854700 1 -- 172.21.15.59:0/1238450866 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c7813e350 con 0x7f2c78142290 2023-09-30T02:06:51.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.965+0000 7f2c7c854700 10 client.4435.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:06:51.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.965+0000 7f2c7c854700 10 monclient: _renew_subs 2023-09-30T02:06:51.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.965+0000 7f2c7c854700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:51.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.965+0000 7f2c7c854700 1 -- 172.21.15.59:0/1238450866 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c7814e5b0 con 0x7f2c78142290 2023-09-30T02:06:51.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.965+0000 7f2c7c854700 1 librados: init done 2023-09-30T02:06:51.970 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.966+0000 7f2c7c854700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:51.970 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.966+0000 7f2c7c854700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:51.970 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.966+0000 7f2c7c854700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:51.970 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.966+0000 7f2c7c854700 1 -- 172.21.15.59:0/1238450866 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c78064a00 con 0x7f2c78142290 2023-09-30T02:06:51.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.987+0000 7f2c667fc700 1 -- 172.21.15.59:0/1238450866 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c7000ab50 con 0x7f2c78142290 2023-09-30T02:06:51.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.987+0000 7f2c667fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:51.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.987+0000 7f2c667fc700 1 -- 172.21.15.59:0/1238450866 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c7000ad00 con 0x7f2c78142290 2023-09-30T02:06:51.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.987+0000 7f2c667fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:51.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.987+0000 7f2c667fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:06:51.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.987+0000 7f2c667fc700 10 monclient: dump: 2023-09-30T02:06:51.988 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:51.989 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:51.989 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:51.989 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:51.989 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:51.989 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:51.989 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:51.989 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:51.989 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:51.989 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:51.989 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.987+0000 7f2c74d99700 4 set_mon_vals no callback set 2023-09-30T02:06:51.989 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.988+0000 7f2c667fc700 1 -- 172.21.15.59:0/1238450866 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f2c70025430 con 0x7f2c78142290 2023-09-30T02:06:51.990 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.988+0000 7f2c667fc700 1 --2- 172.21.15.59:0/1238450866 >> v2:172.21.15.110:6800/115870 conn(0x7f2c58068890 0x7f2c5806ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:51.990 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.988+0000 7f2c7659c700 1 --2- 172.21.15.59:0/1238450866 >> v2:172.21.15.110:6800/115870 conn(0x7f2c58068890 0x7f2c5806ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:51.990 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.988+0000 7f2c7659c700 10 monclient: get_auth_request con 0x7f2c58068890 auth_method 0 2023-09-30T02:06:51.990 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.989+0000 7f2c7659c700 1 --2- 172.21.15.59:0/1238450866 >> v2:172.21.15.110:6800/115870 conn(0x7f2c58068890 0x7f2c5806ad40 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2c68007fd0 tx=0x7f2c68000bc0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:51.991 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:51.989+0000 7f2c667fc700 10 client.4435.objecter ms_handle_connect 0x7f2c58068890 2023-09-30T02:06:52.220 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:52.220+0000 7f2c667fc700 1 -- 172.21.15.59:0/1238450866 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f2c70056760 con 0x7f2c78142290 2023-09-30T02:06:52.220 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:52.220+0000 7f2c667fc700 10 client.4435.objecter ms_dispatch 0x7f2c78001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:06:52.220 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:52.220+0000 7f2c667fc700 3 client.4435.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:06:52.220 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:52.220+0000 7f2c667fc700 3 client.4435.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:06:52.220 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:52.220+0000 7f2c667fc700 20 client.4435.objecter dump_active .. 0 homeless 2023-09-30T02:06:52.230 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:52.231+0000 7f2c667fc700 1 -- 172.21.15.59:0/1238450866 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2c700523a0 con 0x7f2c78142290 2023-09-30T02:06:52.230 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:52.231+0000 7f2c667fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:52.231 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:52.231+0000 7f2c667fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:06:52.487 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:52.486+0000 7f2c7c854700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:06:52.487 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:52.486+0000 7f2c7c854700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:06:52.487 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:52.486+0000 7f2c7c854700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:06:52.487 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:52.486+0000 7f2c7c854700 1 -- 172.21.15.59:0/1238450866 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f2c7806a720 con 0x7f2c78142290 2023-09-30T02:06:52.492 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:52.492+0000 7f2c667fc700 1 -- 172.21.15.59:0/1238450866 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f2c7000ae60 con 0x7f2c78142290 2023-09-30T02:06:52.492 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:52.492+0000 7f2c667fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:06:52.492 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:52.492+0000 7f2c667fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:06:52.493 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:06:52.493 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:06:52.495 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:52.494+0000 7f2c7c854700 10 librados: watch_flush enter 2023-09-30T02:06:52.495 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:52.494+0000 7f2c7c854700 10 librados: watch_flush exit 2023-09-30T02:06:52.495 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:52.494+0000 7f2c7c854700 20 client.4435.objecter shutdown clearing up homeless session... 2023-09-30T02:06:52.495 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:52.494+0000 7f2c7c854700 10 client.4435.objecter successfully canceled tick 2023-09-30T02:06:52.495 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:52.494+0000 7f2c7c854700 1 -- 172.21.15.59:0/1238450866 >> v2:172.21.15.110:6800/115870 conn(0x7f2c58068890 msgr2=0x7f2c5806ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:52.495 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:52.494+0000 7f2c7c854700 1 --2- 172.21.15.59:0/1238450866 >> v2:172.21.15.110:6800/115870 conn(0x7f2c58068890 0x7f2c5806ad40 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f2c68007fd0 tx=0x7f2c68000bc0 comp rx=0 tx=0).stop 2023-09-30T02:06:52.495 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:52.494+0000 7f2c7c854700 10 monclient: shutdown 2023-09-30T02:06:52.496 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:52.494+0000 7f2c7c854700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:52.496 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:52.494+0000 7f2c7c854700 1 -- 172.21.15.59:0/1238450866 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2c78142290 msgr2=0x7f2c7814de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:52.496 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:52.494+0000 7f2c7c854700 1 --2- 172.21.15.59:0/1238450866 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2c78142290 0x7f2c7814de00 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f2c7000cbc0 tx=0x7f2c70001a20 comp rx=0 tx=0).stop 2023-09-30T02:06:52.496 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:52.494+0000 7f2c7c854700 1 -- 172.21.15.59:0/1238450866 shutdown_connections 2023-09-30T02:06:52.496 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:52.494+0000 7f2c7c854700 1 --2- 172.21.15.59:0/1238450866 >> v2:172.21.15.110:6800/115870 conn(0x7f2c58068890 0x7f2c5806ad40 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:52.496 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:52.494+0000 7f2c7c854700 1 --2- 172.21.15.59:0/1238450866 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2c78142290 0x7f2c7814de00 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:52.496 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:52.494+0000 7f2c7c854700 1 --2- 172.21.15.59:0/1238450866 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2c780a5b70 0x7f2c78140ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:52.496 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:52.494+0000 7f2c7c854700 1 --2- 172.21.15.59:0/1238450866 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c780a5f40 0x7f2c7813e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:52.496 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:52.494+0000 7f2c7c854700 1 -- 172.21.15.59:0/1238450866 >> 172.21.15.59:0/1238450866 conn(0x7f2c780631f0 msgr2=0x7f2c78070f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:52.496 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:52.494+0000 7f2c7c854700 1 -- 172.21.15.59:0/1238450866 shutdown_connections 2023-09-30T02:06:52.496 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:52.494+0000 7f2c7c854700 1 -- 172.21.15.59:0/1238450866 wait complete. 2023-09-30T02:06:52.497 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:52.494+0000 7f2c7c854700 1 librados: shutdown 2023-09-30T02:06:53.182 INFO:tasks.ceph.mon.a.smithi059.stderr:==00:00:02:03.048 116256== Warning: unimplemented fcntl command: 1036 2023-09-30T02:06:53.191 INFO:tasks.ceph.mon.a.smithi059.stderr:==00:00:02:03.057 116256== Warning: unimplemented fcntl command: 1036 2023-09-30T02:06:53.232 INFO:tasks.ceph.mon.a.smithi059.stderr:==00:00:02:03.098 116256== Warning: unimplemented fcntl command: 1036 2023-09-30T02:06:53.239 INFO:tasks.ceph.mon.b.smithi110.stderr:==00:00:02:03.087 115869== Warning: unimplemented fcntl command: 1036 2023-09-30T02:06:53.246 INFO:tasks.ceph.mon.b.smithi110.stderr:==00:00:02:03.093 115869== Warning: unimplemented fcntl command: 1036 2023-09-30T02:06:53.259 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:02:03.125 116257== Warning: unimplemented fcntl command: 1036 2023-09-30T02:06:53.267 INFO:tasks.ceph.mon.b.smithi110.stderr:==00:00:02:03.115 115869== Warning: unimplemented fcntl command: 1036 2023-09-30T02:06:53.269 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:02:03.135 116257== Warning: unimplemented fcntl command: 1036 2023-09-30T02:06:53.293 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:02:03.159 116257== Warning: unimplemented fcntl command: 1036 2023-09-30T02:06:55.510 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:06:55.650 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.650+0000 7f85b2e82700 10 monclient: get_monmap_and_config 2023-09-30T02:06:55.651 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.650+0000 7f85b2e82700 10 monclient: build_initial_monmap 2023-09-30T02:06:55.651 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.650+0000 7f85b2e82700 10 monclient: monmap: 2023-09-30T02:06:55.651 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:55.651 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:55.651 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:06:55.651792+0000 2023-09-30T02:06:55.651 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:06:55.651792+0000 2023-09-30T02:06:55.651 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:55.651 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:55.651 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:06:55.651 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:06:55.652 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:06:55.652 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:06:55.652 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:06:55.652 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:55.655 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.651+0000 7f85b2e82700 1 Processor -- start 2023-09-30T02:06:55.655 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.651+0000 7f85b2e82700 1 -- start start 2023-09-30T02:06:55.655 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.651+0000 7f85b2e82700 10 monclient: init 2023-09-30T02:06:55.655 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.651+0000 7f85b2e82700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:55.655 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.651+0000 7f85b2e82700 10 monclient: _add_conns ranks=[3,1,2,0,4] 2023-09-30T02:06:55.655 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.651+0000 7f85b2e82700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f85ac0a5b70 0x7f85ac0b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:55.655 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.651+0000 7f85b2e82700 10 monclient(hunting): picked mon.noname-a con 0x7f85ac0a5b70 addr v2:172.21.15.59:3300/0 2023-09-30T02:06:55.655 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.651+0000 7f85b2e82700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f85ac0a5f40 addr v1:172.21.15.110:6789/0 2023-09-30T02:06:55.656 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.651+0000 7f85b2e82700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f85ac142290 0x7f85ac13af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:55.656 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.652+0000 7f85b2e82700 10 monclient(hunting): picked mon.noname-b con 0x7f85ac142290 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:55.656 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.652+0000 7f85b2e82700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85ac0a75a0 con 0x7f85ac0a5f40 2023-09-30T02:06:55.656 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.652+0000 7f85b2e82700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:55.656 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.652+0000 7f85b2e82700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85ac1426d0 con 0x7f85ac0a5b70 2023-09-30T02:06:55.656 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.652+0000 7f85b2e82700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:55.656 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.652+0000 7f85b2e82700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85ac0a63e0 con 0x7f85ac142290 2023-09-30T02:06:55.656 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.652+0000 7f85b2e82700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:55.656 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.652+0000 7f85b2e82700 10 monclient(hunting): authenticate will time out at 1860.955056s 2023-09-30T02:06:55.656 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.652+0000 7f85b0c1e700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f85ac0a5b70 0x7f85ac0b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:55.656 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.652+0000 7f85b141f700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f85ac142290 0x7f85ac13af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:55.656 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.652+0000 7f85abfff700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f85ac0a5f40 0x7f85ac141b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.59:42256/0 (socket says 172.21.15.59:42256) 2023-09-30T02:06:55.657 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.652+0000 7f85abfff700 1 -- 172.21.15.59:0/2721682879 learned_addr learned my addr 172.21.15.59:0/2721682879 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:06:55.657 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.653+0000 7f85b0c1e700 10 monclient(hunting): get_auth_request con 0x7f85ac0a5b70 auth_method 0 2023-09-30T02:06:55.657 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.653+0000 7f85b0c1e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:55.657 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.653+0000 7f85b0c1e700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:55.657 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.653+0000 7f85b0c1e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:55.657 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.653+0000 7f85b141f700 10 monclient(hunting): get_auth_request con 0x7f85ac142290 auth_method 0 2023-09-30T02:06:55.657 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.653+0000 7f85b141f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:55.657 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.653+0000 7f85b141f700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:55.657 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.653+0000 7f85b141f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:55.657 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.654+0000 7f85b141f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:55.657 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.654+0000 7f85b141f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:55.657 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.654+0000 7f85b141f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:55.658 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.654+0000 7f85b0c1e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:55.658 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.654+0000 7f85b0c1e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:55.658 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.654+0000 7f85b0c1e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:55.659 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.656+0000 7f85ab7fe700 1 -- 172.21.15.59:0/2721682879 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4255115930 0 0) 0x7f85ac0a75a0 con 0x7f85ac0a5f40 2023-09-30T02:06:55.659 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.656+0000 7f85ab7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:55.659 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.656+0000 7f85ab7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:55.659 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.656+0000 7f85ab7fe700 10 monclient(hunting): my global_id is 4438 2023-09-30T02:06:55.659 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.656+0000 7f85ab7fe700 1 -- 172.21.15.59:0/2721682879 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8594003570 con 0x7f85ac0a5f40 2023-09-30T02:06:55.659 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.657+0000 7f85b0c1e700 10 monclient(hunting): handle_auth_done global_id 4428 payload 274 2023-09-30T02:06:55.659 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.657+0000 7f85b0c1e700 1 -- 172.21.15.59:0/2721682879 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f85ac142290 msgr2=0x7f85ac13af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:06:55.659 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.657+0000 7f85b0c1e700 1 --2- 172.21.15.59:0/2721682879 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f85ac142290 0x7f85ac13af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:55.660 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.657+0000 7f85b0c1e700 1 -- 172.21.15.59:0/2721682879 >> v1:172.21.15.110:6789/0 conn(0x7f85ac0a5f40 legacy=0x7f85ac141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:55.660 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.657+0000 7f85b0c1e700 10 monclient: _finish_hunting 0 2023-09-30T02:06:55.660 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.657+0000 7f85b0c1e700 1 monclient: found mon.noname-a 2023-09-30T02:06:55.660 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.657+0000 7f85b0c1e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:55.660 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.657+0000 7f85b0c1e700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.59:3300/0 2023-09-30T02:06:55.660 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.657+0000 7f85b0c1e700 1 -- 172.21.15.59:0/2721682879 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85ac0af3f0 con 0x7f85ac0a5b70 2023-09-30T02:06:55.660 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.659+0000 7f85b0c1e700 1 --2- 172.21.15.59:0/2721682879 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85ac0a5b70 0x7f85ac0b1500 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f859c009910 tx=0x7f859c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55410bc4edd493fe server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:55.664 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.664+0000 7f85ab7fe700 1 -- 172.21.15.59:0/2721682879 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f859c011630 con 0x7f85ac0a5b70 2023-09-30T02:06:55.664 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.664+0000 7f85ab7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:55.665 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.664+0000 7f85ab7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T02:06:55.665 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.664+0000 7f85ab7fe700 10 monclient: dump: 2023-09-30T02:06:55.665 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:55.665 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:55.665 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:55.665 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:55.665 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:55.665 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:55.665 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:55.665 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:55.665 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:55.665 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:55.666 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.664+0000 7f85ab7fe700 10 monclient: _finish_auth 0 2023-09-30T02:06:55.666 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.664+0000 7f85ab7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:55.666 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.664+0000 7f85ab7fe700 1 -- 172.21.15.59:0/2721682879 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f859c011790 con 0x7f85ac0a5b70 2023-09-30T02:06:55.666 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.664+0000 7f85ab7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:55.666 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.664+0000 7f85ab7fe700 1 -- 172.21.15.59:0/2721682879 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f859c011b30 con 0x7f85ac0a5b70 2023-09-30T02:06:55.666 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.664+0000 7f85ab7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:55.666 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.664+0000 7f85ab7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T02:06:55.666 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.664+0000 7f85ab7fe700 10 monclient: dump: 2023-09-30T02:06:55.666 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:55.666 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:55.666 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:55.666 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:55.667 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:55.667 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:55.667 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:55.667 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:55.667 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:55.667 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:55.667 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.664+0000 7f85b2e82700 5 monclient: authenticate success, global_id 4428 2023-09-30T02:06:55.667 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.664+0000 7f85b2e82700 10 monclient: get_monmap_and_config success 2023-09-30T02:06:55.667 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.664+0000 7f85b2e82700 4 set_mon_vals no callback set 2023-09-30T02:06:55.670 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.665+0000 7f85b2e82700 10 monclient: shutdown 2023-09-30T02:06:55.670 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.665+0000 7f85b2e82700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:55.670 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.665+0000 7f85b2e82700 1 -- 172.21.15.59:0/2721682879 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85ac0a5b70 msgr2=0x7f85ac0b1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:55.670 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.665+0000 7f85b2e82700 1 --2- 172.21.15.59:0/2721682879 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85ac0a5b70 0x7f85ac0b1500 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f859c009910 tx=0x7f859c0091a0 comp rx=0 tx=0).stop 2023-09-30T02:06:55.670 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.665+0000 7f85b2e82700 1 -- 172.21.15.59:0/2721682879 shutdown_connections 2023-09-30T02:06:55.670 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.665+0000 7f85b2e82700 1 --2- 172.21.15.59:0/2721682879 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f85ac142290 0x7f85ac13af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:55.670 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.665+0000 7f85b2e82700 1 --2- 172.21.15.59:0/2721682879 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85ac0a5b70 0x7f85ac0b1500 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:55.670 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.665+0000 7f85b2e82700 1 -- 172.21.15.59:0/2721682879 >> 172.21.15.59:0/2721682879 conn(0x7f85ac0631f0 msgr2=0x7f85ac0635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:55.670 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.665+0000 7f85b2e82700 1 -- 172.21.15.59:0/2721682879 shutdown_connections 2023-09-30T02:06:55.671 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.665+0000 7f85b2e82700 1 -- 172.21.15.59:0/2721682879 wait complete. 2023-09-30T02:06:55.671 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.666+0000 7f85b2e82700 10 monclient: build_initial_monmap 2023-09-30T02:06:55.671 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.666+0000 7f85b2e82700 10 monclient: monmap: 2023-09-30T02:06:55.671 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:55.671 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:06:55.671 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:06:55.671 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:06:55.671 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:55.671 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:06:55.671 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:06:55.671 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:06:55.671 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:06:55.672 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:55.672 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.666+0000 7f85b2e82700 1 librados: starting msgr at 2023-09-30T02:06:55.672 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.666+0000 7f85b2e82700 1 librados: starting objecter 2023-09-30T02:06:55.672 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.666+0000 7f85b2e82700 1 Processor -- start 2023-09-30T02:06:55.672 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.666+0000 7f85b2e82700 1 -- start start 2023-09-30T02:06:55.672 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.666+0000 7f85b2e82700 1 librados: setting wanted keys 2023-09-30T02:06:55.672 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.666+0000 7f85b2e82700 1 librados: calling monclient init 2023-09-30T02:06:55.672 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.666+0000 7f85b2e82700 10 monclient: init 2023-09-30T02:06:55.672 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.667+0000 7f85b2e82700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:55.672 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.667+0000 7f85b2e82700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T02:06:55.672 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.667+0000 7f85b2e82700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f85ac0a5b70 0x7f85ac0a7c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:55.672 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.667+0000 7f85b2e82700 10 monclient(hunting): picked mon.noname-c con 0x7f85ac0a5b70 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:55.673 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.667+0000 7f85b2e82700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f85ac0a5f40 0x7f85ac14dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:55.673 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.667+0000 7f85b2e82700 10 monclient(hunting): picked mon.noname-b con 0x7f85ac0a5f40 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:06:55.673 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.667+0000 7f85b2e82700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85ac142290 0x7f85ac14e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:55.673 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.667+0000 7f85b2e82700 10 monclient(hunting): picked mon.noname-a con 0x7f85ac142290 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:06:55.673 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.667+0000 7f85b2e82700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:55.673 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.667+0000 7f85b2e82700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85ac14e7a0 con 0x7f85ac142290 2023-09-30T02:06:55.673 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.667+0000 7f85b2e82700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:55.673 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.667+0000 7f85b2e82700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85ac0a95e0 con 0x7f85ac0a5f40 2023-09-30T02:06:55.673 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.667+0000 7f85b2e82700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:55.673 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.667+0000 7f85b2e82700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85ac142d20 con 0x7f85ac0a5b70 2023-09-30T02:06:55.673 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.667+0000 7f85b2e82700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:55.674 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.667+0000 7f85b2e82700 10 monclient(hunting): authenticate will time out at 1860.970960s 2023-09-30T02:06:55.674 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.668+0000 7f85b0c1e700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f85ac0a5b70 0x7f85ac0a7c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:55.674 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.668+0000 7f85abfff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f85ac0a5f40 0x7f85ac14dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:55.674 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.669+0000 7f85b0c1e700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f85ac0a5b70 0x7f85ac0a7c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:51672/0 (socket says 172.21.15.59:51672) 2023-09-30T02:06:55.674 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.669+0000 7f85b0c1e700 1 -- 172.21.15.59:0/1776953754 learned_addr learned my addr 172.21.15.59:0/1776953754 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:06:55.674 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.669+0000 7f85b0c1e700 10 monclient(hunting): get_auth_request con 0x7f85ac0a5b70 auth_method 0 2023-09-30T02:06:55.674 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.669+0000 7f85b0c1e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:55.674 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.669+0000 7f85b0c1e700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:55.674 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.669+0000 7f85b0c1e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:55.674 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.669+0000 7f85abfff700 10 monclient(hunting): get_auth_request con 0x7f85ac0a5f40 auth_method 0 2023-09-30T02:06:55.674 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.669+0000 7f85abfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:55.674 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.669+0000 7f85abfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:55.675 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.669+0000 7f85abfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:55.675 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.669+0000 7f85abfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:55.675 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.669+0000 7f85abfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:55.675 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.669+0000 7f85abfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:55.675 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.669+0000 7f85b0c1e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:55.675 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.669+0000 7f85b0c1e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:55.675 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.669+0000 7f85b0c1e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:55.675 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.669+0000 7f85b141f700 1 --2- 172.21.15.59:0/1776953754 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85ac142290 0x7f85ac14e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:55.676 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.670+0000 7f85b141f700 10 monclient(hunting): get_auth_request con 0x7f85ac142290 auth_method 0 2023-09-30T02:06:55.676 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.671+0000 7f85b141f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:55.676 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.671+0000 7f85b141f700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:55.676 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.671+0000 7f85b141f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:55.676 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.673+0000 7f85b141f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:55.677 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.673+0000 7f85b141f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:55.677 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.673+0000 7f85b141f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:55.677 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.674+0000 7f85b0c1e700 10 monclient(hunting): handle_auth_done global_id 4385 payload 824 2023-09-30T02:06:55.677 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.674+0000 7f85b0c1e700 1 -- 172.21.15.59:0/1776953754 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f85ac0a5f40 msgr2=0x7f85ac14dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:55.677 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.674+0000 7f85b0c1e700 1 --2- 172.21.15.59:0/1776953754 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f85ac0a5f40 0x7f85ac14dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:55.677 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.674+0000 7f85b0c1e700 1 -- 172.21.15.59:0/1776953754 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85ac142290 msgr2=0x7f85ac14e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:55.677 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.674+0000 7f85b0c1e700 1 --2- 172.21.15.59:0/1776953754 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85ac142290 0x7f85ac14e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:55.677 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.674+0000 7f85b0c1e700 10 monclient: _finish_hunting 0 2023-09-30T02:06:55.677 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.674+0000 7f85b0c1e700 1 monclient: found mon.noname-c 2023-09-30T02:06:55.677 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.674+0000 7f85b0c1e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:55.677 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.674+0000 7f85b0c1e700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.59:3301/0 2023-09-30T02:06:55.677 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.674+0000 7f85b0c1e700 1 -- 172.21.15.59:0/1776953754 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f859c009420 con 0x7f85ac0a5b70 2023-09-30T02:06:55.678 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.676+0000 7f85b0c1e700 1 --2- 172.21.15.59:0/1776953754 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f85ac0a5b70 0x7f85ac0a7c80 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f859c009f00 tx=0x7f859c017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:55.678 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.676+0000 7f85a8ff9700 10 client.?.objecter ms_handle_connect 0x7f85ac0a5b70 2023-09-30T02:06:55.678 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.676+0000 7f85a8ff9700 10 client.?.objecter resend_mon_ops 2023-09-30T02:06:55.683 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.681+0000 7f85a8ff9700 1 -- 172.21.15.59:0/1776953754 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f859c006af0 con 0x7f85ac0a5b70 2023-09-30T02:06:55.683 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.681+0000 7f85a8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:55.683 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.681+0000 7f85a8ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:06:55.683 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.681+0000 7f85a8ff9700 10 monclient: dump: 2023-09-30T02:06:55.683 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:55.683 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:55.683 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:55.683 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:55.683 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:55.683 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:55.684 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:55.684 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:55.684 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:55.684 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:55.684 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.682+0000 7f85a8ff9700 10 monclient: _finish_auth 0 2023-09-30T02:06:55.684 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.682+0000 7f85a8ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:55.684 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.682+0000 7f85a8ff9700 1 -- 172.21.15.59:0/1776953754 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f859c000bc0 con 0x7f85ac0a5b70 2023-09-30T02:06:55.684 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.682+0000 7f85b2e82700 5 monclient: authenticate success, global_id 4385 2023-09-30T02:06:55.684 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.682+0000 7f85a8ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:55.684 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.682+0000 7f85b2e82700 10 monclient: _renew_subs 2023-09-30T02:06:55.684 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.682+0000 7f85a8ff9700 1 -- 172.21.15.59:0/1776953754 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f859c025420 con 0x7f85ac0a5b70 2023-09-30T02:06:55.684 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.682+0000 7f85b2e82700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:06:55.685 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.682+0000 7f85b2e82700 1 -- 172.21.15.59:0/1776953754 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85ac1445f0 con 0x7f85ac0a5b70 2023-09-30T02:06:55.685 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.682+0000 7f85a8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:55.685 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.682+0000 7f85a8ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:06:55.685 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.682+0000 7f85a8ff9700 10 monclient: dump: 2023-09-30T02:06:55.685 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:55.685 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:55.685 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:55.685 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:55.685 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:55.685 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:55.685 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:55.685 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:55.685 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:55.686 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:55.686 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.682+0000 7f85aaffd700 4 set_mon_vals no callback set 2023-09-30T02:06:55.686 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.682+0000 7f85b2e82700 10 client.4385.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:06:55.686 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.682+0000 7f85b2e82700 10 monclient: _renew_subs 2023-09-30T02:06:55.686 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.682+0000 7f85b2e82700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:06:55.686 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.682+0000 7f85b2e82700 1 -- 172.21.15.59:0/1776953754 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85ac0a81c0 con 0x7f85ac0a5b70 2023-09-30T02:06:55.686 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.682+0000 7f85b2e82700 1 librados: init done 2023-09-30T02:06:55.687 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.683+0000 7f85b2e82700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:55.687 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.683+0000 7f85b2e82700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:55.687 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.683+0000 7f85b2e82700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:06:55.687 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.683+0000 7f85b2e82700 1 -- 172.21.15.59:0/1776953754 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85ac14db70 con 0x7f85ac0a5b70 2023-09-30T02:06:55.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.988+0000 7f85a8ff9700 1 -- 172.21.15.59:0/1776953754 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f859c023070 con 0x7f85ac0a5b70 2023-09-30T02:06:55.989 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.989+0000 7f85a8ff9700 1 --2- 172.21.15.59:0/1776953754 >> v2:172.21.15.110:6800/115870 conn(0x7f85980689e0 0x7f859806ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:55.990 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.989+0000 7f85a8ff9700 1 -- 172.21.15.59:0/1776953754 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f859c085900 con 0x7f85ac0a5b70 2023-09-30T02:06:55.990 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.989+0000 7f85a8ff9700 10 client.4385.objecter ms_dispatch 0x7f85ac001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:06:55.990 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.989+0000 7f85a8ff9700 3 client.4385.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:06:55.990 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.989+0000 7f85a8ff9700 3 client.4385.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:06:55.990 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.989+0000 7f85a8ff9700 20 client.4385.objecter dump_active .. 0 homeless 2023-09-30T02:06:55.990 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.989+0000 7f85abfff700 1 --2- 172.21.15.59:0/1776953754 >> v2:172.21.15.110:6800/115870 conn(0x7f85980689e0 0x7f859806ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:55.990 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.989+0000 7f85a8ff9700 1 -- 172.21.15.59:0/1776953754 <== mon.2 v2:172.21.15.59:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f859c0b0640 con 0x7f85ac0a5b70 2023-09-30T02:06:55.990 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.989+0000 7f85a8ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:55.990 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.989+0000 7f85a8ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:06:55.990 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.989+0000 7f85abfff700 10 monclient: get_auth_request con 0x7f85980689e0 auth_method 0 2023-09-30T02:06:55.990 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.989+0000 7f85a8ff9700 10 client.4385.objecter ms_handle_connect 0x7f85980689e0 2023-09-30T02:06:55.991 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:55.989+0000 7f85abfff700 1 --2- 172.21.15.59:0/1776953754 >> v2:172.21.15.110:6800/115870 conn(0x7f85980689e0 0x7f859806ae90 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f85ac0b5600 tx=0x7f85a0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:56.243 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:56.242+0000 7f85b2e82700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:06:56.243 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:56.242+0000 7f85b2e82700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:06:56.243 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:56.242+0000 7f85b2e82700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:06:56.243 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:56.242+0000 7f85b2e82700 1 -- 172.21.15.59:0/1776953754 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f85ac14d980 con 0x7f85ac0a5b70 2023-09-30T02:06:56.260 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:56.259+0000 7f85a8ff9700 1 -- 172.21.15.59:0/1776953754 <== mon.2 v2:172.21.15.59:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f859c053310 con 0x7f85ac0a5b70 2023-09-30T02:06:56.260 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:56.259+0000 7f85a8ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:06:56.260 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:56.259+0000 7f85a8ff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:06:56.260 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:06:56.260 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:06:56.262 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:56.261+0000 7f85b2e82700 10 librados: watch_flush enter 2023-09-30T02:06:56.263 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:56.262+0000 7f85b2e82700 10 librados: watch_flush exit 2023-09-30T02:06:56.263 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:56.262+0000 7f85b2e82700 20 client.4385.objecter shutdown clearing up homeless session... 2023-09-30T02:06:56.263 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:56.262+0000 7f85b2e82700 10 client.4385.objecter successfully canceled tick 2023-09-30T02:06:56.263 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:56.262+0000 7f85b2e82700 1 -- 172.21.15.59:0/1776953754 >> v2:172.21.15.110:6800/115870 conn(0x7f85980689e0 msgr2=0x7f859806ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:56.263 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:56.262+0000 7f85b2e82700 1 --2- 172.21.15.59:0/1776953754 >> v2:172.21.15.110:6800/115870 conn(0x7f85980689e0 0x7f859806ae90 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f85ac0b5600 tx=0x7f85a0009040 comp rx=0 tx=0).stop 2023-09-30T02:06:56.263 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:56.262+0000 7f85b2e82700 10 monclient: shutdown 2023-09-30T02:06:56.263 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:56.262+0000 7f85b2e82700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:56.263 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:56.262+0000 7f85b2e82700 1 -- 172.21.15.59:0/1776953754 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f85ac0a5b70 msgr2=0x7f85ac0a7c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:56.263 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:56.262+0000 7f85b2e82700 1 --2- 172.21.15.59:0/1776953754 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f85ac0a5b70 0x7f85ac0a7c80 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f859c009f00 tx=0x7f859c017000 comp rx=0 tx=0).stop 2023-09-30T02:06:56.264 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:56.262+0000 7f85b2e82700 1 -- 172.21.15.59:0/1776953754 shutdown_connections 2023-09-30T02:06:56.264 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:56.262+0000 7f85b2e82700 1 --2- 172.21.15.59:0/1776953754 >> v2:172.21.15.110:6800/115870 conn(0x7f85980689e0 0x7f859806ae90 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:56.264 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:56.262+0000 7f85b2e82700 1 --2- 172.21.15.59:0/1776953754 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85ac142290 0x7f85ac14e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:56.264 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:56.262+0000 7f85b2e82700 1 --2- 172.21.15.59:0/1776953754 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f85ac0a5b70 0x7f85ac0a7c80 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:56.264 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:56.262+0000 7f85b2e82700 1 --2- 172.21.15.59:0/1776953754 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f85ac0a5f40 0x7f85ac14dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:56.264 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:56.262+0000 7f85b2e82700 1 -- 172.21.15.59:0/1776953754 >> 172.21.15.59:0/1776953754 conn(0x7f85ac0631f0 msgr2=0x7f85ac139610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:56.264 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:56.262+0000 7f85b2e82700 1 -- 172.21.15.59:0/1776953754 shutdown_connections 2023-09-30T02:06:56.264 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:56.262+0000 7f85b2e82700 1 -- 172.21.15.59:0/1776953754 wait complete. 2023-09-30T02:06:56.264 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:56.262+0000 7f85b2e82700 1 librados: shutdown 2023-09-30T02:06:59.277 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:06:59.414 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.413+0000 7f56e7796700 10 monclient: get_monmap_and_config 2023-09-30T02:06:59.414 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.413+0000 7f56e7796700 10 monclient: build_initial_monmap 2023-09-30T02:06:59.414 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.413+0000 7f56e7796700 10 monclient: monmap: 2023-09-30T02:06:59.414 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:59.414 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:59.414 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:06:59.415090+0000 2023-09-30T02:06:59.414 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:06:59.415090+0000 2023-09-30T02:06:59.415 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:59.415 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:59.415 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:06:59.415 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:06:59.415 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:06:59.415 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:06:59.415 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:06:59.415 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:59.417 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.414+0000 7f56e7796700 1 Processor -- start 2023-09-30T02:06:59.417 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.414+0000 7f56e7796700 1 -- start start 2023-09-30T02:06:59.417 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.414+0000 7f56e7796700 10 monclient: init 2023-09-30T02:06:59.417 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.415+0000 7f56e7796700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:59.417 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.415+0000 7f56e7796700 10 monclient: _add_conns ranks=[3,0,2,4,1] 2023-09-30T02:06:59.417 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.415+0000 7f56e7796700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f56e00a5b70 0x7f56e00b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:59.418 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.415+0000 7f56e7796700 10 monclient(hunting): picked mon.noname-a con 0x7f56e00a5b70 addr v2:172.21.15.59:3300/0 2023-09-30T02:06:59.418 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.415+0000 7f56e7796700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f56e00a5f40 addr v1:172.21.15.59:6789/0 2023-09-30T02:06:59.418 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.415+0000 7f56e7796700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f56e0142290 0x7f56e013af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:59.418 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.415+0000 7f56e7796700 10 monclient(hunting): picked mon.noname-b con 0x7f56e0142290 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:59.418 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.415+0000 7f56e7796700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56e00a75a0 con 0x7f56e00a5f40 2023-09-30T02:06:59.418 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.415+0000 7f56e7796700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:59.418 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.415+0000 7f56e7796700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56e01426d0 con 0x7f56e00a5b70 2023-09-30T02:06:59.418 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.415+0000 7f56e7796700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:59.418 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.415+0000 7f56e7796700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56e00a63e0 con 0x7f56e0142290 2023-09-30T02:06:59.418 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.415+0000 7f56e7796700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:59.418 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.415+0000 7f56e7796700 10 monclient(hunting): authenticate will time out at 1864.718340s 2023-09-30T02:06:59.418 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.415+0000 7f56e5532700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f56e00a5b70 0x7f56e00b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:59.419 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.416+0000 7f56e5d33700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f56e0142290 0x7f56e013af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:59.419 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.416+0000 7f56e5d33700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f56e0142290 0x7f56e013af30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:54424/0 (socket says 172.21.15.59:54424) 2023-09-30T02:06:59.419 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.416+0000 7f56e5d33700 1 -- 172.21.15.59:0/3292057539 learned_addr learned my addr 172.21.15.59:0/3292057539 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:06:59.419 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.416+0000 7f56e5d33700 10 monclient(hunting): get_auth_request con 0x7f56e0142290 auth_method 0 2023-09-30T02:06:59.419 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.416+0000 7f56e5d33700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:59.419 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.416+0000 7f56e5d33700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:59.419 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.416+0000 7f56e5d33700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:59.419 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.416+0000 7f56e5532700 10 monclient(hunting): get_auth_request con 0x7f56e00a5b70 auth_method 0 2023-09-30T02:06:59.419 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.416+0000 7f56e5532700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:59.419 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.416+0000 7f56e5532700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:59.419 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.416+0000 7f56e5532700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:59.421 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.417+0000 7f56e5d33700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:59.421 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.417+0000 7f56e5d33700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:59.421 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.417+0000 7f56e5d33700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:59.421 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.418+0000 7f56e5532700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:59.421 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.418+0000 7f56e5532700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:59.421 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.418+0000 7f56e5532700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:59.421 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.419+0000 7f56e5d33700 10 monclient(hunting): handle_auth_done global_id 4388 payload 274 2023-09-30T02:06:59.421 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.419+0000 7f56e5d33700 1 -- 172.21.15.59:0/3292057539 >> v2:172.21.15.59:3300/0 conn(0x7f56e00a5b70 msgr2=0x7f56e00b1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:06:59.421 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.419+0000 7f56e5d33700 1 --2- 172.21.15.59:0/3292057539 >> v2:172.21.15.59:3300/0 conn(0x7f56e00a5b70 0x7f56e00b1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:59.422 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.419+0000 7f56e5d33700 1 -- 172.21.15.59:0/3292057539 >> v1:172.21.15.59:6789/0 conn(0x7f56e00a5f40 legacy=0x7f56e0141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:59.422 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.419+0000 7f56e5d33700 10 monclient: _finish_hunting 0 2023-09-30T02:06:59.422 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.419+0000 7f56e5d33700 1 monclient: found mon.noname-b 2023-09-30T02:06:59.422 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.419+0000 7f56e5d33700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:59.422 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.419+0000 7f56e5d33700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.59:3301/0 2023-09-30T02:06:59.422 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.419+0000 7f56e5d33700 1 -- 172.21.15.59:0/3292057539 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56e00af3f0 con 0x7f56e0142290 2023-09-30T02:06:59.422 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.422+0000 7f56e5d33700 1 --2- 172.21.15.59:0/3292057539 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f56e0142290 0x7f56e013af30 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f56d8009f20 tx=0x7f56d8009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c8b9b3608c7b99c7 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:59.428 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.427+0000 7f56d7fff700 1 -- 172.21.15.59:0/3292057539 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56d800e070 con 0x7f56e0142290 2023-09-30T02:06:59.428 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.427+0000 7f56d7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:59.428 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.427+0000 7f56d7fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:06:59.428 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.427+0000 7f56d7fff700 10 monclient: dump: 2023-09-30T02:06:59.428 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:59.428 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:59.428 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:59.428 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:59.428 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:59.428 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:59.429 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:59.429 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:59.429 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:59.429 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:59.429 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.427+0000 7f56d7fff700 10 monclient: _finish_auth 0 2023-09-30T02:06:59.429 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.427+0000 7f56d7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:59.429 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.427+0000 7f56d7fff700 1 -- 172.21.15.59:0/3292057539 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56d800b9d0 con 0x7f56e0142290 2023-09-30T02:06:59.429 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.427+0000 7f56e7796700 5 monclient: authenticate success, global_id 4388 2023-09-30T02:06:59.429 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.427+0000 7f56e7796700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:06:59.429 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.427+0000 7f56d7fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:59.429 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.427+0000 7f56e7796700 10 monclient: get_monmap_and_config success 2023-09-30T02:06:59.430 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.427+0000 7f56d7fff700 1 -- 172.21.15.59:0/3292057539 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56d80092e0 con 0x7f56e0142290 2023-09-30T02:06:59.430 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.427+0000 7f56d7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:59.430 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.427+0000 7f56e7796700 4 set_mon_vals no callback set 2023-09-30T02:06:59.430 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.427+0000 7f56d7fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:06:59.430 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.427+0000 7f56d7fff700 10 monclient: dump: 2023-09-30T02:06:59.430 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:59.430 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:59.430 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:59.430 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:59.430 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:59.430 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:59.431 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:59.431 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:59.431 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:59.431 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:59.434 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.428+0000 7f56e7796700 10 monclient: shutdown 2023-09-30T02:06:59.434 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.428+0000 7f56e7796700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:59.434 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.428+0000 7f56e7796700 1 -- 172.21.15.59:0/3292057539 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f56e0142290 msgr2=0x7f56e013af30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:59.434 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.428+0000 7f56e7796700 1 --2- 172.21.15.59:0/3292057539 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f56e0142290 0x7f56e013af30 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f56d8009f20 tx=0x7f56d8009840 comp rx=0 tx=0).stop 2023-09-30T02:06:59.434 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.428+0000 7f56e7796700 1 -- 172.21.15.59:0/3292057539 shutdown_connections 2023-09-30T02:06:59.434 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.428+0000 7f56e7796700 1 --2- 172.21.15.59:0/3292057539 >> v2:172.21.15.59:3300/0 conn(0x7f56e00a5b70 0x7f56e00b1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:59.434 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.428+0000 7f56e7796700 1 --2- 172.21.15.59:0/3292057539 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f56e0142290 0x7f56e013af30 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:59.435 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.428+0000 7f56e7796700 1 -- 172.21.15.59:0/3292057539 >> 172.21.15.59:0/3292057539 conn(0x7f56e00631f0 msgr2=0x7f56e00635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:59.435 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.428+0000 7f56e7796700 1 -- 172.21.15.59:0/3292057539 shutdown_connections 2023-09-30T02:06:59.435 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.428+0000 7f56e7796700 1 -- 172.21.15.59:0/3292057539 wait complete. 2023-09-30T02:06:59.435 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.429+0000 7f56e7796700 10 monclient: build_initial_monmap 2023-09-30T02:06:59.435 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.429+0000 7f56e7796700 10 monclient: monmap: 2023-09-30T02:06:59.435 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:06:59.435 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:06:59.435 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:06:59.435 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:06:59.435 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:06:59.435 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:06:59.435 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:06:59.436 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:06:59.436 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:06:59.436 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:59.436 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.429+0000 7f56e7796700 1 librados: starting msgr at 2023-09-30T02:06:59.436 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.429+0000 7f56e7796700 1 librados: starting objecter 2023-09-30T02:06:59.436 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.429+0000 7f56e7796700 1 Processor -- start 2023-09-30T02:06:59.436 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.429+0000 7f56e7796700 1 -- start start 2023-09-30T02:06:59.436 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.429+0000 7f56e7796700 1 librados: setting wanted keys 2023-09-30T02:06:59.436 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.429+0000 7f56e7796700 1 librados: calling monclient init 2023-09-30T02:06:59.436 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.429+0000 7f56e7796700 10 monclient: init 2023-09-30T02:06:59.436 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.430+0000 7f56e7796700 10 monclient: _reopen_session rank -1 2023-09-30T02:06:59.436 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.430+0000 7f56e7796700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T02:06:59.437 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.430+0000 7f56e7796700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f56e00a5b70 0x7f56e00a8290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:59.437 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.430+0000 7f56e7796700 10 monclient(hunting): picked mon.noname-b con 0x7f56e00a5b70 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:06:59.437 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.430+0000 7f56e7796700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56e00a5f40 0x7f56e014dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:59.437 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.430+0000 7f56e7796700 10 monclient(hunting): picked mon.noname-a con 0x7f56e00a5f40 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:06:59.437 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.430+0000 7f56e7796700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f56e0142290 0x7f56e014e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:59.437 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.430+0000 7f56e7796700 10 monclient(hunting): picked mon.noname-c con 0x7f56e0142290 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:06:59.437 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.430+0000 7f56e7796700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:59.437 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.430+0000 7f56e7796700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56e00a9c60 con 0x7f56e00a5f40 2023-09-30T02:06:59.437 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.430+0000 7f56e7796700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:59.437 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.430+0000 7f56e7796700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56e014e7e0 con 0x7f56e00a5b70 2023-09-30T02:06:59.437 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.430+0000 7f56e7796700 10 monclient(hunting): start opening mon connection 2023-09-30T02:06:59.437 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.430+0000 7f56e7796700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56e013c3f0 con 0x7f56e0142290 2023-09-30T02:06:59.438 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.430+0000 7f56e7796700 10 monclient(hunting): _renew_subs 2023-09-30T02:06:59.438 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.430+0000 7f56e7796700 10 monclient(hunting): authenticate will time out at 1864.733838s 2023-09-30T02:06:59.438 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.431+0000 7f56e5d33700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f56e0142290 0x7f56e014e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:59.438 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.431+0000 7f56e4d31700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56e00a5f40 0x7f56e014dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:59.438 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.431+0000 7f56e5532700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f56e00a5b70 0x7f56e00a8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:59.438 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.431+0000 7f56e5d33700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f56e0142290 0x7f56e014e2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:54438/0 (socket says 172.21.15.59:54438) 2023-09-30T02:06:59.438 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.431+0000 7f56e5d33700 1 -- 172.21.15.59:0/2472711274 learned_addr learned my addr 172.21.15.59:0/2472711274 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:06:59.438 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.431+0000 7f56e5d33700 10 monclient(hunting): get_auth_request con 0x7f56e0142290 auth_method 0 2023-09-30T02:06:59.438 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.431+0000 7f56e5d33700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:59.438 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.431+0000 7f56e5d33700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:59.438 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.431+0000 7f56e5d33700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:59.439 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.431+0000 7f56e4d31700 10 monclient(hunting): get_auth_request con 0x7f56e00a5f40 auth_method 0 2023-09-30T02:06:59.439 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.431+0000 7f56e4d31700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:59.439 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.431+0000 7f56e4d31700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:59.439 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.431+0000 7f56e4d31700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:59.439 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.432+0000 7f56e5532700 10 monclient(hunting): get_auth_request con 0x7f56e00a5b70 auth_method 0 2023-09-30T02:06:59.439 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.432+0000 7f56e5532700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:06:59.439 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.432+0000 7f56e5532700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:06:59.439 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.432+0000 7f56e5532700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:06:59.439 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.432+0000 7f56e5d33700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:59.439 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.432+0000 7f56e5d33700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:59.439 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.432+0000 7f56e5d33700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:59.439 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.432+0000 7f56e4d31700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:59.440 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.432+0000 7f56e4d31700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:59.440 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.432+0000 7f56e4d31700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:59.440 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.432+0000 7f56e5532700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:06:59.440 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.432+0000 7f56e5532700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:06:59.440 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.432+0000 7f56e5532700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:06:59.441 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.436+0000 7f56e5d33700 10 monclient(hunting): handle_auth_done global_id 4391 payload 824 2023-09-30T02:06:59.441 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.436+0000 7f56e5d33700 1 -- 172.21.15.59:0/2472711274 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f56e00a5b70 msgr2=0x7f56e00a8290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:59.441 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.436+0000 7f56e5d33700 1 --2- 172.21.15.59:0/2472711274 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f56e00a5b70 0x7f56e00a8290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:59.441 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.436+0000 7f56e5d33700 1 -- 172.21.15.59:0/2472711274 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56e00a5f40 msgr2=0x7f56e014dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:59.441 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.436+0000 7f56e5d33700 1 --2- 172.21.15.59:0/2472711274 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56e00a5f40 0x7f56e014dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:59.441 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.436+0000 7f56e5d33700 10 monclient: _finish_hunting 0 2023-09-30T02:06:59.441 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.436+0000 7f56e5d33700 1 monclient: found mon.noname-c 2023-09-30T02:06:59.442 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.436+0000 7f56e5d33700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:06:59.442 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.436+0000 7f56e5d33700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.59:3301/0 2023-09-30T02:06:59.442 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.436+0000 7f56e5d33700 1 -- 172.21.15.59:0/2472711274 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56d8009ae0 con 0x7f56e0142290 2023-09-30T02:06:59.442 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.438+0000 7f56e5d33700 1 --2- 172.21.15.59:0/2472711274 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f56e0142290 0x7f56e014e2a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f56d800b610 tx=0x7f56d8018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:59.442 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.438+0000 7f56d57fa700 10 client.?.objecter ms_handle_connect 0x7f56e0142290 2023-09-30T02:06:59.442 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.438+0000 7f56d57fa700 10 client.?.objecter resend_mon_ops 2023-09-30T02:06:59.447 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.443+0000 7f56d57fa700 1 -- 172.21.15.59:0/2472711274 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56d8008b10 con 0x7f56e0142290 2023-09-30T02:06:59.447 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.443+0000 7f56d57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:59.447 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.443+0000 7f56d57fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:06:59.447 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.443+0000 7f56d57fa700 10 monclient: dump: 2023-09-30T02:06:59.447 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:59.447 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:59.447 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:59.447 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:59.448 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:59.448 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:59.448 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:59.448 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:59.448 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:59.448 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:59.448 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.443+0000 7f56d57fa700 10 monclient: _finish_auth 0 2023-09-30T02:06:59.448 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.443+0000 7f56d57fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:06:59.448 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.443+0000 7f56d57fa700 1 -- 172.21.15.59:0/2472711274 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56d8008cb0 con 0x7f56e0142290 2023-09-30T02:06:59.448 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.443+0000 7f56d57fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:06:59.448 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.443+0000 7f56d57fa700 1 -- 172.21.15.59:0/2472711274 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56d80256b0 con 0x7f56e0142290 2023-09-30T02:06:59.448 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.443+0000 7f56e7796700 5 monclient: authenticate success, global_id 4391 2023-09-30T02:06:59.449 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.443+0000 7f56e7796700 10 monclient: _renew_subs 2023-09-30T02:06:59.449 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.443+0000 7f56e7796700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:06:59.449 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.443+0000 7f56e7796700 1 -- 172.21.15.59:0/2472711274 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56e013ed40 con 0x7f56e0142290 2023-09-30T02:06:59.449 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.443+0000 7f56d77fe700 4 set_mon_vals no callback set 2023-09-30T02:06:59.449 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.443+0000 7f56d57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:06:59.449 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.443+0000 7f56d57fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:06:59.449 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.443+0000 7f56d57fa700 10 monclient: dump: 2023-09-30T02:06:59.449 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:06:59.449 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:06:59.449 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:59.449 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:06:59.449 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:06:59.450 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:06:59.450 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:06:59.450 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:06:59.450 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:06:59.450 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:06:59.450 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.443+0000 7f56e7796700 10 client.4391.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:06:59.450 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.443+0000 7f56e7796700 10 monclient: _renew_subs 2023-09-30T02:06:59.450 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.443+0000 7f56e7796700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:06:59.450 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.443+0000 7f56e7796700 1 -- 172.21.15.59:0/2472711274 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56e013e100 con 0x7f56e0142290 2023-09-30T02:06:59.450 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.443+0000 7f56e7796700 1 librados: init done 2023-09-30T02:06:59.450 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.444+0000 7f56e7796700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:59.450 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.444+0000 7f56e7796700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:59.451 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.444+0000 7f56e7796700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:06:59.451 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.444+0000 7f56e7796700 1 -- 172.21.15.59:0/2472711274 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56e0064a00 con 0x7f56e0142290 2023-09-30T02:06:59.673 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.672+0000 7f56d57fa700 1 -- 172.21.15.59:0/2472711274 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f56d8003040 con 0x7f56e0142290 2023-09-30T02:06:59.673 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.673+0000 7f56d57fa700 1 --2- 172.21.15.59:0/2472711274 >> v2:172.21.15.110:6800/115870 conn(0x7f56c4068960 0x7f56c406ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:06:59.673 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.673+0000 7f56e5532700 1 --2- 172.21.15.59:0/2472711274 >> v2:172.21.15.110:6800/115870 conn(0x7f56c4068960 0x7f56c406ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:06:59.673 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.673+0000 7f56e5532700 10 monclient: get_auth_request con 0x7f56c4068960 auth_method 0 2023-09-30T02:06:59.674 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.673+0000 7f56d57fa700 10 client.4391.objecter ms_handle_connect 0x7f56c4068960 2023-09-30T02:06:59.674 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.673+0000 7f56e5532700 1 --2- 172.21.15.59:0/2472711274 >> v2:172.21.15.110:6800/115870 conn(0x7f56c4068960 0x7f56c406ae10 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f56dc009db0 tx=0x7f56dc0091a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:06:59.742 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.742+0000 7f56d57fa700 1 -- 172.21.15.59:0/2472711274 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f56d8057460 con 0x7f56e0142290 2023-09-30T02:06:59.743 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.742+0000 7f56d57fa700 10 client.4391.objecter ms_dispatch 0x7f56e0001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:06:59.743 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.742+0000 7f56d57fa700 3 client.4391.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:06:59.743 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.742+0000 7f56d57fa700 3 client.4391.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:06:59.743 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.742+0000 7f56d57fa700 20 client.4391.objecter dump_active .. 0 homeless 2023-09-30T02:06:59.743 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.742+0000 7f56d57fa700 1 -- 172.21.15.59:0/2472711274 <== mon.2 v2:172.21.15.59:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f56d80537a0 con 0x7f56e0142290 2023-09-30T02:06:59.743 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.742+0000 7f56d57fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:06:59.743 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.742+0000 7f56d57fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:06:59.990 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.989+0000 7f56e7796700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:06:59.990 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.989+0000 7f56e7796700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:06:59.990 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.989+0000 7f56e7796700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:06:59.990 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.989+0000 7f56e7796700 1 -- 172.21.15.59:0/2472711274 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f56e006a720 con 0x7f56e0142290 2023-09-30T02:06:59.994 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.993+0000 7f56d57fa700 1 -- 172.21.15.59:0/2472711274 <== mon.2 v2:172.21.15.59:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f56d8020100 con 0x7f56e0142290 2023-09-30T02:06:59.994 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.993+0000 7f56d57fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:06:59.994 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.993+0000 7f56d57fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:06:59.994 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:06:59.994 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:06:59.997 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.996+0000 7f56e7796700 10 librados: watch_flush enter 2023-09-30T02:06:59.997 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.996+0000 7f56e7796700 10 librados: watch_flush exit 2023-09-30T02:06:59.997 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.996+0000 7f56e7796700 20 client.4391.objecter shutdown clearing up homeless session... 2023-09-30T02:06:59.997 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.996+0000 7f56e7796700 10 client.4391.objecter successfully canceled tick 2023-09-30T02:06:59.998 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.996+0000 7f56e7796700 1 -- 172.21.15.59:0/2472711274 >> v2:172.21.15.110:6800/115870 conn(0x7f56c4068960 msgr2=0x7f56c406ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:59.998 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.996+0000 7f56e7796700 1 --2- 172.21.15.59:0/2472711274 >> v2:172.21.15.110:6800/115870 conn(0x7f56c4068960 0x7f56c406ae10 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f56dc009db0 tx=0x7f56dc0091a0 comp rx=0 tx=0).stop 2023-09-30T02:06:59.998 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.996+0000 7f56e7796700 10 monclient: shutdown 2023-09-30T02:06:59.998 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.996+0000 7f56e7796700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:06:59.998 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.996+0000 7f56e7796700 1 -- 172.21.15.59:0/2472711274 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f56e0142290 msgr2=0x7f56e014e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:06:59.998 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.996+0000 7f56e7796700 1 --2- 172.21.15.59:0/2472711274 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f56e0142290 0x7f56e014e2a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f56d800b610 tx=0x7f56d8018000 comp rx=0 tx=0).stop 2023-09-30T02:06:59.998 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.996+0000 7f56e7796700 1 -- 172.21.15.59:0/2472711274 shutdown_connections 2023-09-30T02:06:59.998 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.996+0000 7f56e7796700 1 --2- 172.21.15.59:0/2472711274 >> v2:172.21.15.110:6800/115870 conn(0x7f56c4068960 0x7f56c406ae10 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:59.998 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.996+0000 7f56e7796700 1 --2- 172.21.15.59:0/2472711274 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f56e00a5b70 0x7f56e00a8290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:59.998 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.996+0000 7f56e7796700 1 --2- 172.21.15.59:0/2472711274 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f56e0142290 0x7f56e014e2a0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:59.999 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.996+0000 7f56e7796700 1 --2- 172.21.15.59:0/2472711274 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f56e00a5f40 0x7f56e014dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:06:59.999 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.996+0000 7f56e7796700 1 -- 172.21.15.59:0/2472711274 >> 172.21.15.59:0/2472711274 conn(0x7f56e00631f0 msgr2=0x7f56e0141b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:06:59.999 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.996+0000 7f56e7796700 1 -- 172.21.15.59:0/2472711274 shutdown_connections 2023-09-30T02:06:59.999 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.996+0000 7f56e7796700 1 -- 172.21.15.59:0/2472711274 wait complete. 2023-09-30T02:06:59.999 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:06:59.996+0000 7f56e7796700 1 librados: shutdown 2023-09-30T02:07:03.012 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:07:03.154 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.152+0000 7f5947b60700 10 monclient: get_monmap_and_config 2023-09-30T02:07:03.154 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.152+0000 7f5947b60700 10 monclient: build_initial_monmap 2023-09-30T02:07:03.154 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.152+0000 7f5947b60700 10 monclient: monmap: 2023-09-30T02:07:03.154 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:07:03.154 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:03.154 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:07:03.154174+0000 2023-09-30T02:07:03.154 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:07:03.154174+0000 2023-09-30T02:07:03.154 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:07:03.155 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:03.155 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:07:03.155 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:07:03.155 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:07:03.155 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:07:03.155 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:07:03.155 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:03.155 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.153+0000 7f5947b60700 1 Processor -- start 2023-09-30T02:07:03.155 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.153+0000 7f5947b60700 1 -- start start 2023-09-30T02:07:03.155 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.153+0000 7f5947b60700 10 monclient: init 2023-09-30T02:07:03.157 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.154+0000 7f5947b60700 10 monclient: _reopen_session rank -1 2023-09-30T02:07:03.157 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.154+0000 7f5947b60700 10 monclient: _add_conns ranks=[0,2,4,1,3] 2023-09-30T02:07:03.157 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.154+0000 7f5947b60700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f59400a5b70 addr v1:172.21.15.59:6789/0 2023-09-30T02:07:03.157 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.154+0000 7f5947b60700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5940142290 0x7f59400b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:03.157 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.154+0000 7f5947b60700 10 monclient(hunting): picked mon.noname-b con 0x7f5940142290 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:07:03.157 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.154+0000 7f5947b60700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f59400a5f40 0x7f594013af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:03.158 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.154+0000 7f5947b60700 10 monclient(hunting): picked mon.noname-c con 0x7f59400a5f40 addr v2:172.21.15.110:3300/0 2023-09-30T02:07:03.158 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.154+0000 7f5947b60700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59400a6380 con 0x7f59400a5b70 2023-09-30T02:07:03.158 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.154+0000 7f5947b60700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:03.158 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.154+0000 7f5947b60700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59400a75a0 con 0x7f59400a5f40 2023-09-30T02:07:03.158 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.154+0000 7f5947b60700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:03.158 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.154+0000 7f5947b60700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59400af3f0 con 0x7f5940142290 2023-09-30T02:07:03.158 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.154+0000 7f5947b60700 10 monclient(hunting): _renew_subs 2023-09-30T02:07:03.158 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.154+0000 7f5947b60700 10 monclient(hunting): authenticate will time out at 1868.457767s 2023-09-30T02:07:03.158 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.155+0000 7f59460fd700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f59400a5f40 0x7f594013af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:03.158 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.155+0000 7f59450fb700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5940142290 0x7f59400b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:03.158 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.155+0000 7f59460fd700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f59400a5f40 0x7f594013af30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:51648/0 (socket says 172.21.15.59:51648) 2023-09-30T02:07:03.158 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.155+0000 7f59460fd700 1 -- 172.21.15.59:0/2837725513 learned_addr learned my addr 172.21.15.59:0/2837725513 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:07:03.159 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.155+0000 7f59460fd700 10 monclient(hunting): get_auth_request con 0x7f59400a5f40 auth_method 0 2023-09-30T02:07:03.159 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.155+0000 7f59460fd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:07:03.159 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.155+0000 7f59460fd700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:07:03.159 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.155+0000 7f59460fd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:07:03.159 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.156+0000 7f59450fb700 10 monclient(hunting): get_auth_request con 0x7f5940142290 auth_method 0 2023-09-30T02:07:03.159 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.156+0000 7f59450fb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:07:03.159 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.156+0000 7f59450fb700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:07:03.159 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.156+0000 7f59450fb700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:07:03.159 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.156+0000 7f59460fd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:07:03.159 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.156+0000 7f59460fd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:07:03.159 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.156+0000 7f59460fd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:07:03.159 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.156+0000 7f59450fb700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:07:03.159 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.156+0000 7f59450fb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:07:03.160 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.156+0000 7f59450fb700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:07:03.160 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.159+0000 7f59460fd700 10 monclient(hunting): handle_auth_done global_id 4447 payload 274 2023-09-30T02:07:03.160 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.159+0000 7f59460fd700 1 -- 172.21.15.59:0/2837725513 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5940142290 msgr2=0x7f59400b1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:07:03.160 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.159+0000 7f59460fd700 1 --2- 172.21.15.59:0/2837725513 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5940142290 0x7f59400b1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:03.160 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.159+0000 7f59460fd700 1 -- 172.21.15.59:0/2837725513 >> v1:172.21.15.59:6789/0 conn(0x7f59400a5b70 legacy=0x7f5940141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:07:03.160 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.159+0000 7f59460fd700 10 monclient: _finish_hunting 0 2023-09-30T02:07:03.161 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.159+0000 7f59460fd700 1 monclient: found mon.noname-c 2023-09-30T02:07:03.161 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.159+0000 7f59460fd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:07:03.161 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.159+0000 7f59460fd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3300/0 2023-09-30T02:07:03.161 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.159+0000 7f59460fd700 1 -- 172.21.15.59:0/2837725513 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f594013b470 con 0x7f59400a5f40 2023-09-30T02:07:03.161 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.161+0000 7f59460fd700 1 --2- 172.21.15.59:0/2837725513 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f59400a5f40 0x7f594013af30 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f59380099b0 tx=0x7f5938009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=24b5ef69b61ad9f3 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:07:03.167 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.166+0000 7f59448fa700 1 -- 172.21.15.59:0/2837725513 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5938009ea0 con 0x7f59400a5f40 2023-09-30T02:07:03.167 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.166+0000 7f59448fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:07:03.167 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.166+0000 7f59448fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:07:03.167 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.166+0000 7f59448fa700 10 monclient: dump: 2023-09-30T02:07:03.167 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:07:03.167 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:03.167 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:03.167 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:03.167 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:07:03.168 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:03.168 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:07:03.168 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:07:03.168 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:07:03.168 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:03.168 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.166+0000 7f59448fa700 10 monclient: _finish_auth 0 2023-09-30T02:07:03.168 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.166+0000 7f59448fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:07:03.168 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.166+0000 7f59448fa700 1 -- 172.21.15.59:0/2837725513 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f593800b820 con 0x7f59400a5f40 2023-09-30T02:07:03.168 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.166+0000 7f5947b60700 5 monclient: authenticate success, global_id 4447 2023-09-30T02:07:03.168 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.166+0000 7f5947b60700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:07:03.168 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.166+0000 7f59448fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:07:03.168 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.166+0000 7f5947b60700 10 monclient: get_monmap_and_config success 2023-09-30T02:07:03.168 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.166+0000 7f5947b60700 4 set_mon_vals no callback set 2023-09-30T02:07:03.169 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.166+0000 7f59448fa700 1 -- 172.21.15.59:0/2837725513 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f593800bbc0 con 0x7f59400a5f40 2023-09-30T02:07:03.169 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.166+0000 7f59448fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:07:03.169 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.166+0000 7f59448fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:07:03.169 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.166+0000 7f59448fa700 10 monclient: dump: 2023-09-30T02:07:03.169 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:07:03.169 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:03.169 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:03.169 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:03.169 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:07:03.169 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:03.169 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:07:03.169 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:07:03.169 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:07:03.170 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:03.170 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.166+0000 7f5947b60700 10 monclient: shutdown 2023-09-30T02:07:03.170 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.166+0000 7f5947b60700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:07:03.170 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.166+0000 7f5947b60700 1 -- 172.21.15.59:0/2837725513 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f59400a5f40 msgr2=0x7f594013af30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:03.170 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.166+0000 7f5947b60700 1 --2- 172.21.15.59:0/2837725513 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f59400a5f40 0x7f594013af30 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f59380099b0 tx=0x7f5938009250 comp rx=0 tx=0).stop 2023-09-30T02:07:03.170 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.166+0000 7f5947b60700 1 -- 172.21.15.59:0/2837725513 shutdown_connections 2023-09-30T02:07:03.170 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.166+0000 7f5947b60700 1 --2- 172.21.15.59:0/2837725513 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f59400a5f40 0x7f594013af30 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:03.170 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.166+0000 7f5947b60700 1 --2- 172.21.15.59:0/2837725513 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5940142290 0x7f59400b1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:03.170 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.166+0000 7f5947b60700 1 -- 172.21.15.59:0/2837725513 >> 172.21.15.59:0/2837725513 conn(0x7f59400631f0 msgr2=0x7f59400635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:07:03.170 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.167+0000 7f5947b60700 1 -- 172.21.15.59:0/2837725513 shutdown_connections 2023-09-30T02:07:03.170 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.167+0000 7f5947b60700 1 -- 172.21.15.59:0/2837725513 wait complete. 2023-09-30T02:07:03.172 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.167+0000 7f5947b60700 10 monclient: build_initial_monmap 2023-09-30T02:07:03.172 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.167+0000 7f5947b60700 10 monclient: monmap: 2023-09-30T02:07:03.172 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:07:03.172 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:07:03.173 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:07:03.173 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:07:03.173 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:07:03.173 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:07:03.173 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:07:03.173 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:07:03.173 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:07:03.173 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:03.173 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.167+0000 7f5947b60700 1 librados: starting msgr at 2023-09-30T02:07:03.173 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.167+0000 7f5947b60700 1 librados: starting objecter 2023-09-30T02:07:03.173 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.168+0000 7f5947b60700 1 Processor -- start 2023-09-30T02:07:03.173 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.168+0000 7f5947b60700 1 -- start start 2023-09-30T02:07:03.173 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.168+0000 7f5947b60700 1 librados: setting wanted keys 2023-09-30T02:07:03.174 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.168+0000 7f5947b60700 1 librados: calling monclient init 2023-09-30T02:07:03.174 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.168+0000 7f5947b60700 10 monclient: init 2023-09-30T02:07:03.174 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.168+0000 7f5947b60700 10 monclient: _reopen_session rank -1 2023-09-30T02:07:03.174 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.168+0000 7f5947b60700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T02:07:03.174 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.168+0000 7f5947b60700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f59400a5b70 0x7f59400a78a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:03.174 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.168+0000 7f5947b60700 10 monclient(hunting): picked mon.noname-a con 0x7f59400a5b70 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:07:03.174 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.168+0000 7f5947b60700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f59400a5f40 0x7f594013c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:03.174 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.168+0000 7f5947b60700 10 monclient(hunting): picked mon.noname-b con 0x7f59400a5f40 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:07:03.174 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.168+0000 7f5947b60700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5940142290 0x7f594013c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:03.174 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.168+0000 7f5947b60700 10 monclient(hunting): picked mon.noname-c con 0x7f5940142290 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:07:03.174 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.168+0000 7f5947b60700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:03.175 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.168+0000 7f5947b60700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59400a9170 con 0x7f59400a5b70 2023-09-30T02:07:03.175 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.168+0000 7f5947b60700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:03.175 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.168+0000 7f5947b60700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f594013f020 con 0x7f59400a5f40 2023-09-30T02:07:03.175 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.168+0000 7f5947b60700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:03.175 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.168+0000 7f5947b60700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59400a9450 con 0x7f5940142290 2023-09-30T02:07:03.175 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.168+0000 7f5947b60700 10 monclient(hunting): _renew_subs 2023-09-30T02:07:03.175 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.168+0000 7f5947b60700 10 monclient(hunting): authenticate will time out at 1868.471687s 2023-09-30T02:07:03.175 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.169+0000 7f59460fd700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5940142290 0x7f594013c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:03.175 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.170+0000 7f59460fd700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5940142290 0x7f594013c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:54460/0 (socket says 172.21.15.59:54460) 2023-09-30T02:07:03.175 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.170+0000 7f59460fd700 1 -- 172.21.15.59:0/611206025 learned_addr learned my addr 172.21.15.59:0/611206025 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:07:03.175 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.170+0000 7f59460fd700 10 monclient(hunting): get_auth_request con 0x7f5940142290 auth_method 0 2023-09-30T02:07:03.175 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.170+0000 7f59460fd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:07:03.175 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.170+0000 7f59460fd700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:07:03.175 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.170+0000 7f59460fd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:07:03.176 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.170+0000 7f59460fd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:07:03.176 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.170+0000 7f59460fd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:07:03.176 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.170+0000 7f59460fd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:07:03.176 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.170+0000 7f59450fb700 1 --2- 172.21.15.59:0/611206025 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f59400a5f40 0x7f594013c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:03.176 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.171+0000 7f59450fb700 10 monclient(hunting): get_auth_request con 0x7f59400a5f40 auth_method 0 2023-09-30T02:07:03.176 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.171+0000 7f59450fb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:07:03.176 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.171+0000 7f59450fb700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:07:03.176 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.171+0000 7f59450fb700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:07:03.176 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.171+0000 7f59450fb700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:07:03.176 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.171+0000 7f59450fb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:07:03.176 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.171+0000 7f59450fb700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:07:03.177 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.174+0000 7f59460fd700 10 monclient(hunting): handle_auth_done global_id 4397 payload 824 2023-09-30T02:07:03.177 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.174+0000 7f59460fd700 1 -- 172.21.15.59:0/611206025 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f59400a5f40 msgr2=0x7f594013c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:03.177 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.174+0000 7f59460fd700 1 --2- 172.21.15.59:0/611206025 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f59400a5f40 0x7f594013c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:03.177 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.174+0000 7f59460fd700 1 -- 172.21.15.59:0/611206025 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f59400a5b70 msgr2=0x7f59400a78a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:03.177 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.174+0000 7f59460fd700 1 --2- 172.21.15.59:0/611206025 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f59400a5b70 0x7f59400a78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:03.177 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.174+0000 7f59460fd700 10 monclient: _finish_hunting 0 2023-09-30T02:07:03.177 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.174+0000 7f59460fd700 1 monclient: found mon.noname-c 2023-09-30T02:07:03.177 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.174+0000 7f59460fd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:07:03.177 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.174+0000 7f59460fd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.59:3301/0 2023-09-30T02:07:03.178 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.174+0000 7f59460fd700 1 -- 172.21.15.59:0/611206025 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59380094f0 con 0x7f5940142290 2023-09-30T02:07:03.178 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.178+0000 7f59460fd700 1 --2- 172.21.15.59:0/611206025 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5940142290 0x7f594013c570 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f59380094c0 tx=0x7f5938016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:07:03.178 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.178+0000 7f593dffb700 10 client.?.objecter ms_handle_connect 0x7f5940142290 2023-09-30T02:07:03.178 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.178+0000 7f593dffb700 10 client.?.objecter resend_mon_ops 2023-09-30T02:07:03.184 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.183+0000 7f593dffb700 1 -- 172.21.15.59:0/611206025 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f593801b470 con 0x7f5940142290 2023-09-30T02:07:03.184 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.183+0000 7f593dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:07:03.184 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.183+0000 7f593dffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:07:03.184 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.183+0000 7f593dffb700 10 monclient: dump: 2023-09-30T02:07:03.184 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:07:03.184 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:03.184 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:03.184 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:03.185 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:07:03.185 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:03.185 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:07:03.185 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:07:03.185 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:07:03.185 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:03.185 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.183+0000 7f593dffb700 10 monclient: _finish_auth 0 2023-09-30T02:07:03.185 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.183+0000 7f593dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:07:03.185 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.183+0000 7f593dffb700 1 -- 172.21.15.59:0/611206025 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f593801b5d0 con 0x7f5940142290 2023-09-30T02:07:03.185 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.183+0000 7f5947b60700 5 monclient: authenticate success, global_id 4397 2023-09-30T02:07:03.185 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.183+0000 7f5947b60700 10 monclient: _renew_subs 2023-09-30T02:07:03.185 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.183+0000 7f5947b60700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:07:03.185 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.183+0000 7f5947b60700 1 -- 172.21.15.59:0/611206025 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f594013e820 con 0x7f5940142290 2023-09-30T02:07:03.185 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.183+0000 7f593dffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:07:03.186 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.183+0000 7f593dffb700 1 -- 172.21.15.59:0/611206025 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f593801b970 con 0x7f5940142290 2023-09-30T02:07:03.186 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.183+0000 7f593dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:07:03.186 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.183+0000 7f593dffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:07:03.186 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.183+0000 7f593dffb700 10 monclient: dump: 2023-09-30T02:07:03.186 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:07:03.186 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:03.186 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:03.186 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:03.186 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:07:03.186 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:03.186 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:07:03.186 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:07:03.187 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:07:03.187 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:03.187 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.183+0000 7f593ffff700 4 set_mon_vals no callback set 2023-09-30T02:07:03.187 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.183+0000 7f5947b60700 10 client.4397.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:07:03.187 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.183+0000 7f5947b60700 10 monclient: _renew_subs 2023-09-30T02:07:03.187 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.183+0000 7f5947b60700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:07:03.187 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.183+0000 7f5947b60700 1 -- 172.21.15.59:0/611206025 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59400a7de0 con 0x7f5940142290 2023-09-30T02:07:03.187 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.183+0000 7f5947b60700 1 librados: init done 2023-09-30T02:07:03.188 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.185+0000 7f5947b60700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:07:03.188 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.185+0000 7f5947b60700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:07:03.188 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.185+0000 7f5947b60700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:07:03.188 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.185+0000 7f5947b60700 1 -- 172.21.15.59:0/611206025 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5940064a00 con 0x7f5940142290 2023-09-30T02:07:03.491 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.490+0000 7f593dffb700 1 -- 172.21.15.59:0/611206025 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f593801bd20 con 0x7f5940142290 2023-09-30T02:07:03.491 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.490+0000 7f593dffb700 1 --2- 172.21.15.59:0/611206025 >> v2:172.21.15.110:6800/115870 conn(0x7f592c068960 0x7f592c06ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:03.492 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.490+0000 7f593dffb700 1 -- 172.21.15.59:0/611206025 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f5938085fa0 con 0x7f5940142290 2023-09-30T02:07:03.492 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.490+0000 7f593dffb700 10 client.4397.objecter ms_dispatch 0x7f5940001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:07:03.492 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.490+0000 7f593dffb700 3 client.4397.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:07:03.492 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.490+0000 7f593dffb700 3 client.4397.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:07:03.492 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.491+0000 7f593dffb700 20 client.4397.objecter dump_active .. 0 homeless 2023-09-30T02:07:03.492 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.491+0000 7f593dffb700 1 -- 172.21.15.59:0/611206025 <== mon.2 v2:172.21.15.59:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f59380863f0 con 0x7f5940142290 2023-09-30T02:07:03.492 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.491+0000 7f593dffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:07:03.492 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.491+0000 7f593dffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:07:03.492 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.491+0000 7f59458fc700 1 --2- 172.21.15.59:0/611206025 >> v2:172.21.15.110:6800/115870 conn(0x7f592c068960 0x7f592c06ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:03.492 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.491+0000 7f59458fc700 10 monclient: get_auth_request con 0x7f592c068960 auth_method 0 2023-09-30T02:07:03.492 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.491+0000 7f593dffb700 10 client.4397.objecter ms_handle_connect 0x7f592c068960 2023-09-30T02:07:03.492 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.491+0000 7f59458fc700 1 --2- 172.21.15.59:0/611206025 >> v2:172.21.15.110:6800/115870 conn(0x7f592c068960 0x7f592c06ae10 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f593000ad10 tx=0x7f5930009370 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:07:03.771 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.771+0000 7f5947b60700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:07:03.771 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.771+0000 7f5947b60700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:07:03.771 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.771+0000 7f5947b60700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:07:03.772 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.771+0000 7f5947b60700 1 -- 172.21.15.59:0/611206025 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f5940142050 con 0x7f5940142290 2023-09-30T02:07:03.777 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.776+0000 7f593dffb700 1 -- 172.21.15.59:0/611206025 <== mon.2 v2:172.21.15.59:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f593802a100 con 0x7f5940142290 2023-09-30T02:07:03.777 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.776+0000 7f593dffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:07:03.777 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.776+0000 7f593dffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:07:03.777 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:07:03.777 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:07:03.780 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.779+0000 7f5947b60700 10 librados: watch_flush enter 2023-09-30T02:07:03.780 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.779+0000 7f5947b60700 10 librados: watch_flush exit 2023-09-30T02:07:03.780 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.779+0000 7f5947b60700 20 client.4397.objecter shutdown clearing up homeless session... 2023-09-30T02:07:03.780 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.779+0000 7f5947b60700 10 client.4397.objecter successfully canceled tick 2023-09-30T02:07:03.780 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.779+0000 7f5947b60700 1 -- 172.21.15.59:0/611206025 >> v2:172.21.15.110:6800/115870 conn(0x7f592c068960 msgr2=0x7f592c06ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:03.780 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.779+0000 7f5947b60700 1 --2- 172.21.15.59:0/611206025 >> v2:172.21.15.110:6800/115870 conn(0x7f592c068960 0x7f592c06ae10 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f593000ad10 tx=0x7f5930009370 comp rx=0 tx=0).stop 2023-09-30T02:07:03.780 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.779+0000 7f5947b60700 10 monclient: shutdown 2023-09-30T02:07:03.780 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.779+0000 7f5947b60700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:07:03.780 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.779+0000 7f5947b60700 1 -- 172.21.15.59:0/611206025 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5940142290 msgr2=0x7f594013c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:03.780 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.779+0000 7f5947b60700 1 --2- 172.21.15.59:0/611206025 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5940142290 0x7f594013c570 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f59380094c0 tx=0x7f5938016000 comp rx=0 tx=0).stop 2023-09-30T02:07:03.780 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.779+0000 7f5947b60700 1 -- 172.21.15.59:0/611206025 shutdown_connections 2023-09-30T02:07:03.781 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.779+0000 7f5947b60700 1 --2- 172.21.15.59:0/611206025 >> v2:172.21.15.110:6800/115870 conn(0x7f592c068960 0x7f592c06ae10 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:03.781 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.779+0000 7f5947b60700 1 --2- 172.21.15.59:0/611206025 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5940142290 0x7f594013c570 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:03.781 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.779+0000 7f5947b60700 1 --2- 172.21.15.59:0/611206025 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f59400a5b70 0x7f59400a78a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:03.781 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.779+0000 7f5947b60700 1 --2- 172.21.15.59:0/611206025 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f59400a5f40 0x7f594013c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:03.781 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.779+0000 7f5947b60700 1 -- 172.21.15.59:0/611206025 >> 172.21.15.59:0/611206025 conn(0x7f59400631f0 msgr2=0x7f59400b3320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:07:03.781 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.779+0000 7f5947b60700 1 -- 172.21.15.59:0/611206025 shutdown_connections 2023-09-30T02:07:03.781 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.779+0000 7f5947b60700 1 -- 172.21.15.59:0/611206025 wait complete. 2023-09-30T02:07:03.781 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:03.779+0000 7f5947b60700 1 librados: shutdown 2023-09-30T02:07:06.795 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:07:06.930 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.930+0000 7fb077d90700 10 monclient: get_monmap_and_config 2023-09-30T02:07:06.930 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.930+0000 7fb077d90700 10 monclient: build_initial_monmap 2023-09-30T02:07:06.931 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.930+0000 7fb077d90700 10 monclient: monmap: 2023-09-30T02:07:06.931 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:07:06.931 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:06.931 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:07:06.931508+0000 2023-09-30T02:07:06.931 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:07:06.931508+0000 2023-09-30T02:07:06.931 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:07:06.931 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:06.931 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:07:06.931 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:07:06.931 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:07:06.931 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:07:06.931 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:07:06.932 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:06.932 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.930+0000 7fb077d90700 1 Processor -- start 2023-09-30T02:07:06.934 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.931+0000 7fb077d90700 1 -- start start 2023-09-30T02:07:06.934 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.931+0000 7fb077d90700 10 monclient: init 2023-09-30T02:07:06.934 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.931+0000 7fb077d90700 10 monclient: _reopen_session rank -1 2023-09-30T02:07:06.934 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.931+0000 7fb077d90700 10 monclient: _add_conns ranks=[2,4,3,1,0] 2023-09-30T02:07:06.934 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.931+0000 7fb077d90700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb0700a5b70 0x7fb0700b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:06.934 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.931+0000 7fb077d90700 10 monclient(hunting): picked mon.noname-b con 0x7fb0700a5b70 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:07:06.934 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.931+0000 7fb077d90700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fb0700a5f40 0x7fb070141b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:06.934 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.931+0000 7fb077d90700 10 monclient(hunting): picked mon.noname-c con 0x7fb0700a5f40 addr v2:172.21.15.110:3300/0 2023-09-30T02:07:06.934 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.931+0000 7fb077d90700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fb0701420c0 0x7fb07013e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:06.934 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.931+0000 7fb077d90700 10 monclient(hunting): picked mon.noname-a con 0x7fb0701420c0 addr v2:172.21.15.59:3300/0 2023-09-30T02:07:06.935 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.931+0000 7fb077d90700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:06.935 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.931+0000 7fb077d90700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0700a63b0 con 0x7fb0701420c0 2023-09-30T02:07:06.935 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.931+0000 7fb077d90700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:06.935 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.931+0000 7fb077d90700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0700a75a0 con 0x7fb0700a5f40 2023-09-30T02:07:06.935 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.931+0000 7fb077d90700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:06.935 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.931+0000 7fb077d90700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0700af3f0 con 0x7fb0700a5b70 2023-09-30T02:07:06.935 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.931+0000 7fb077d90700 10 monclient(hunting): _renew_subs 2023-09-30T02:07:06.935 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.931+0000 7fb077d90700 10 monclient(hunting): authenticate will time out at 1872.234693s 2023-09-30T02:07:06.935 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.932+0000 7fb07532b700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fb0700a5f40 0x7fb070141b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:06.935 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.932+0000 7fb07632d700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fb0701420c0 0x7fb07013e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:06.935 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.932+0000 7fb075b2c700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb0700a5b70 0x7fb0700b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:06.935 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.932+0000 7fb07532b700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fb0700a5f40 0x7fb070141b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:51664/0 (socket says 172.21.15.59:51664) 2023-09-30T02:07:06.935 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.932+0000 7fb07532b700 1 -- 172.21.15.59:0/3316485285 learned_addr learned my addr 172.21.15.59:0/3316485285 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:07:06.936 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.932+0000 7fb07532b700 10 monclient(hunting): get_auth_request con 0x7fb0700a5f40 auth_method 0 2023-09-30T02:07:06.936 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.932+0000 7fb07532b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:07:06.936 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.932+0000 7fb07532b700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:07:06.936 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.932+0000 7fb07532b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:07:06.936 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.932+0000 7fb075b2c700 10 monclient(hunting): get_auth_request con 0x7fb0700a5b70 auth_method 0 2023-09-30T02:07:06.936 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.932+0000 7fb075b2c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:07:06.936 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.932+0000 7fb075b2c700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:07:06.936 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.932+0000 7fb075b2c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:07:06.936 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.932+0000 7fb07632d700 10 monclient(hunting): get_auth_request con 0x7fb0701420c0 auth_method 0 2023-09-30T02:07:06.936 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.932+0000 7fb07632d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:07:06.936 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.932+0000 7fb07632d700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:07:06.936 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.932+0000 7fb07632d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:07:06.936 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.933+0000 7fb07532b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:07:06.936 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.933+0000 7fb07532b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:07:06.937 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.933+0000 7fb07532b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:07:06.937 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.933+0000 7fb075b2c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:07:06.937 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.933+0000 7fb075b2c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:07:06.937 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.933+0000 7fb075b2c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:07:06.937 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.933+0000 7fb07632d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:07:06.937 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.933+0000 7fb07632d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:07:06.937 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.933+0000 7fb07632d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:07:06.938 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.936+0000 7fb075b2c700 10 monclient(hunting): handle_auth_done global_id 4400 payload 274 2023-09-30T02:07:06.938 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.936+0000 7fb075b2c700 1 -- 172.21.15.59:0/3316485285 >> v2:172.21.15.110:3300/0 conn(0x7fb0700a5f40 msgr2=0x7fb070141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:07:06.938 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.936+0000 7fb075b2c700 1 --2- 172.21.15.59:0/3316485285 >> v2:172.21.15.110:3300/0 conn(0x7fb0700a5f40 0x7fb070141b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:06.938 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.936+0000 7fb075b2c700 1 -- 172.21.15.59:0/3316485285 >> v2:172.21.15.59:3300/0 conn(0x7fb0701420c0 msgr2=0x7fb07013e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:07:06.938 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.936+0000 7fb075b2c700 1 --2- 172.21.15.59:0/3316485285 >> v2:172.21.15.59:3300/0 conn(0x7fb0701420c0 0x7fb07013e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:06.938 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.936+0000 7fb075b2c700 10 monclient: _finish_hunting 0 2023-09-30T02:07:06.938 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.936+0000 7fb075b2c700 1 monclient: found mon.noname-b 2023-09-30T02:07:06.938 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.936+0000 7fb075b2c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:07:06.938 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.936+0000 7fb075b2c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.59:3301/0 2023-09-30T02:07:06.938 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.936+0000 7fb075b2c700 1 -- 172.21.15.59:0/3316485285 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb070142500 con 0x7fb0700a5b70 2023-09-30T02:07:06.939 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.938+0000 7fb075b2c700 1 --2- 172.21.15.59:0/3316485285 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb0700a5b70 0x7fb0700b1500 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb070069af0 tx=0x7fb06c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3a9539c94be0637 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:07:06.943 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.942+0000 7fb074b2a700 1 -- 172.21.15.59:0/3316485285 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb06c011630 con 0x7fb0700a5b70 2023-09-30T02:07:06.943 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.942+0000 7fb074b2a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:07:06.943 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.942+0000 7fb074b2a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:07:06.943 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.942+0000 7fb074b2a700 10 monclient: dump: 2023-09-30T02:07:06.943 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:07:06.943 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:06.943 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:06.943 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:06.943 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:07:06.943 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:06.943 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:07:06.943 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:07:06.944 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:07:06.944 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:06.944 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.942+0000 7fb074b2a700 10 monclient: _finish_auth 0 2023-09-30T02:07:06.944 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.942+0000 7fb074b2a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:07:06.944 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.942+0000 7fb074b2a700 1 -- 172.21.15.59:0/3316485285 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb06c0117b0 con 0x7fb0700a5b70 2023-09-30T02:07:06.944 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.942+0000 7fb074b2a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:07:06.944 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.942+0000 7fb074b2a700 1 -- 172.21.15.59:0/3316485285 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb06c011b70 con 0x7fb0700a5b70 2023-09-30T02:07:06.944 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.942+0000 7fb074b2a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:07:06.944 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.942+0000 7fb074b2a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:07:06.944 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.942+0000 7fb074b2a700 10 monclient: dump: 2023-09-30T02:07:06.944 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:07:06.944 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:06.944 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:06.945 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:06.945 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:07:06.945 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:06.945 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:07:06.945 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:07:06.945 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:07:06.945 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:06.945 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.942+0000 7fb077d90700 5 monclient: authenticate success, global_id 4400 2023-09-30T02:07:06.945 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.942+0000 7fb077d90700 10 monclient: get_monmap_and_config success 2023-09-30T02:07:06.945 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.942+0000 7fb077d90700 4 set_mon_vals no callback set 2023-09-30T02:07:06.948 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.943+0000 7fb077d90700 10 monclient: shutdown 2023-09-30T02:07:06.948 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.943+0000 7fb077d90700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:07:06.948 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.943+0000 7fb077d90700 1 -- 172.21.15.59:0/3316485285 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb0700a5b70 msgr2=0x7fb0700b1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:06.948 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.943+0000 7fb077d90700 1 --2- 172.21.15.59:0/3316485285 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb0700a5b70 0x7fb0700b1500 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb070069af0 tx=0x7fb06c0091a0 comp rx=0 tx=0).stop 2023-09-30T02:07:06.948 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.943+0000 7fb077d90700 1 -- 172.21.15.59:0/3316485285 shutdown_connections 2023-09-30T02:07:06.948 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.943+0000 7fb077d90700 1 --2- 172.21.15.59:0/3316485285 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb0700a5b70 0x7fb0700b1500 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:06.948 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.943+0000 7fb077d90700 1 --2- 172.21.15.59:0/3316485285 >> v2:172.21.15.59:3300/0 conn(0x7fb0701420c0 0x7fb07013e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:06.948 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.943+0000 7fb077d90700 1 --2- 172.21.15.59:0/3316485285 >> v2:172.21.15.110:3300/0 conn(0x7fb0700a5f40 0x7fb070141b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:06.948 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.943+0000 7fb077d90700 1 -- 172.21.15.59:0/3316485285 >> 172.21.15.59:0/3316485285 conn(0x7fb0700631f0 msgr2=0x7fb0700635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:07:06.948 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.943+0000 7fb077d90700 1 -- 172.21.15.59:0/3316485285 shutdown_connections 2023-09-30T02:07:06.948 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.943+0000 7fb077d90700 1 -- 172.21.15.59:0/3316485285 wait complete. 2023-09-30T02:07:06.949 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.944+0000 7fb077d90700 10 monclient: build_initial_monmap 2023-09-30T02:07:06.949 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.944+0000 7fb077d90700 10 monclient: monmap: 2023-09-30T02:07:06.949 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:07:06.949 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:07:06.949 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:07:06.949 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:07:06.949 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:07:06.949 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:07:06.949 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:07:06.949 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:07:06.949 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:07:06.949 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:06.949 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.944+0000 7fb077d90700 1 librados: starting msgr at 2023-09-30T02:07:06.950 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.944+0000 7fb077d90700 1 librados: starting objecter 2023-09-30T02:07:06.950 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.944+0000 7fb077d90700 1 Processor -- start 2023-09-30T02:07:06.950 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.944+0000 7fb077d90700 1 -- start start 2023-09-30T02:07:06.950 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.944+0000 7fb077d90700 1 librados: setting wanted keys 2023-09-30T02:07:06.950 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.944+0000 7fb077d90700 1 librados: calling monclient init 2023-09-30T02:07:06.950 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.944+0000 7fb077d90700 10 monclient: init 2023-09-30T02:07:06.950 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.945+0000 7fb077d90700 10 monclient: _reopen_session rank -1 2023-09-30T02:07:06.950 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.945+0000 7fb077d90700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T02:07:06.950 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.945+0000 7fb077d90700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb0700a5b70 0x7fb070139580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:06.950 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.945+0000 7fb077d90700 10 monclient(hunting): picked mon.noname-b con 0x7fb0700a5b70 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:07:06.950 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.945+0000 7fb077d90700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0700a5f40 0x7fb070137880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:06.950 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.945+0000 7fb077d90700 10 monclient(hunting): picked mon.noname-a con 0x7fb0700a5f40 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:07:06.951 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.945+0000 7fb077d90700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb0701420c0 0x7fb0700a8400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:06.951 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.945+0000 7fb077d90700 10 monclient(hunting): picked mon.noname-c con 0x7fb0701420c0 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:07:06.951 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.945+0000 7fb077d90700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:06.951 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.945+0000 7fb077d90700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb07014eee0 con 0x7fb0700a5f40 2023-09-30T02:07:06.951 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.945+0000 7fb077d90700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:06.951 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.945+0000 7fb077d90700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0700a9c60 con 0x7fb0700a5b70 2023-09-30T02:07:06.951 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.945+0000 7fb077d90700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:06.951 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.945+0000 7fb077d90700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb070139ba0 con 0x7fb0701420c0 2023-09-30T02:07:06.951 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.945+0000 7fb077d90700 10 monclient(hunting): _renew_subs 2023-09-30T02:07:06.951 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.945+0000 7fb077d90700 10 monclient(hunting): authenticate will time out at 1872.248446s 2023-09-30T02:07:06.951 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.946+0000 7fb07532b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0700a5f40 0x7fb070137880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:06.951 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.946+0000 7fb07632d700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb0701420c0 0x7fb0700a8400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:06.951 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.946+0000 7fb075b2c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb0700a5b70 0x7fb070139580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:06.952 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.946+0000 7fb075b2c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb0700a5b70 0x7fb070139580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:51668/0 (socket says 172.21.15.59:51668) 2023-09-30T02:07:06.952 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.946+0000 7fb075b2c700 1 -- 172.21.15.59:0/2706361397 learned_addr learned my addr 172.21.15.59:0/2706361397 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:07:06.952 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.946+0000 7fb075b2c700 10 monclient(hunting): get_auth_request con 0x7fb0700a5b70 auth_method 0 2023-09-30T02:07:06.952 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.946+0000 7fb075b2c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:07:06.952 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.946+0000 7fb075b2c700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:07:06.952 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.946+0000 7fb075b2c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:07:06.952 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.947+0000 7fb07632d700 10 monclient(hunting): get_auth_request con 0x7fb0701420c0 auth_method 0 2023-09-30T02:07:06.952 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.947+0000 7fb07632d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:07:06.952 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.947+0000 7fb07632d700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:07:06.952 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.947+0000 7fb07632d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:07:06.952 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.947+0000 7fb075b2c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:07:06.952 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.947+0000 7fb075b2c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:07:06.952 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.947+0000 7fb075b2c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:07:06.953 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.947+0000 7fb07532b700 10 monclient(hunting): get_auth_request con 0x7fb0700a5f40 auth_method 0 2023-09-30T02:07:06.953 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.947+0000 7fb07532b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:07:06.953 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.947+0000 7fb07532b700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:07:06.953 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.947+0000 7fb07532b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:07:06.953 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.947+0000 7fb07632d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:07:06.953 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.947+0000 7fb07632d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:07:06.953 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.947+0000 7fb07632d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:07:06.954 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.948+0000 7fb07532b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:07:06.954 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.948+0000 7fb07532b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:07:06.954 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.948+0000 7fb07532b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:07:06.954 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.951+0000 7fb075b2c700 10 monclient(hunting): handle_auth_done global_id 4456 payload 824 2023-09-30T02:07:06.954 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.951+0000 7fb075b2c700 1 -- 172.21.15.59:0/2706361397 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb0701420c0 msgr2=0x7fb0700a8400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:06.954 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.951+0000 7fb075b2c700 1 --2- 172.21.15.59:0/2706361397 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb0701420c0 0x7fb0700a8400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:06.955 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.951+0000 7fb075b2c700 1 -- 172.21.15.59:0/2706361397 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0700a5f40 msgr2=0x7fb070137880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:06.955 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.951+0000 7fb075b2c700 1 --2- 172.21.15.59:0/2706361397 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0700a5f40 0x7fb070137880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:06.955 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.951+0000 7fb075b2c700 10 monclient: _finish_hunting 0 2023-09-30T02:07:06.955 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.951+0000 7fb075b2c700 1 monclient: found mon.noname-b 2023-09-30T02:07:06.955 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.951+0000 7fb075b2c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:07:06.955 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.951+0000 7fb075b2c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:07:06.955 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.951+0000 7fb075b2c700 1 -- 172.21.15.59:0/2706361397 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb06c009420 con 0x7fb0700a5b70 2023-09-30T02:07:06.955 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.954+0000 7fb075b2c700 1 --2- 172.21.15.59:0/2706361397 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb0700a5b70 0x7fb070139580 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb06c013000 tx=0x7fb06c017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:07:06.955 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.954+0000 7fb065ffb700 10 client.?.objecter ms_handle_connect 0x7fb0700a5b70 2023-09-30T02:07:06.955 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.954+0000 7fb065ffb700 10 client.?.objecter resend_mon_ops 2023-09-30T02:07:06.960 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.959+0000 7fb065ffb700 1 -- 172.21.15.59:0/2706361397 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb06c01f070 con 0x7fb0700a5b70 2023-09-30T02:07:06.960 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.959+0000 7fb065ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:07:06.960 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.959+0000 7fb065ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:07:06.960 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.959+0000 7fb065ffb700 10 monclient: dump: 2023-09-30T02:07:06.960 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:07:06.960 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:06.960 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:06.960 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:06.960 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:07:06.960 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:06.961 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:07:06.961 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:07:06.961 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:07:06.961 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:06.961 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.959+0000 7fb065ffb700 10 monclient: _finish_auth 0 2023-09-30T02:07:06.961 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.959+0000 7fb065ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:07:06.961 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.959+0000 7fb065ffb700 1 -- 172.21.15.59:0/2706361397 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb06c011920 con 0x7fb0700a5b70 2023-09-30T02:07:06.961 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.959+0000 7fb077d90700 5 monclient: authenticate success, global_id 4456 2023-09-30T02:07:06.961 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.959+0000 7fb077d90700 10 monclient: _renew_subs 2023-09-30T02:07:06.961 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.959+0000 7fb077d90700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:07:06.961 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.959+0000 7fb077d90700 1 -- 172.21.15.59:0/2706361397 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb070137dc0 con 0x7fb0700a5b70 2023-09-30T02:07:06.961 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.959+0000 7fb065ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:07:06.961 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.959+0000 7fb065ffb700 1 -- 172.21.15.59:0/2706361397 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb06c023410 con 0x7fb0700a5b70 2023-09-30T02:07:06.962 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.959+0000 7fb065ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:07:06.962 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.959+0000 7fb065ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:07:06.962 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.959+0000 7fb065ffb700 10 monclient: dump: 2023-09-30T02:07:06.962 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:07:06.962 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:06.962 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:06.962 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:06.962 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:07:06.962 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:06.962 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:07:06.962 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:07:06.962 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:07:06.963 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:06.963 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.959+0000 7fb067fff700 4 set_mon_vals no callback set 2023-09-30T02:07:06.963 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.959+0000 7fb077d90700 10 client.4456.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:07:06.963 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.959+0000 7fb077d90700 10 monclient: _renew_subs 2023-09-30T02:07:06.963 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.959+0000 7fb077d90700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:07:06.963 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.959+0000 7fb077d90700 1 -- 172.21.15.59:0/2706361397 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0700a8940 con 0x7fb0700a5b70 2023-09-30T02:07:06.963 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.959+0000 7fb077d90700 1 librados: init done 2023-09-30T02:07:06.964 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.960+0000 7fb077d90700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:07:06.964 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.960+0000 7fb077d90700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:07:06.964 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.960+0000 7fb077d90700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:07:06.964 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.960+0000 7fb077d90700 1 -- 172.21.15.59:0/2706361397 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb070064a00 con 0x7fb0700a5b70 2023-09-30T02:07:06.980 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.980+0000 7fb065ffb700 1 -- 172.21.15.59:0/2706361397 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fb06c0223b0 con 0x7fb0700a5b70 2023-09-30T02:07:06.980 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.980+0000 7fb065ffb700 1 --2- 172.21.15.59:0/2706361397 >> v2:172.21.15.110:6800/115870 conn(0x7fb054068890 0x7fb05406ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:06.980 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.980+0000 7fb07532b700 1 --2- 172.21.15.59:0/2706361397 >> v2:172.21.15.110:6800/115870 conn(0x7fb054068890 0x7fb05406ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:06.981 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.980+0000 7fb07532b700 10 monclient: get_auth_request con 0x7fb054068890 auth_method 0 2023-09-30T02:07:06.981 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.981+0000 7fb065ffb700 10 client.4456.objecter ms_handle_connect 0x7fb054068890 2023-09-30T02:07:06.981 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.981+0000 7fb07532b700 1 --2- 172.21.15.59:0/2706361397 >> v2:172.21.15.110:6800/115870 conn(0x7fb054068890 0x7fb05406ad40 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb0700a95d0 tx=0x7fb060009250 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:07:06.993 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.994+0000 7fb065ffb700 1 -- 172.21.15.59:0/2706361397 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fb06c0571b0 con 0x7fb0700a5b70 2023-09-30T02:07:06.994 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.994+0000 7fb065ffb700 10 client.4456.objecter ms_dispatch 0x7fb070001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:07:06.994 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.994+0000 7fb065ffb700 3 client.4456.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:07:06.994 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.994+0000 7fb065ffb700 3 client.4456.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:07:06.994 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:06.994+0000 7fb065ffb700 20 client.4456.objecter dump_active .. 0 homeless 2023-09-30T02:07:07.222 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:07.222+0000 7fb065ffb700 1 -- 172.21.15.59:0/2706361397 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb06c052df0 con 0x7fb0700a5b70 2023-09-30T02:07:07.223 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:07.222+0000 7fb065ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:07:07.223 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:07.222+0000 7fb065ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:07:07.473 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:07.472+0000 7fb077d90700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:07:07.473 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:07.472+0000 7fb077d90700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:07:07.473 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:07.472+0000 7fb077d90700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:07:07.473 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:07.472+0000 7fb077d90700 1 -- 172.21.15.59:0/2706361397 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fb0700a9da0 con 0x7fb0700a5b70 2023-09-30T02:07:07.477 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:07.477+0000 7fb065ffb700 1 -- 172.21.15.59:0/2706361397 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fb0700a9da0 con 0x7fb0700a5b70 2023-09-30T02:07:07.477 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:07.477+0000 7fb065ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:07:07.477 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:07.477+0000 7fb065ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:07:07.477 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:07:07.477 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:07:07.479 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:07.478+0000 7fb077d90700 10 librados: watch_flush enter 2023-09-30T02:07:07.480 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:07.478+0000 7fb077d90700 10 librados: watch_flush exit 2023-09-30T02:07:07.480 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:07.478+0000 7fb077d90700 20 client.4456.objecter shutdown clearing up homeless session... 2023-09-30T02:07:07.480 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:07.478+0000 7fb077d90700 10 client.4456.objecter successfully canceled tick 2023-09-30T02:07:07.480 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:07.479+0000 7fb077d90700 1 -- 172.21.15.59:0/2706361397 >> v2:172.21.15.110:6800/115870 conn(0x7fb054068890 msgr2=0x7fb05406ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:07.480 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:07.479+0000 7fb077d90700 1 --2- 172.21.15.59:0/2706361397 >> v2:172.21.15.110:6800/115870 conn(0x7fb054068890 0x7fb05406ad40 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb0700a95d0 tx=0x7fb060009250 comp rx=0 tx=0).stop 2023-09-30T02:07:07.480 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:07.479+0000 7fb077d90700 10 monclient: shutdown 2023-09-30T02:07:07.480 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:07.479+0000 7fb077d90700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:07:07.480 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:07.479+0000 7fb077d90700 1 -- 172.21.15.59:0/2706361397 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb0700a5b70 msgr2=0x7fb070139580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:07.480 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:07.479+0000 7fb077d90700 1 --2- 172.21.15.59:0/2706361397 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb0700a5b70 0x7fb070139580 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb06c013000 tx=0x7fb06c017000 comp rx=0 tx=0).stop 2023-09-30T02:07:07.480 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:07.479+0000 7fb077d90700 1 -- 172.21.15.59:0/2706361397 shutdown_connections 2023-09-30T02:07:07.480 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:07.479+0000 7fb077d90700 1 --2- 172.21.15.59:0/2706361397 >> v2:172.21.15.110:6800/115870 conn(0x7fb054068890 0x7fb05406ad40 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:07.481 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:07.479+0000 7fb077d90700 1 --2- 172.21.15.59:0/2706361397 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb0700a5b70 0x7fb070139580 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:07.481 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:07.479+0000 7fb077d90700 1 --2- 172.21.15.59:0/2706361397 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb0701420c0 0x7fb0700a8400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:07.481 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:07.479+0000 7fb077d90700 1 --2- 172.21.15.59:0/2706361397 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb0700a5f40 0x7fb070137880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:07.481 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:07.479+0000 7fb077d90700 1 -- 172.21.15.59:0/2706361397 >> 172.21.15.59:0/2706361397 conn(0x7fb0700631f0 msgr2=0x7fb0700af560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:07:07.481 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:07.479+0000 7fb077d90700 1 -- 172.21.15.59:0/2706361397 shutdown_connections 2023-09-30T02:07:07.481 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:07.479+0000 7fb077d90700 1 -- 172.21.15.59:0/2706361397 wait complete. 2023-09-30T02:07:07.481 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:07.479+0000 7fb077d90700 1 librados: shutdown 2023-09-30T02:07:10.493 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:07:10.631 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.631+0000 7f736c103700 10 monclient: get_monmap_and_config 2023-09-30T02:07:10.631 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.631+0000 7f736c103700 10 monclient: build_initial_monmap 2023-09-30T02:07:10.631 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.631+0000 7f736c103700 10 monclient: monmap: 2023-09-30T02:07:10.631 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:07:10.631 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:10.632 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:07:10.632439+0000 2023-09-30T02:07:10.632 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:07:10.632439+0000 2023-09-30T02:07:10.632 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:07:10.632 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:10.632 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:07:10.632 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:07:10.632 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:07:10.632 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:07:10.632 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:07:10.632 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:10.634 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.631+0000 7f736c103700 1 Processor -- start 2023-09-30T02:07:10.634 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.631+0000 7f736c103700 1 -- start start 2023-09-30T02:07:10.634 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.631+0000 7f736c103700 10 monclient: init 2023-09-30T02:07:10.634 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.632+0000 7f736c103700 10 monclient: _reopen_session rank -1 2023-09-30T02:07:10.634 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.632+0000 7f736c103700 10 monclient: _add_conns ranks=[2,3,4,1,0] 2023-09-30T02:07:10.634 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.632+0000 7f736c103700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f73640a5b70 0x7f73640b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:10.635 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.632+0000 7f736c103700 10 monclient(hunting): picked mon.noname-b con 0x7f73640a5b70 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:07:10.635 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.632+0000 7f736c103700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f73640a5f40 0x7f7364141b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:10.635 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.632+0000 7f736c103700 10 monclient(hunting): picked mon.noname-a con 0x7f73640a5f40 addr v2:172.21.15.59:3300/0 2023-09-30T02:07:10.635 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.632+0000 7f736c103700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f73641420c0 0x7f736413e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:10.635 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.632+0000 7f736c103700 10 monclient(hunting): picked mon.noname-c con 0x7f73641420c0 addr v2:172.21.15.110:3300/0 2023-09-30T02:07:10.635 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.632+0000 7f736c103700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:10.635 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.632+0000 7f736c103700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73640a63b0 con 0x7f73640a5f40 2023-09-30T02:07:10.635 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.632+0000 7f736c103700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:10.635 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.632+0000 7f736c103700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73640a75a0 con 0x7f73641420c0 2023-09-30T02:07:10.635 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.632+0000 7f736c103700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:10.635 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.632+0000 7f736c103700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73640af3f0 con 0x7f73640a5b70 2023-09-30T02:07:10.635 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.632+0000 7f736c103700 10 monclient(hunting): _renew_subs 2023-09-30T02:07:10.635 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.632+0000 7f736c103700 10 monclient(hunting): authenticate will time out at 1875.935758s 2023-09-30T02:07:10.636 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.633+0000 7f736969e700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f73640a5f40 0x7f7364141b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:10.636 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.633+0000 7f7369e9f700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f73640a5b70 0x7f73640b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:10.636 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.633+0000 7f736a6a0700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f73641420c0 0x7f736413e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:10.636 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.634+0000 7f736969e700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f73640a5f40 0x7f7364141b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:39000/0 (socket says 172.21.15.59:39000) 2023-09-30T02:07:10.636 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.634+0000 7f7369e9f700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f73640a5b70 0x7f73640b1500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:51882/0 (socket says 172.21.15.59:51882) 2023-09-30T02:07:10.636 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.634+0000 7f736969e700 1 -- 172.21.15.59:0/2684834176 learned_addr learned my addr 172.21.15.59:0/2684834176 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:07:10.636 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.634+0000 7f736969e700 10 monclient(hunting): get_auth_request con 0x7f73640a5f40 auth_method 0 2023-09-30T02:07:10.636 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.634+0000 7f736969e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:07:10.636 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.634+0000 7f736969e700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:07:10.636 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.634+0000 7f736969e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:07:10.636 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.634+0000 7f7369e9f700 10 monclient(hunting): get_auth_request con 0x7f73640a5b70 auth_method 0 2023-09-30T02:07:10.636 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.634+0000 7f7369e9f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:07:10.637 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.634+0000 7f7369e9f700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:07:10.637 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.634+0000 7f7369e9f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:07:10.637 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.634+0000 7f736a6a0700 10 monclient(hunting): get_auth_request con 0x7f73641420c0 auth_method 0 2023-09-30T02:07:10.637 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.634+0000 7f736a6a0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:07:10.637 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.634+0000 7f736a6a0700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:07:10.637 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.634+0000 7f736a6a0700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:07:10.638 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.634+0000 7f7369e9f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:07:10.638 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.634+0000 7f7369e9f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:07:10.638 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.634+0000 7f7369e9f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:07:10.638 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.634+0000 7f736969e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:07:10.638 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.634+0000 7f736969e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:07:10.638 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.634+0000 7f736969e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:07:10.638 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.634+0000 7f736a6a0700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:07:10.639 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.634+0000 7f736a6a0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:07:10.639 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.634+0000 7f736a6a0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:07:10.639 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.637+0000 7f7369e9f700 10 monclient(hunting): handle_auth_done global_id 4406 payload 274 2023-09-30T02:07:10.639 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.637+0000 7f7369e9f700 1 -- 172.21.15.59:0/2684834176 >> v2:172.21.15.110:3300/0 conn(0x7f73641420c0 msgr2=0x7f736413e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:07:10.639 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.637+0000 7f7369e9f700 1 --2- 172.21.15.59:0/2684834176 >> v2:172.21.15.110:3300/0 conn(0x7f73641420c0 0x7f736413e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:10.639 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.637+0000 7f7369e9f700 1 -- 172.21.15.59:0/2684834176 >> v2:172.21.15.59:3300/0 conn(0x7f73640a5f40 msgr2=0x7f7364141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:07:10.639 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.637+0000 7f7369e9f700 1 --2- 172.21.15.59:0/2684834176 >> v2:172.21.15.59:3300/0 conn(0x7f73640a5f40 0x7f7364141b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:10.639 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.637+0000 7f7369e9f700 10 monclient: _finish_hunting 0 2023-09-30T02:07:10.639 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.637+0000 7f7369e9f700 1 monclient: found mon.noname-b 2023-09-30T02:07:10.639 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.637+0000 7f7369e9f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:07:10.639 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.637+0000 7f7369e9f700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.59:3301/0 2023-09-30T02:07:10.639 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.637+0000 7f7369e9f700 1 -- 172.21.15.59:0/2684834176 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7364142500 con 0x7f73640a5b70 2023-09-30T02:07:10.640 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.640+0000 7f7369e9f700 1 --2- 172.21.15.59:0/2684834176 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f73640a5b70 0x7f73640b1500 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f73640b0970 tx=0x7f73540091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9bb2812059af6025 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:07:10.645 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.644+0000 7f7368e9d700 1 -- 172.21.15.59:0/2684834176 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7354011630 con 0x7f73640a5b70 2023-09-30T02:07:10.645 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.644+0000 7f7368e9d700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:07:10.645 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.644+0000 7f7368e9d700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:07:10.645 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.644+0000 7f7368e9d700 10 monclient: dump: 2023-09-30T02:07:10.645 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:07:10.645 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:10.645 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:10.645 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:10.645 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:07:10.646 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:10.646 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:07:10.646 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:07:10.646 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:07:10.646 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:10.646 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.644+0000 7f7368e9d700 10 monclient: _finish_auth 0 2023-09-30T02:07:10.646 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.644+0000 7f7368e9d700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:07:10.646 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.644+0000 7f7368e9d700 1 -- 172.21.15.59:0/2684834176 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73540117b0 con 0x7f73640a5b70 2023-09-30T02:07:10.646 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.644+0000 7f736c103700 5 monclient: authenticate success, global_id 4406 2023-09-30T02:07:10.646 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.644+0000 7f736c103700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:07:10.646 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.644+0000 7f7368e9d700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:07:10.646 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.644+0000 7f736c103700 10 monclient: get_monmap_and_config success 2023-09-30T02:07:10.647 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.644+0000 7f736c103700 4 set_mon_vals no callback set 2023-09-30T02:07:10.647 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.644+0000 7f7368e9d700 1 -- 172.21.15.59:0/2684834176 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7354011b70 con 0x7f73640a5b70 2023-09-30T02:07:10.647 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.644+0000 7f7368e9d700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:07:10.647 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.644+0000 7f7368e9d700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:07:10.647 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.644+0000 7f7368e9d700 10 monclient: dump: 2023-09-30T02:07:10.647 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:07:10.647 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:10.647 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:10.647 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:10.647 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:07:10.647 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:10.647 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:07:10.647 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:07:10.648 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:07:10.648 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:10.650 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.645+0000 7f736c103700 10 monclient: shutdown 2023-09-30T02:07:10.650 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.645+0000 7f736c103700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:07:10.650 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.645+0000 7f736c103700 1 -- 172.21.15.59:0/2684834176 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f73640a5b70 msgr2=0x7f73640b1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:10.650 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.645+0000 7f736c103700 1 --2- 172.21.15.59:0/2684834176 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f73640a5b70 0x7f73640b1500 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f73640b0970 tx=0x7f73540091a0 comp rx=0 tx=0).stop 2023-09-30T02:07:10.650 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.645+0000 7f736c103700 1 -- 172.21.15.59:0/2684834176 shutdown_connections 2023-09-30T02:07:10.651 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.645+0000 7f736c103700 1 --2- 172.21.15.59:0/2684834176 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f73640a5b70 0x7f73640b1500 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:10.651 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.645+0000 7f736c103700 1 --2- 172.21.15.59:0/2684834176 >> v2:172.21.15.110:3300/0 conn(0x7f73641420c0 0x7f736413e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:10.651 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.645+0000 7f736c103700 1 --2- 172.21.15.59:0/2684834176 >> v2:172.21.15.59:3300/0 conn(0x7f73640a5f40 0x7f7364141b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:10.651 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.645+0000 7f736c103700 1 -- 172.21.15.59:0/2684834176 >> 172.21.15.59:0/2684834176 conn(0x7f73640631f0 msgr2=0x7f73640635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:07:10.651 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.645+0000 7f736c103700 1 -- 172.21.15.59:0/2684834176 shutdown_connections 2023-09-30T02:07:10.651 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.645+0000 7f736c103700 1 -- 172.21.15.59:0/2684834176 wait complete. 2023-09-30T02:07:10.651 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.646+0000 7f736c103700 10 monclient: build_initial_monmap 2023-09-30T02:07:10.651 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.646+0000 7f736c103700 10 monclient: monmap: 2023-09-30T02:07:10.651 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:07:10.651 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:07:10.651 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:07:10.651 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:07:10.651 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:07:10.652 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:07:10.652 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:07:10.652 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:07:10.652 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:07:10.652 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:10.652 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.646+0000 7f736c103700 1 librados: starting msgr at 2023-09-30T02:07:10.652 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.646+0000 7f736c103700 1 librados: starting objecter 2023-09-30T02:07:10.652 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.646+0000 7f736c103700 1 Processor -- start 2023-09-30T02:07:10.652 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.647+0000 7f736c103700 1 -- start start 2023-09-30T02:07:10.652 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.647+0000 7f736c103700 1 librados: setting wanted keys 2023-09-30T02:07:10.652 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.647+0000 7f736c103700 1 librados: calling monclient init 2023-09-30T02:07:10.652 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.647+0000 7f736c103700 10 monclient: init 2023-09-30T02:07:10.653 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.647+0000 7f736c103700 10 monclient: _reopen_session rank -1 2023-09-30T02:07:10.653 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.647+0000 7f736c103700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T02:07:10.653 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.647+0000 7f736c103700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f73640a5b70 0x7f73640aa020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:10.653 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.647+0000 7f736c103700 10 monclient(hunting): picked mon.noname-c con 0x7f73640a5b70 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:07:10.653 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.647+0000 7f736c103700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f73640a5f40 0x7f736413d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:10.653 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.647+0000 7f736c103700 10 monclient(hunting): picked mon.noname-b con 0x7f73640a5f40 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:07:10.653 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.647+0000 7f736c103700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73641420c0 0x7f736414d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:10.653 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.647+0000 7f736c103700 10 monclient(hunting): picked mon.noname-a con 0x7f73641420c0 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:07:10.653 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.647+0000 7f736c103700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:10.653 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.647+0000 7f736c103700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f736414e750 con 0x7f73641420c0 2023-09-30T02:07:10.653 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.647+0000 7f736c103700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:10.653 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.647+0000 7f736c103700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73640a9b00 con 0x7f73640a5f40 2023-09-30T02:07:10.654 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.647+0000 7f736c103700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:10.654 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.647+0000 7f736c103700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f736413e050 con 0x7f73640a5b70 2023-09-30T02:07:10.654 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.647+0000 7f736c103700 10 monclient(hunting): _renew_subs 2023-09-30T02:07:10.654 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.647+0000 7f736c103700 10 monclient(hunting): authenticate will time out at 1875.950806s 2023-09-30T02:07:10.654 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.648+0000 7f736969e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f73640a5f40 0x7f736413d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:10.654 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.648+0000 7f7369e9f700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f73640a5b70 0x7f73640aa020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:10.654 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.648+0000 7f736969e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f73640a5f40 0x7f736413d610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:50322/0 (socket says 172.21.15.59:50322) 2023-09-30T02:07:10.654 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.648+0000 7f736969e700 1 -- 172.21.15.59:0/135473183 learned_addr learned my addr 172.21.15.59:0/135473183 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:07:10.654 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.648+0000 7f736969e700 10 monclient(hunting): get_auth_request con 0x7f73640a5f40 auth_method 0 2023-09-30T02:07:10.654 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.648+0000 7f736969e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:07:10.654 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.648+0000 7f736969e700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:07:10.654 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.648+0000 7f736969e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:07:10.654 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.649+0000 7f7369e9f700 10 monclient(hunting): get_auth_request con 0x7f73640a5b70 auth_method 0 2023-09-30T02:07:10.655 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.649+0000 7f7369e9f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:07:10.655 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.649+0000 7f7369e9f700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:07:10.655 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.649+0000 7f7369e9f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:07:10.655 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.649+0000 7f736969e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:07:10.655 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.649+0000 7f736969e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:07:10.655 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.649+0000 7f736969e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:07:10.655 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.649+0000 7f7369e9f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:07:10.655 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.649+0000 7f7369e9f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:07:10.655 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.649+0000 7f7369e9f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:07:10.656 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.654+0000 7f736969e700 10 monclient(hunting): handle_auth_done global_id 4462 payload 824 2023-09-30T02:07:10.656 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.654+0000 7f736969e700 1 -- 172.21.15.59:0/135473183 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f73640a5b70 msgr2=0x7f73640aa020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:10.656 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.654+0000 7f736969e700 1 --2- 172.21.15.59:0/135473183 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f73640a5b70 0x7f73640aa020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:10.656 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.654+0000 7f736969e700 1 -- 172.21.15.59:0/135473183 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73641420c0 msgr2=0x7f736414d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:10.656 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.654+0000 7f736969e700 1 --2- 172.21.15.59:0/135473183 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73641420c0 0x7f736414d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:10.656 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.654+0000 7f736969e700 10 monclient: _finish_hunting 0 2023-09-30T02:07:10.656 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.654+0000 7f736969e700 1 monclient: found mon.noname-b 2023-09-30T02:07:10.656 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.654+0000 7f736969e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:07:10.657 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.654+0000 7f736969e700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:07:10.657 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.654+0000 7f736969e700 1 -- 172.21.15.59:0/135473183 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7354009420 con 0x7f73640a5f40 2023-09-30T02:07:10.657 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.657+0000 7f736969e700 1 --2- 172.21.15.59:0/135473183 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f73640a5f40 0x7f736413d610 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f735800ea80 tx=0x7f7358009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:07:10.657 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.657+0000 7f7361ffb700 10 client.?.objecter ms_handle_connect 0x7f73640a5f40 2023-09-30T02:07:10.657 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.657+0000 7f7361ffb700 10 client.?.objecter resend_mon_ops 2023-09-30T02:07:10.662 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.661+0000 7f7361ffb700 1 -- 172.21.15.59:0/135473183 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f735800c850 con 0x7f73640a5f40 2023-09-30T02:07:10.662 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.661+0000 7f7361ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:07:10.662 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.661+0000 7f7361ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:07:10.662 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.661+0000 7f7361ffb700 10 monclient: dump: 2023-09-30T02:07:10.663 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:07:10.663 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:10.663 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:10.663 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:10.663 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:07:10.663 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:10.663 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:07:10.663 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:07:10.663 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:07:10.663 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:10.663 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.661+0000 7f7361ffb700 10 monclient: _finish_auth 0 2023-09-30T02:07:10.663 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.661+0000 7f7361ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:07:10.663 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.661+0000 7f7361ffb700 1 -- 172.21.15.59:0/135473183 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f735800c9b0 con 0x7f73640a5f40 2023-09-30T02:07:10.664 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.661+0000 7f736c103700 5 monclient: authenticate success, global_id 4462 2023-09-30T02:07:10.664 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.661+0000 7f736c103700 10 monclient: _renew_subs 2023-09-30T02:07:10.664 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.661+0000 7f736c103700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:07:10.664 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.661+0000 7f736c103700 1 -- 172.21.15.59:0/135473183 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73640a9670 con 0x7f73640a5f40 2023-09-30T02:07:10.664 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.661+0000 7f7361ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:07:10.664 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.661+0000 7f7361ffb700 1 -- 172.21.15.59:0/135473183 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f735800cd30 con 0x7f73640a5f40 2023-09-30T02:07:10.664 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.661+0000 7f7361ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:07:10.664 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.661+0000 7f7361ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:07:10.664 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.661+0000 7f7361ffb700 10 monclient: dump: 2023-09-30T02:07:10.664 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:07:10.664 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:10.664 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:10.665 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:10.665 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:07:10.665 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:10.665 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:07:10.665 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:07:10.665 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:07:10.665 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:10.665 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.661+0000 7f7363fff700 4 set_mon_vals no callback set 2023-09-30T02:07:10.665 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.662+0000 7f736c103700 10 client.4462.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:07:10.665 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.662+0000 7f736c103700 10 monclient: _renew_subs 2023-09-30T02:07:10.665 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.662+0000 7f736c103700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:07:10.665 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.662+0000 7f736c103700 1 -- 172.21.15.59:0/135473183 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73640a93b0 con 0x7f73640a5f40 2023-09-30T02:07:10.665 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.662+0000 7f736c103700 1 librados: init done 2023-09-30T02:07:10.666 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.663+0000 7f736c103700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:07:10.666 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.663+0000 7f736c103700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:07:10.666 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.663+0000 7f736c103700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:07:10.666 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.663+0000 7f736c103700 1 -- 172.21.15.59:0/135473183 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7364064a00 con 0x7f73640a5f40 2023-09-30T02:07:10.923 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.922+0000 7f7361ffb700 1 -- 172.21.15.59:0/135473183 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f7358013680 con 0x7f73640a5f40 2023-09-30T02:07:10.924 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.923+0000 7f7361ffb700 1 --2- 172.21.15.59:0/135473183 >> v2:172.21.15.110:6800/115870 conn(0x7f7350068940 0x7f735006adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:10.924 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.923+0000 7f7361ffb700 1 -- 172.21.15.59:0/135473183 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f73580848e0 con 0x7f73640a5f40 2023-09-30T02:07:10.924 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.923+0000 7f7361ffb700 10 client.4462.objecter ms_dispatch 0x7f7364001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:07:10.924 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.923+0000 7f7361ffb700 3 client.4462.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:07:10.924 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.923+0000 7f7361ffb700 3 client.4462.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:07:10.924 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.923+0000 7f7361ffb700 20 client.4462.objecter dump_active .. 0 homeless 2023-09-30T02:07:10.924 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.923+0000 7f7369e9f700 1 --2- 172.21.15.59:0/135473183 >> v2:172.21.15.110:6800/115870 conn(0x7f7350068940 0x7f735006adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:10.924 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.923+0000 7f7369e9f700 10 monclient: get_auth_request con 0x7f7350068940 auth_method 0 2023-09-30T02:07:10.925 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.924+0000 7f7361ffb700 10 client.4462.objecter ms_handle_connect 0x7f7350068940 2023-09-30T02:07:10.925 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.924+0000 7f7369e9f700 1 --2- 172.21.15.59:0/135473183 >> v2:172.21.15.110:6800/115870 conn(0x7f7350068940 0x7f735006adf0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f73640b0970 tx=0x7f7354017000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:07:10.931 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.931+0000 7f7361ffb700 1 -- 172.21.15.59:0/135473183 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7358052180 con 0x7f73640a5f40 2023-09-30T02:07:10.932 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.931+0000 7f7361ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:07:10.932 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:10.931+0000 7f7361ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:07:11.194 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:11.194+0000 7f736c103700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:07:11.195 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:11.194+0000 7f736c103700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:07:11.195 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:11.194+0000 7f736c103700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:07:11.195 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:11.194+0000 7f736c103700 1 -- 172.21.15.59:0/135473183 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f736406a720 con 0x7f73640a5f40 2023-09-30T02:07:11.200 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:11.200+0000 7f7361ffb700 1 -- 172.21.15.59:0/135473183 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f73580557a0 con 0x7f73640a5f40 2023-09-30T02:07:11.201 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:11.200+0000 7f7361ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:07:11.201 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:11.200+0000 7f7361ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:07:11.201 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:07:11.201 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:07:11.204 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:11.203+0000 7f736c103700 10 librados: watch_flush enter 2023-09-30T02:07:11.204 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:11.203+0000 7f736c103700 10 librados: watch_flush exit 2023-09-30T02:07:11.204 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:11.203+0000 7f736c103700 20 client.4462.objecter shutdown clearing up homeless session... 2023-09-30T02:07:11.204 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:11.203+0000 7f736c103700 10 client.4462.objecter successfully canceled tick 2023-09-30T02:07:11.205 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:11.203+0000 7f736c103700 1 -- 172.21.15.59:0/135473183 >> v2:172.21.15.110:6800/115870 conn(0x7f7350068940 msgr2=0x7f735006adf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:11.205 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:11.203+0000 7f736c103700 1 --2- 172.21.15.59:0/135473183 >> v2:172.21.15.110:6800/115870 conn(0x7f7350068940 0x7f735006adf0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f73640b0970 tx=0x7f7354017000 comp rx=0 tx=0).stop 2023-09-30T02:07:11.205 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:11.203+0000 7f736c103700 10 monclient: shutdown 2023-09-30T02:07:11.205 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:11.203+0000 7f736c103700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:07:11.205 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:11.203+0000 7f736c103700 1 -- 172.21.15.59:0/135473183 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f73640a5f40 msgr2=0x7f736413d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:11.205 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:11.203+0000 7f736c103700 1 --2- 172.21.15.59:0/135473183 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f73640a5f40 0x7f736413d610 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f735800ea80 tx=0x7f7358009250 comp rx=0 tx=0).stop 2023-09-30T02:07:11.205 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:11.203+0000 7f736c103700 1 -- 172.21.15.59:0/135473183 shutdown_connections 2023-09-30T02:07:11.205 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:11.203+0000 7f736c103700 1 --2- 172.21.15.59:0/135473183 >> v2:172.21.15.110:6800/115870 conn(0x7f7350068940 0x7f735006adf0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:11.205 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:11.203+0000 7f736c103700 1 --2- 172.21.15.59:0/135473183 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73641420c0 0x7f736414d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:11.205 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:11.203+0000 7f736c103700 1 --2- 172.21.15.59:0/135473183 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f73640a5b70 0x7f73640aa020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:11.205 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:11.203+0000 7f736c103700 1 --2- 172.21.15.59:0/135473183 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f73640a5f40 0x7f736413d610 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:11.205 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:11.203+0000 7f736c103700 1 -- 172.21.15.59:0/135473183 >> 172.21.15.59:0/135473183 conn(0x7f73640631f0 msgr2=0x7f73640635c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:07:11.206 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:11.204+0000 7f736c103700 1 -- 172.21.15.59:0/135473183 shutdown_connections 2023-09-30T02:07:11.206 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:11.204+0000 7f736c103700 1 -- 172.21.15.59:0/135473183 wait complete. 2023-09-30T02:07:11.206 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:11.204+0000 7f736c103700 1 librados: shutdown 2023-09-30T02:07:13.251 INFO:tasks.ceph.mon.a.smithi059.stderr:==00:00:02:23.116 116256== Warning: unimplemented fcntl command: 1036 2023-09-30T02:07:13.255 INFO:tasks.ceph.mon.a.smithi059.stderr:==00:00:02:23.120 116256== Warning: unimplemented fcntl command: 1036 2023-09-30T02:07:13.297 INFO:tasks.ceph.mon.a.smithi059.stderr:==00:00:02:23.162 116256== Warning: unimplemented fcntl command: 1036 2023-09-30T02:07:13.312 INFO:tasks.ceph.mon.b.smithi110.stderr:==00:00:02:23.159 115869== Warning: unimplemented fcntl command: 1036 2023-09-30T02:07:13.317 INFO:tasks.ceph.mon.b.smithi110.stderr:==00:00:02:23.164 115869== Warning: unimplemented fcntl command: 1036 2023-09-30T02:07:13.327 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:02:23.192 116257== Warning: unimplemented fcntl command: 1036 2023-09-30T02:07:13.329 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:02:23.195 116257== Warning: unimplemented fcntl command: 1036 2023-09-30T02:07:13.342 INFO:tasks.ceph.mon.b.smithi110.stderr:==00:00:02:23.189 115869== Warning: unimplemented fcntl command: 1036 2023-09-30T02:07:13.355 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:02:23.220 116257== Warning: unimplemented fcntl command: 1036 2023-09-30T02:07:14.220 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:07:14.360 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.360+0000 7f7f71470700 10 monclient: get_monmap_and_config 2023-09-30T02:07:14.360 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.360+0000 7f7f71470700 10 monclient: build_initial_monmap 2023-09-30T02:07:14.360 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.360+0000 7f7f71470700 10 monclient: monmap: 2023-09-30T02:07:14.361 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:07:14.361 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:14.361 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:07:14.361762+0000 2023-09-30T02:07:14.361 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:07:14.361762+0000 2023-09-30T02:07:14.361 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:07:14.361 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:14.361 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:07:14.361 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:07:14.361 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:07:14.361 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:07:14.361 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:07:14.362 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:14.363 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.361+0000 7f7f71470700 1 Processor -- start 2023-09-30T02:07:14.363 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.361+0000 7f7f71470700 1 -- start start 2023-09-30T02:07:14.363 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.361+0000 7f7f71470700 10 monclient: init 2023-09-30T02:07:14.363 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.361+0000 7f7f71470700 10 monclient: _reopen_session rank -1 2023-09-30T02:07:14.363 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.361+0000 7f7f71470700 10 monclient: _add_conns ranks=[4,1,0,3,2] 2023-09-30T02:07:14.363 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.361+0000 7f7f71470700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f7f6c0b16e0 0x7f7f6c0a54e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:14.364 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.361+0000 7f7f71470700 10 monclient(hunting): picked mon.noname-c con 0x7f7f6c0b16e0 addr v2:172.21.15.110:3300/0 2023-09-30T02:07:14.364 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.361+0000 7f7f71470700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7f6c0a7650 addr v1:172.21.15.110:6789/0 2023-09-30T02:07:14.364 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.361+0000 7f7f71470700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7f6c13aef0 addr v1:172.21.15.59:6789/0 2023-09-30T02:07:14.364 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.361+0000 7f7f71470700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f6c0a5d40 con 0x7f7f6c13aef0 2023-09-30T02:07:14.364 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.361+0000 7f7f71470700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f6c0b1ab0 con 0x7f7f6c0a7650 2023-09-30T02:07:14.364 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.361+0000 7f7f71470700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:14.364 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.361+0000 7f7f71470700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f6c0a5a20 con 0x7f7f6c0b16e0 2023-09-30T02:07:14.364 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.361+0000 7f7f71470700 10 monclient(hunting): _renew_subs 2023-09-30T02:07:14.364 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.361+0000 7f7f71470700 10 monclient(hunting): authenticate will time out at 1879.664792s 2023-09-30T02:07:14.364 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.362+0000 7f7f6b7fe700 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f7f6c13aef0 0x7f7f6c13b390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:60632/0 (socket says 172.21.15.59:60632) 2023-09-30T02:07:14.364 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.362+0000 7f7f6b7fe700 1 -- 172.21.15.59:0/4187410594 learned_addr learned my addr 172.21.15.59:0/4187410594 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:07:14.364 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.362+0000 7f7f6affd700 1 --2- 172.21.15.59:0/4187410594 >> v2:172.21.15.110:3300/0 conn(0x7f7f6c0b16e0 0x7f7f6c0a54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:14.365 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.363+0000 7f7f6affd700 10 monclient(hunting): get_auth_request con 0x7f7f6c0b16e0 auth_method 0 2023-09-30T02:07:14.365 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.363+0000 7f7f6affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:07:14.365 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.363+0000 7f7f6affd700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:07:14.365 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.363+0000 7f7f6affd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:07:14.365 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.364+0000 7f7f6affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:07:14.365 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.364+0000 7f7f6affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:07:14.365 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.364+0000 7f7f6affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:07:14.366 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.366+0000 7f7f69ffb700 1 -- 172.21.15.59:0/4187410594 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3711830921 0 0) 0x7f7f6c0a5d40 con 0x7f7f6c13aef0 2023-09-30T02:07:14.366 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.366+0000 7f7f69ffb700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:07:14.366 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.366+0000 7f7f69ffb700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:07:14.366 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.366+0000 7f7f69ffb700 10 monclient(hunting): my global_id is 4452 2023-09-30T02:07:14.366 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.366+0000 7f7f69ffb700 1 -- 172.21.15.59:0/4187410594 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f54003570 con 0x7f7f6c13aef0 2023-09-30T02:07:14.370 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.369+0000 7f7f6affd700 10 monclient(hunting): handle_auth_done global_id 4465 payload 274 2023-09-30T02:07:14.370 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.369+0000 7f7f6affd700 1 -- 172.21.15.59:0/4187410594 >> v1:172.21.15.110:6789/0 conn(0x7f7f6c0a7650 legacy=0x7f7f6c0a7ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:14.370 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.369+0000 7f7f6affd700 1 -- 172.21.15.59:0/4187410594 >> v1:172.21.15.59:6789/0 conn(0x7f7f6c13aef0 legacy=0x7f7f6c13b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:14.370 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.369+0000 7f7f6affd700 10 monclient: _finish_hunting 0 2023-09-30T02:07:14.370 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.369+0000 7f7f6affd700 1 monclient: found mon.noname-c 2023-09-30T02:07:14.370 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.369+0000 7f7f6affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:07:14.370 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.369+0000 7f7f6affd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3300/0 2023-09-30T02:07:14.370 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.369+0000 7f7f6affd700 1 -- 172.21.15.59:0/4187410594 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f6c0a6c70 con 0x7f7f6c0b16e0 2023-09-30T02:07:14.372 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.372+0000 7f7f6affd700 1 --2- 172.21.15.59:0/4187410594 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7f6c0b16e0 0x7f7f6c0a54e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c0098e0 tx=0x7f7f5c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3458e9ce2158b88a server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:07:14.382 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.381+0000 7f7f69ffb700 1 -- 172.21.15.59:0/4187410594 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f5c00b640 con 0x7f7f6c0b16e0 2023-09-30T02:07:14.382 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.381+0000 7f7f69ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:07:14.382 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.381+0000 7f7f69ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:07:14.382 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.381+0000 7f7f69ffb700 10 monclient: dump: 2023-09-30T02:07:14.382 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:07:14.382 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:14.382 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:14.382 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:14.382 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:07:14.382 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:14.383 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:07:14.383 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:07:14.383 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:07:14.383 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:14.383 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.381+0000 7f7f69ffb700 10 monclient: _finish_auth 0 2023-09-30T02:07:14.383 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.381+0000 7f7f69ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:07:14.383 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.381+0000 7f7f69ffb700 1 -- 172.21.15.59:0/4187410594 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f5c00b7a0 con 0x7f7f6c0b16e0 2023-09-30T02:07:14.383 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.381+0000 7f7f69ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:07:14.383 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.381+0000 7f7f69ffb700 1 -- 172.21.15.59:0/4187410594 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f5c00bb40 con 0x7f7f6c0b16e0 2023-09-30T02:07:14.383 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.381+0000 7f7f71470700 5 monclient: authenticate success, global_id 4465 2023-09-30T02:07:14.383 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.381+0000 7f7f71470700 10 monclient: get_monmap_and_config success 2023-09-30T02:07:14.384 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.381+0000 7f7f69ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:07:14.384 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.381+0000 7f7f71470700 4 set_mon_vals no callback set 2023-09-30T02:07:14.384 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.381+0000 7f7f69ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:07:14.384 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.381+0000 7f7f69ffb700 10 monclient: dump: 2023-09-30T02:07:14.384 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:07:14.384 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:14.384 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:14.384 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:14.384 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:07:14.384 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:14.384 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:07:14.384 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:07:14.385 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:07:14.385 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:14.387 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.382+0000 7f7f71470700 10 monclient: shutdown 2023-09-30T02:07:14.387 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.382+0000 7f7f71470700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:07:14.388 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.382+0000 7f7f71470700 1 -- 172.21.15.59:0/4187410594 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7f6c0b16e0 msgr2=0x7f7f6c0a54e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:14.388 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.382+0000 7f7f71470700 1 --2- 172.21.15.59:0/4187410594 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7f6c0b16e0 0x7f7f6c0a54e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c0098e0 tx=0x7f7f5c0091a0 comp rx=0 tx=0).stop 2023-09-30T02:07:14.388 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.382+0000 7f7f71470700 1 -- 172.21.15.59:0/4187410594 shutdown_connections 2023-09-30T02:07:14.388 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.382+0000 7f7f71470700 1 --2- 172.21.15.59:0/4187410594 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7f6c0b16e0 0x7f7f6c0a54e0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:14.388 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.382+0000 7f7f71470700 1 -- 172.21.15.59:0/4187410594 >> 172.21.15.59:0/4187410594 conn(0x7f7f6c068fe0 msgr2=0x7f7f6c0631f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:07:14.388 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.382+0000 7f7f71470700 1 -- 172.21.15.59:0/4187410594 shutdown_connections 2023-09-30T02:07:14.388 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.382+0000 7f7f71470700 1 -- 172.21.15.59:0/4187410594 wait complete. 2023-09-30T02:07:14.388 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.383+0000 7f7f71470700 10 monclient: build_initial_monmap 2023-09-30T02:07:14.388 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.383+0000 7f7f71470700 10 monclient: monmap: 2023-09-30T02:07:14.388 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:07:14.388 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:07:14.389 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:07:14.389 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:07:14.389 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:07:14.389 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:07:14.389 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:07:14.389 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:07:14.389 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:07:14.389 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:14.389 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.383+0000 7f7f71470700 1 librados: starting msgr at 2023-09-30T02:07:14.389 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.383+0000 7f7f71470700 1 librados: starting objecter 2023-09-30T02:07:14.389 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.383+0000 7f7f71470700 1 Processor -- start 2023-09-30T02:07:14.389 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.383+0000 7f7f71470700 1 -- start start 2023-09-30T02:07:14.390 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.383+0000 7f7f71470700 1 librados: setting wanted keys 2023-09-30T02:07:14.390 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.383+0000 7f7f71470700 1 librados: calling monclient init 2023-09-30T02:07:14.390 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.383+0000 7f7f71470700 10 monclient: init 2023-09-30T02:07:14.390 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.384+0000 7f7f71470700 10 monclient: _reopen_session rank -1 2023-09-30T02:07:14.390 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.384+0000 7f7f71470700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T02:07:14.390 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.384+0000 7f7f71470700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7f6c0a7650 0x7f7f6c13e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:14.390 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.384+0000 7f7f71470700 10 monclient(hunting): picked mon.noname-b con 0x7f7f6c0a7650 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:07:14.390 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.384+0000 7f7f71470700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7f6c0b16e0 0x7f7f6c0ab030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:14.390 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.384+0000 7f7f71470700 10 monclient(hunting): picked mon.noname-c con 0x7f7f6c0b16e0 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:07:14.390 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.384+0000 7f7f71470700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f6c13aef0 0x7f7f6c151180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:14.391 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.384+0000 7f7f71470700 10 monclient(hunting): picked mon.noname-a con 0x7f7f6c13aef0 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:07:14.391 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.384+0000 7f7f71470700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:14.391 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.384+0000 7f7f71470700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f6c0aa540 con 0x7f7f6c13aef0 2023-09-30T02:07:14.391 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.384+0000 7f7f71470700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:14.391 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.384+0000 7f7f71470700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f6c0a9780 con 0x7f7f6c0a7650 2023-09-30T02:07:14.391 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.384+0000 7f7f71470700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:14.391 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.384+0000 7f7f71470700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f6c141ff0 con 0x7f7f6c0b16e0 2023-09-30T02:07:14.391 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.384+0000 7f7f71470700 10 monclient(hunting): _renew_subs 2023-09-30T02:07:14.391 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.384+0000 7f7f71470700 10 monclient(hunting): authenticate will time out at 1879.687506s 2023-09-30T02:07:14.391 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.385+0000 7f7f6affd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7f6c0a7650 0x7f7f6c13e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:14.391 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.385+0000 7f7f6a7fc700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7f6c0b16e0 0x7f7f6c0ab030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:14.391 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.385+0000 7f7f6b7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f6c13aef0 0x7f7f6c151180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:14.392 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.385+0000 7f7f6affd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7f6c0a7650 0x7f7f6c13e950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:50344/0 (socket says 172.21.15.59:50344) 2023-09-30T02:07:14.392 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.385+0000 7f7f6affd700 1 -- 172.21.15.59:0/4289433001 learned_addr learned my addr 172.21.15.59:0/4289433001 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:07:14.392 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.385+0000 7f7f6affd700 10 monclient(hunting): get_auth_request con 0x7f7f6c0a7650 auth_method 0 2023-09-30T02:07:14.392 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.385+0000 7f7f6affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:07:14.392 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.385+0000 7f7f6affd700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:07:14.392 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.385+0000 7f7f6affd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:07:14.392 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.385+0000 7f7f6a7fc700 10 monclient(hunting): get_auth_request con 0x7f7f6c0b16e0 auth_method 0 2023-09-30T02:07:14.392 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.385+0000 7f7f6a7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:07:14.392 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.385+0000 7f7f6a7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:07:14.392 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.385+0000 7f7f6a7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:07:14.392 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.385+0000 7f7f6b7fe700 10 monclient(hunting): get_auth_request con 0x7f7f6c13aef0 auth_method 0 2023-09-30T02:07:14.392 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.385+0000 7f7f6b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:07:14.393 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.385+0000 7f7f6b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:07:14.393 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.385+0000 7f7f6b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:07:14.393 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.386+0000 7f7f6affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:07:14.393 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.386+0000 7f7f6affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:07:14.393 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.386+0000 7f7f6affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:07:14.393 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.386+0000 7f7f6a7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:07:14.393 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.386+0000 7f7f6a7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:07:14.393 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.386+0000 7f7f6a7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:07:14.393 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.386+0000 7f7f6b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:07:14.393 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.386+0000 7f7f6b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:07:14.393 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.386+0000 7f7f6b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:07:14.394 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.390+0000 7f7f6affd700 10 monclient(hunting): handle_auth_done global_id 4471 payload 824 2023-09-30T02:07:14.394 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.390+0000 7f7f6affd700 1 -- 172.21.15.59:0/4289433001 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7f6c0b16e0 msgr2=0x7f7f6c0ab030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:14.395 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.390+0000 7f7f6affd700 1 --2- 172.21.15.59:0/4289433001 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7f6c0b16e0 0x7f7f6c0ab030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:14.395 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.390+0000 7f7f6affd700 1 -- 172.21.15.59:0/4289433001 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f6c13aef0 msgr2=0x7f7f6c151180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:14.395 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.390+0000 7f7f6affd700 1 --2- 172.21.15.59:0/4289433001 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f6c13aef0 0x7f7f6c151180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:14.395 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.390+0000 7f7f6affd700 10 monclient: _finish_hunting 0 2023-09-30T02:07:14.395 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.390+0000 7f7f6affd700 1 monclient: found mon.noname-b 2023-09-30T02:07:14.395 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.390+0000 7f7f6affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:07:14.395 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.390+0000 7f7f6affd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:07:14.395 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.390+0000 7f7f6affd700 1 -- 172.21.15.59:0/4289433001 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f5c009420 con 0x7f7f6c0a7650 2023-09-30T02:07:14.395 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.393+0000 7f7f6affd700 1 --2- 172.21.15.59:0/4289433001 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7f6c0a7650 0x7f7f6c13e950 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c00a010 tx=0x7f7f5c017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:07:14.395 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.393+0000 7f7f52ffd700 10 client.?.objecter ms_handle_connect 0x7f7f6c0a7650 2023-09-30T02:07:14.395 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.393+0000 7f7f52ffd700 10 client.?.objecter resend_mon_ops 2023-09-30T02:07:14.398 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.398+0000 7f7f52ffd700 1 -- 172.21.15.59:0/4289433001 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f5c01bcc0 con 0x7f7f6c0a7650 2023-09-30T02:07:14.399 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.398+0000 7f7f52ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:07:14.399 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.398+0000 7f7f52ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:07:14.399 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.398+0000 7f7f52ffd700 10 monclient: dump: 2023-09-30T02:07:14.399 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:07:14.399 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:14.399 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:14.399 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:14.399 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:07:14.399 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:14.399 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:07:14.399 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:07:14.400 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:07:14.400 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:14.400 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.398+0000 7f7f52ffd700 10 monclient: _finish_auth 0 2023-09-30T02:07:14.400 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.398+0000 7f7f52ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:07:14.400 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.398+0000 7f7f52ffd700 1 -- 172.21.15.59:0/4289433001 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f5c01be40 con 0x7f7f6c0a7650 2023-09-30T02:07:14.400 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.398+0000 7f7f52ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:07:14.400 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.398+0000 7f7f52ffd700 1 -- 172.21.15.59:0/4289433001 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f5c011670 con 0x7f7f6c0a7650 2023-09-30T02:07:14.400 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.398+0000 7f7f71470700 5 monclient: authenticate success, global_id 4471 2023-09-30T02:07:14.400 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.398+0000 7f7f71470700 10 monclient: _renew_subs 2023-09-30T02:07:14.400 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.398+0000 7f7f71470700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:07:14.400 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.398+0000 7f7f71470700 1 -- 172.21.15.59:0/4289433001 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f6c1516c0 con 0x7f7f6c0a7650 2023-09-30T02:07:14.400 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.398+0000 7f7f697fa700 4 set_mon_vals no callback set 2023-09-30T02:07:14.401 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.398+0000 7f7f52ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:07:14.401 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.398+0000 7f7f52ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:07:14.401 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.398+0000 7f7f52ffd700 10 monclient: dump: 2023-09-30T02:07:14.401 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:07:14.401 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:14.401 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:14.401 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:14.401 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:07:14.401 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:14.401 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:07:14.401 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:07:14.401 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:07:14.402 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:14.402 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.398+0000 7f7f71470700 10 client.4471.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:07:14.402 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.398+0000 7f7f71470700 10 monclient: _renew_subs 2023-09-30T02:07:14.402 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.398+0000 7f7f71470700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:07:14.402 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.398+0000 7f7f71470700 1 -- 172.21.15.59:0/4289433001 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f6c13f690 con 0x7f7f6c0a7650 2023-09-30T02:07:14.402 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.398+0000 7f7f71470700 1 librados: init done 2023-09-30T02:07:14.402 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.399+0000 7f7f71470700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:07:14.403 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.399+0000 7f7f71470700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:07:14.403 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.399+0000 7f7f71470700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:07:14.403 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.399+0000 7f7f71470700 1 -- 172.21.15.59:0/4289433001 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f6c140920 con 0x7f7f6c0a7650 2023-09-30T02:07:14.654 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.653+0000 7f7f52ffd700 1 -- 172.21.15.59:0/4289433001 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f7f5c011a60 con 0x7f7f6c0a7650 2023-09-30T02:07:14.655 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.654+0000 7f7f52ffd700 1 --2- 172.21.15.59:0/4289433001 >> v2:172.21.15.110:6800/115870 conn(0x7f7f58068960 0x7f7f5806ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:14.655 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.654+0000 7f7f52ffd700 1 -- 172.21.15.59:0/4289433001 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f7f5c085a30 con 0x7f7f6c0a7650 2023-09-30T02:07:14.655 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.654+0000 7f7f52ffd700 10 client.4471.objecter ms_dispatch 0x7f7f6c001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:07:14.655 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.654+0000 7f7f52ffd700 3 client.4471.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:07:14.656 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.654+0000 7f7f52ffd700 3 client.4471.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:07:14.656 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.654+0000 7f7f52ffd700 20 client.4471.objecter dump_active .. 0 homeless 2023-09-30T02:07:14.656 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.654+0000 7f7f6a7fc700 1 --2- 172.21.15.59:0/4289433001 >> v2:172.21.15.110:6800/115870 conn(0x7f7f58068960 0x7f7f5806ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:14.656 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.654+0000 7f7f6a7fc700 10 monclient: get_auth_request con 0x7f7f58068960 auth_method 0 2023-09-30T02:07:14.656 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.655+0000 7f7f52ffd700 10 client.4471.objecter ms_handle_connect 0x7f7f58068960 2023-09-30T02:07:14.656 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.655+0000 7f7f6a7fc700 1 --2- 172.21.15.59:0/4289433001 >> v2:172.21.15.110:6800/115870 conn(0x7f7f58068960 0x7f7f5806ae10 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f7f6c140420 tx=0x7f7f60007ec0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:07:14.663 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.663+0000 7f7f52ffd700 1 -- 172.21.15.59:0/4289433001 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7f5c053350 con 0x7f7f6c0a7650 2023-09-30T02:07:14.663 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.663+0000 7f7f52ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:07:14.663 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.663+0000 7f7f52ffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:07:14.926 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.926+0000 7f7f71470700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:07:14.926 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.926+0000 7f7f71470700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:07:14.927 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.926+0000 7f7f71470700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:07:14.927 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.926+0000 7f7f71470700 1 -- 172.21.15.59:0/4289433001 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f7f6c0aa3a0 con 0x7f7f6c0a7650 2023-09-30T02:07:14.931 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.932+0000 7f7f52ffd700 1 -- 172.21.15.59:0/4289433001 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f7f6c0aa3a0 con 0x7f7f6c0a7650 2023-09-30T02:07:14.931 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.932+0000 7f7f52ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:07:14.931 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.932+0000 7f7f52ffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:07:14.932 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:07:14.932 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:07:14.935 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.933+0000 7f7f71470700 10 librados: watch_flush enter 2023-09-30T02:07:14.935 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.933+0000 7f7f71470700 10 librados: watch_flush exit 2023-09-30T02:07:14.935 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.933+0000 7f7f71470700 20 client.4471.objecter shutdown clearing up homeless session... 2023-09-30T02:07:14.935 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.933+0000 7f7f71470700 10 client.4471.objecter successfully canceled tick 2023-09-30T02:07:14.935 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.933+0000 7f7f71470700 1 -- 172.21.15.59:0/4289433001 >> v2:172.21.15.110:6800/115870 conn(0x7f7f58068960 msgr2=0x7f7f5806ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:14.935 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.933+0000 7f7f71470700 1 --2- 172.21.15.59:0/4289433001 >> v2:172.21.15.110:6800/115870 conn(0x7f7f58068960 0x7f7f5806ae10 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f7f6c140420 tx=0x7f7f60007ec0 comp rx=0 tx=0).stop 2023-09-30T02:07:14.935 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.933+0000 7f7f71470700 10 monclient: shutdown 2023-09-30T02:07:14.935 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.933+0000 7f7f71470700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:07:14.935 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.933+0000 7f7f71470700 1 -- 172.21.15.59:0/4289433001 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7f6c0a7650 msgr2=0x7f7f6c13e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:14.936 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.933+0000 7f7f71470700 1 --2- 172.21.15.59:0/4289433001 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7f6c0a7650 0x7f7f6c13e950 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c00a010 tx=0x7f7f5c017000 comp rx=0 tx=0).stop 2023-09-30T02:07:14.936 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.933+0000 7f7f71470700 1 -- 172.21.15.59:0/4289433001 shutdown_connections 2023-09-30T02:07:14.936 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.933+0000 7f7f71470700 1 --2- 172.21.15.59:0/4289433001 >> v2:172.21.15.110:6800/115870 conn(0x7f7f58068960 0x7f7f5806ae10 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:14.936 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.933+0000 7f7f71470700 1 --2- 172.21.15.59:0/4289433001 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7f6c0a7650 0x7f7f6c13e950 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:14.936 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.933+0000 7f7f71470700 1 --2- 172.21.15.59:0/4289433001 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7f6c13aef0 0x7f7f6c151180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:14.936 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.933+0000 7f7f71470700 1 --2- 172.21.15.59:0/4289433001 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7f6c0b16e0 0x7f7f6c0ab030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:14.936 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.934+0000 7f7f71470700 1 -- 172.21.15.59:0/4289433001 >> 172.21.15.59:0/4289433001 conn(0x7f7f6c068fe0 msgr2=0x7f7f6c0b4e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:07:14.936 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.934+0000 7f7f71470700 1 -- 172.21.15.59:0/4289433001 shutdown_connections 2023-09-30T02:07:14.936 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.934+0000 7f7f71470700 1 -- 172.21.15.59:0/4289433001 wait complete. 2023-09-30T02:07:14.936 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:14.934+0000 7f7f71470700 1 librados: shutdown 2023-09-30T02:07:17.948 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:07:18.095 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.095+0000 7f2829c1a700 10 monclient: get_monmap_and_config 2023-09-30T02:07:18.096 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.095+0000 7f2829c1a700 10 monclient: build_initial_monmap 2023-09-30T02:07:18.096 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.095+0000 7f2829c1a700 10 monclient: monmap: 2023-09-30T02:07:18.096 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:07:18.096 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:18.096 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:07:18.096890+0000 2023-09-30T02:07:18.096 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:07:18.096890+0000 2023-09-30T02:07:18.096 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:07:18.096 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:18.096 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:07:18.096 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:07:18.097 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:07:18.097 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:07:18.097 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:07:18.097 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:18.098 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.096+0000 7f2829c1a700 1 Processor -- start 2023-09-30T02:07:18.098 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.096+0000 7f2829c1a700 1 -- start start 2023-09-30T02:07:18.098 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.096+0000 7f2829c1a700 10 monclient: init 2023-09-30T02:07:18.098 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.097+0000 7f2829c1a700 10 monclient: _reopen_session rank -1 2023-09-30T02:07:18.098 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.097+0000 7f2829c1a700 10 monclient: _add_conns ranks=[4,2,0,3,1] 2023-09-30T02:07:18.099 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.097+0000 7f2829c1a700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f28240a5b70 0x7f28240b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:18.099 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.097+0000 7f2829c1a700 10 monclient(hunting): picked mon.noname-c con 0x7f28240a5b70 addr v2:172.21.15.110:3300/0 2023-09-30T02:07:18.099 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.097+0000 7f2829c1a700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f28240a5f40 0x7f2824141b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:18.099 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.097+0000 7f2829c1a700 10 monclient(hunting): picked mon.noname-b con 0x7f28240a5f40 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:07:18.099 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.097+0000 7f2829c1a700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f28241420c0 addr v1:172.21.15.59:6789/0 2023-09-30T02:07:18.099 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.097+0000 7f2829c1a700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28240a63b0 con 0x7f28241420c0 2023-09-30T02:07:18.099 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.097+0000 7f2829c1a700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:18.099 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.097+0000 7f2829c1a700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28240a75a0 con 0x7f28240a5b70 2023-09-30T02:07:18.099 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.097+0000 7f2829c1a700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:18.099 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.097+0000 7f2829c1a700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28240af3f0 con 0x7f28240a5f40 2023-09-30T02:07:18.099 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.097+0000 7f2829c1a700 10 monclient(hunting): _renew_subs 2023-09-30T02:07:18.099 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.097+0000 7f2829c1a700 10 monclient(hunting): authenticate will time out at 1883.400305s 2023-09-30T02:07:18.100 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.098+0000 7f2823fff700 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f28241420c0 0x7f282413e230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:48974/0 (socket says 172.21.15.59:48974) 2023-09-30T02:07:18.100 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.098+0000 7f2823fff700 1 -- 172.21.15.59:0/1727960518 learned_addr learned my addr 172.21.15.59:0/1727960518 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:07:18.101 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.101+0000 7f2822ffd700 1 -- 172.21.15.59:0/1727960518 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 192220030 0 0) 0x7f28240a63b0 con 0x7f28241420c0 2023-09-30T02:07:18.101 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.101+0000 7f2822ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:07:18.101 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.101+0000 7f2822ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:07:18.102 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.101+0000 7f2822ffd700 10 monclient(hunting): my global_id is 4458 2023-09-30T02:07:18.102 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.101+0000 7f2822ffd700 1 -- 172.21.15.59:0/1727960518 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2810003570 con 0x7f28241420c0 2023-09-30T02:07:18.107 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.106+0000 7f2822ffd700 1 -- 172.21.15.59:0/1727960518 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3317536073 0 0) 0x7f2810003570 con 0x7f28241420c0 2023-09-30T02:07:18.107 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.106+0000 7f2822ffd700 1 -- 172.21.15.59:0/1727960518 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f28240a5f40 msgr2=0x7f2824141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:07:18.107 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.106+0000 7f2822ffd700 1 --2- 172.21.15.59:0/1727960518 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f28240a5f40 0x7f2824141b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:18.108 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.106+0000 7f2822ffd700 1 -- 172.21.15.59:0/1727960518 >> v2:172.21.15.110:3300/0 conn(0x7f28240a5b70 msgr2=0x7f28240b1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:07:18.108 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.106+0000 7f2822ffd700 1 --2- 172.21.15.59:0/1727960518 >> v2:172.21.15.110:3300/0 conn(0x7f28240a5b70 0x7f28240b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:18.108 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.106+0000 7f2822ffd700 10 monclient: _finish_hunting 0 2023-09-30T02:07:18.108 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.106+0000 7f2822ffd700 1 monclient: found mon.noname-a-legacy 2023-09-30T02:07:18.108 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.106+0000 7f2822ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:07:18.108 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.106+0000 7f2822ffd700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.59:6789/0 2023-09-30T02:07:18.108 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.106+0000 7f2822ffd700 1 -- 172.21.15.59:0/1727960518 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2824142500 con 0x7f28241420c0 2023-09-30T02:07:18.108 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.106+0000 7f2822ffd700 10 monclient: _finish_auth 0 2023-09-30T02:07:18.108 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.106+0000 7f2822ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:07:18.108 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.106+0000 7f2822ffd700 1 -- 172.21.15.59:0/1727960518 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2174283827 0 0) 0x7f281c0017f0 con 0x7f28241420c0 2023-09-30T02:07:18.108 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.106+0000 7f2829c1a700 5 monclient: authenticate success, global_id 4458 2023-09-30T02:07:18.109 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.106+0000 7f2829c1a700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:07:18.109 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.106+0000 7f2822ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:07:18.109 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.106+0000 7f2822ffd700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-09-30T02:07:18.109 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.106+0000 7f2822ffd700 10 monclient: dump: 2023-09-30T02:07:18.109 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:07:18.109 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:18.109 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:18.109 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:18.109 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:07:18.109 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:18.109 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:07:18.109 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:07:18.110 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:07:18.110 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:18.110 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.106+0000 7f2822ffd700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] but i'm connected to v1:172.21.15.59:6789/0, reconnecting 2023-09-30T02:07:18.110 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.106+0000 7f2822ffd700 10 monclient: _reopen_session rank -1 2023-09-30T02:07:18.110 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.106+0000 7f2822ffd700 1 -- 172.21.15.59:0/1727960518 >> v1:172.21.15.59:6789/0 conn(0x7f28241420c0 legacy=0x7f282413e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:18.110 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.106+0000 7f2822ffd700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T02:07:18.110 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.106+0000 7f2822ffd700 1 --2- 172.21.15.59:0/1727960518 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2810003ae0 0x7f2810003ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:18.110 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.106+0000 7f2822ffd700 10 monclient(hunting): picked mon.b con 0x7f2810003ae0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:07:18.110 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.106+0000 7f2822ffd700 1 --2- 172.21.15.59:0/1727960518 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2810004810 0x7f2810004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:18.110 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.106+0000 7f2822ffd700 10 monclient(hunting): picked mon.a con 0x7f2810004810 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:07:18.110 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.106+0000 7f2822ffd700 1 --2- 172.21.15.59:0/1727960518 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f28240a5f40 0x7f281000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:18.110 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.106+0000 7f2822ffd700 10 monclient(hunting): picked mon.c con 0x7f28240a5f40 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:07:18.111 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.106+0000 7f2822ffd700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:18.111 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.106+0000 7f2822ffd700 1 -- 172.21.15.59:0/1727960518 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28240a75a0 con 0x7f2810004810 2023-09-30T02:07:18.111 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.106+0000 7f2822ffd700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:18.111 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.106+0000 7f2822ffd700 1 -- 172.21.15.59:0/1727960518 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28240af3f0 con 0x7f2810003ae0 2023-09-30T02:07:18.111 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.106+0000 7f2822ffd700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:18.111 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.106+0000 7f2822ffd700 1 -- 172.21.15.59:0/1727960518 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28100056a0 con 0x7f28240a5f40 2023-09-30T02:07:18.111 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.106+0000 7f2822ffd700 10 monclient(hunting): _renew_subs 2023-09-30T02:07:18.111 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.106+0000 7f2822ffd700 10 monclient(hunting): _finish_auth 0 2023-09-30T02:07:18.111 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.106+0000 7f2829c1a700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-09-30T02:07:18.112 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.109+0000 7f281bfff700 1 --2- 172.21.15.59:0/1727960518 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2810004810 0x7f2810004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:18.112 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.110+0000 7f2823fff700 1 --2- 172.21.15.59:0/1727960518 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f28240a5f40 0x7f281000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:18.112 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.111+0000 7f281bfff700 10 monclient(hunting): get_auth_request con 0x7f2810004810 auth_method 0 2023-09-30T02:07:18.112 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.111+0000 7f281bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:07:18.112 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.111+0000 7f281bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:07:18.112 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.111+0000 7f281bfff700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T02:07:18.113 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.113+0000 7f281bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:07:18.113 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.113+0000 7f281bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:07:18.113 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.113+0000 7f281bfff700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T02:07:18.116 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.116+0000 7f281bfff700 10 monclient(hunting): handle_auth_done global_id 4458 payload 293 2023-09-30T02:07:18.116 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.116+0000 7f281bfff700 1 -- 172.21.15.59:0/1727960518 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f28240a5f40 msgr2=0x7f281000ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:07:18.117 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.116+0000 7f281bfff700 1 --2- 172.21.15.59:0/1727960518 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f28240a5f40 0x7f281000ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:18.117 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.116+0000 7f281bfff700 1 -- 172.21.15.59:0/1727960518 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2810003ae0 msgr2=0x7f2810003ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:07:18.117 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.116+0000 7f281bfff700 1 --2- 172.21.15.59:0/1727960518 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2810003ae0 0x7f2810003ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:18.117 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.116+0000 7f281bfff700 10 monclient: _finish_hunting 0 2023-09-30T02:07:18.117 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.116+0000 7f281bfff700 1 monclient: found mon.a 2023-09-30T02:07:18.117 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.116+0000 7f281bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:07:18.117 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.116+0000 7f281bfff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:07:18.117 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.116+0000 7f281bfff700 1 -- 172.21.15.59:0/1727960518 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2810003570 con 0x7f2810004810 2023-09-30T02:07:18.117 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.116+0000 7f281bfff700 10 monclient: _finish_auth 0 2023-09-30T02:07:18.117 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.116+0000 7f281bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:07:18.119 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.118+0000 7f281bfff700 1 --2- 172.21.15.59:0/1727960518 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2810004810 0x7f2810004c80 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f280c0099f0 tx=0x7f280c009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df8b3dc95ee16e89 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:07:18.125 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.124+0000 7f2822ffd700 1 -- 172.21.15.59:0/1727960518 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f280c011400 con 0x7f2810004810 2023-09-30T02:07:18.125 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.124+0000 7f2822ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:07:18.125 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.124+0000 7f2822ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T02:07:18.126 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.124+0000 7f2822ffd700 10 monclient: dump: 2023-09-30T02:07:18.126 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:07:18.126 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:18.126 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:18.126 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:18.126 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:07:18.126 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:18.126 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:07:18.126 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:07:18.126 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:07:18.126 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:18.127 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.124+0000 7f2822ffd700 1 -- 172.21.15.59:0/1727960518 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f280c011560 con 0x7f2810004810 2023-09-30T02:07:18.127 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.124+0000 7f2829c1a700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:07:18.127 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.124+0000 7f2822ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:07:18.127 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.124+0000 7f2829c1a700 10 monclient: get_monmap_and_config success 2023-09-30T02:07:18.127 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.124+0000 7f2829c1a700 4 set_mon_vals no callback set 2023-09-30T02:07:18.127 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.124+0000 7f2822ffd700 1 -- 172.21.15.59:0/1727960518 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f280c011920 con 0x7f2810004810 2023-09-30T02:07:18.127 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.124+0000 7f2822ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:07:18.127 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.124+0000 7f2822ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T02:07:18.127 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.124+0000 7f2822ffd700 10 monclient: dump: 2023-09-30T02:07:18.127 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:07:18.127 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:18.127 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:18.128 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:18.128 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:07:18.128 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:18.128 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:07:18.128 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:07:18.128 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:07:18.128 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:18.128 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.125+0000 7f2829c1a700 10 monclient: shutdown 2023-09-30T02:07:18.128 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.125+0000 7f2829c1a700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:07:18.128 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.125+0000 7f2829c1a700 1 -- 172.21.15.59:0/1727960518 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2810004810 msgr2=0x7f2810004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:18.128 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.125+0000 7f2829c1a700 1 --2- 172.21.15.59:0/1727960518 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2810004810 0x7f2810004c80 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f280c0099f0 tx=0x7f280c009060 comp rx=0 tx=0).stop 2023-09-30T02:07:18.128 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.125+0000 7f2823fff700 1 -- 172.21.15.59:0/1727960518 reap_dead start 2023-09-30T02:07:18.129 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.125+0000 7f2829c1a700 1 -- 172.21.15.59:0/1727960518 shutdown_connections 2023-09-30T02:07:18.129 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.125+0000 7f2829c1a700 1 -- 172.21.15.59:0/1727960518 >> 172.21.15.59:0/1727960518 conn(0x7f28240631f0 msgr2=0x7f28240635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:07:18.129 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.125+0000 7f2829c1a700 1 -- 172.21.15.59:0/1727960518 shutdown_connections 2023-09-30T02:07:18.129 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.125+0000 7f2829c1a700 1 -- 172.21.15.59:0/1727960518 wait complete. 2023-09-30T02:07:18.131 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.125+0000 7f2829c1a700 10 monclient: build_initial_monmap 2023-09-30T02:07:18.131 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.126+0000 7f2829c1a700 10 monclient: monmap: 2023-09-30T02:07:18.131 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:07:18.131 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:07:18.131 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:07:18.131 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:07:18.131 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:07:18.131 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:07:18.131 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:07:18.131 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:07:18.131 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:07:18.132 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:18.132 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.126+0000 7f2829c1a700 1 librados: starting msgr at 2023-09-30T02:07:18.132 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.126+0000 7f2829c1a700 1 librados: starting objecter 2023-09-30T02:07:18.132 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.126+0000 7f2829c1a700 1 Processor -- start 2023-09-30T02:07:18.132 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.126+0000 7f2829c1a700 1 -- start start 2023-09-30T02:07:18.132 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.126+0000 7f2829c1a700 1 librados: setting wanted keys 2023-09-30T02:07:18.132 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.126+0000 7f2829c1a700 1 librados: calling monclient init 2023-09-30T02:07:18.132 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.126+0000 7f2829c1a700 10 monclient: init 2023-09-30T02:07:18.132 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.127+0000 7f2829c1a700 10 monclient: _reopen_session rank -1 2023-09-30T02:07:18.132 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.127+0000 7f2829c1a700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T02:07:18.132 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.127+0000 7f2829c1a700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2810004810 0x7f282413ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:18.132 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.127+0000 7f2829c1a700 10 monclient(hunting): picked mon.noname-b con 0x7f2810004810 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:07:18.133 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.127+0000 7f2829c1a700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f282413e200 0x7f282414d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:18.133 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.127+0000 7f2829c1a700 10 monclient(hunting): picked mon.noname-c con 0x7f282413e200 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:07:18.133 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.127+0000 7f2829c1a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f282413ec20 0x7f2824150e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:18.133 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.127+0000 7f2829c1a700 10 monclient(hunting): picked mon.noname-a con 0x7f282413ec20 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:07:18.133 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.127+0000 7f2829c1a700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:18.133 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.127+0000 7f2829c1a700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f282414b500 con 0x7f282413ec20 2023-09-30T02:07:18.133 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.127+0000 7f2829c1a700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:18.133 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.127+0000 7f2829c1a700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f282413e790 con 0x7f2810004810 2023-09-30T02:07:18.133 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.127+0000 7f2829c1a700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:18.133 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.127+0000 7f2829c1a700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f282413b9f0 con 0x7f282413e200 2023-09-30T02:07:18.133 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.127+0000 7f2829c1a700 10 monclient(hunting): _renew_subs 2023-09-30T02:07:18.133 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.127+0000 7f2829c1a700 10 monclient(hunting): authenticate will time out at 1883.430175s 2023-09-30T02:07:18.134 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.127+0000 7f2823fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f282413ec20 0x7f2824150e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:18.134 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.128+0000 7f2823fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f282413ec20 0x7f2824150e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58246/0 (socket says 172.21.15.59:58246) 2023-09-30T02:07:18.134 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.128+0000 7f2823fff700 1 -- 172.21.15.59:0/832436166 learned_addr learned my addr 172.21.15.59:0/832436166 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:07:18.134 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.128+0000 7f2823fff700 10 monclient(hunting): get_auth_request con 0x7f282413ec20 auth_method 0 2023-09-30T02:07:18.134 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.128+0000 7f2823fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:07:18.134 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.128+0000 7f2823fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:07:18.134 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.128+0000 7f2823fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:07:18.134 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.128+0000 7f2823fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:07:18.134 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.128+0000 7f2823fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:07:18.134 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.128+0000 7f2823fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:07:18.135 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.132+0000 7f2823fff700 10 monclient(hunting): handle_auth_done global_id 4461 payload 824 2023-09-30T02:07:18.135 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.132+0000 7f2823fff700 1 -- 172.21.15.59:0/832436166 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f282413e200 msgr2=0x7f282414d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:18.135 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.132+0000 7f2823fff700 1 --2- 172.21.15.59:0/832436166 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f282413e200 0x7f282414d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:18.135 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.132+0000 7f2823fff700 1 -- 172.21.15.59:0/832436166 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2810004810 msgr2=0x7f282413ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:18.135 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.132+0000 7f2823fff700 1 --2- 172.21.15.59:0/832436166 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2810004810 0x7f282413ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:18.136 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.132+0000 7f2823fff700 10 monclient: _finish_hunting 0 2023-09-30T02:07:18.136 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.132+0000 7f2823fff700 1 monclient: found mon.noname-a 2023-09-30T02:07:18.136 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.132+0000 7f2823fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:07:18.136 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.132+0000 7f2823fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.59:3300/0 2023-09-30T02:07:18.136 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.132+0000 7f2823fff700 1 -- 172.21.15.59:0/832436166 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f282413d920 con 0x7f282413ec20 2023-09-30T02:07:18.136 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.136+0000 7f2823fff700 1 --2- 172.21.15.59:0/832436166 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f282413ec20 0x7f2824150e10 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f281c001350 tx=0x7f281c009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:07:18.136 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.136+0000 7f281b7fe700 10 client.?.objecter ms_handle_connect 0x7f282413ec20 2023-09-30T02:07:18.137 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.136+0000 7f281b7fe700 10 client.?.objecter resend_mon_ops 2023-09-30T02:07:18.142 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.142+0000 7f281b7fe700 1 -- 172.21.15.59:0/832436166 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f281c004dc0 con 0x7f282413ec20 2023-09-30T02:07:18.143 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.142+0000 7f281b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:07:18.143 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.142+0000 7f281b7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T02:07:18.143 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.142+0000 7f281b7fe700 10 monclient: dump: 2023-09-30T02:07:18.143 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:07:18.143 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:18.143 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:18.143 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:18.143 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:07:18.143 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:18.143 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:07:18.143 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:07:18.144 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:07:18.144 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:18.144 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.142+0000 7f281b7fe700 10 monclient: _finish_auth 0 2023-09-30T02:07:18.144 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.142+0000 7f281b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:07:18.144 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.142+0000 7f281b7fe700 1 -- 172.21.15.59:0/832436166 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f281c00b9f0 con 0x7f282413ec20 2023-09-30T02:07:18.144 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.142+0000 7f2829c1a700 5 monclient: authenticate success, global_id 4461 2023-09-30T02:07:18.144 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.142+0000 7f281b7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:07:18.144 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.142+0000 7f281b7fe700 1 -- 172.21.15.59:0/832436166 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f281c00bdb0 con 0x7f282413ec20 2023-09-30T02:07:18.144 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.142+0000 7f2829c1a700 10 monclient: _renew_subs 2023-09-30T02:07:18.144 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.142+0000 7f2829c1a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:07:18.144 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.142+0000 7f2829c1a700 1 -- 172.21.15.59:0/832436166 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f282414ddc0 con 0x7f282413ec20 2023-09-30T02:07:18.144 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.142+0000 7f281b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:07:18.145 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.142+0000 7f281b7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T02:07:18.145 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.142+0000 7f281b7fe700 10 monclient: dump: 2023-09-30T02:07:18.145 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:07:18.145 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:18.145 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:18.145 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:18.145 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:07:18.145 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:18.145 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:07:18.145 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:07:18.145 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:07:18.145 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:18.146 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.142+0000 7f28227fc700 4 set_mon_vals no callback set 2023-09-30T02:07:18.146 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.142+0000 7f2829c1a700 10 client.4461.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:07:18.146 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.142+0000 7f2829c1a700 10 monclient: _renew_subs 2023-09-30T02:07:18.146 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.142+0000 7f2829c1a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:07:18.146 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.142+0000 7f2829c1a700 1 -- 172.21.15.59:0/832436166 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2824151350 con 0x7f282413ec20 2023-09-30T02:07:18.146 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.142+0000 7f2829c1a700 1 librados: init done 2023-09-30T02:07:18.146 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.143+0000 7f2829c1a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:07:18.147 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.143+0000 7f2829c1a700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:07:18.147 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.143+0000 7f2829c1a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:07:18.147 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.143+0000 7f2829c1a700 1 -- 172.21.15.59:0/832436166 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2824064a00 con 0x7f282413ec20 2023-09-30T02:07:18.175 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.175+0000 7f281b7fe700 1 -- 172.21.15.59:0/832436166 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f281c0073a0 con 0x7f282413ec20 2023-09-30T02:07:18.176 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.176+0000 7f281b7fe700 1 --2- 172.21.15.59:0/832436166 >> v2:172.21.15.110:6800/115870 conn(0x7f28040688c0 0x7f280406ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:18.176 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.176+0000 7f281b7fe700 1 -- 172.21.15.59:0/832436166 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f281c0849a0 con 0x7f282413ec20 2023-09-30T02:07:18.176 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.176+0000 7f281b7fe700 10 client.4461.objecter ms_dispatch 0x7f2824001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:07:18.177 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.176+0000 7f281b7fe700 3 client.4461.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:07:18.177 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.176+0000 7f281b7fe700 3 client.4461.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:07:18.177 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.176+0000 7f281b7fe700 20 client.4461.objecter dump_active .. 0 homeless 2023-09-30T02:07:18.177 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.176+0000 7f28237fe700 1 --2- 172.21.15.59:0/832436166 >> v2:172.21.15.110:6800/115870 conn(0x7f28040688c0 0x7f280406ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:18.177 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.176+0000 7f28237fe700 10 monclient: get_auth_request con 0x7f28040688c0 auth_method 0 2023-09-30T02:07:18.177 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.176+0000 7f28237fe700 1 --2- 172.21.15.59:0/832436166 >> v2:172.21.15.110:6800/115870 conn(0x7f28040688c0 0x7f280406ad70 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2814006fd0 tx=0x7f2814006db0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:07:18.178 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.177+0000 7f281b7fe700 10 client.4461.objecter ms_handle_connect 0x7f28040688c0 2023-09-30T02:07:18.454 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.454+0000 7f281b7fe700 1 -- 172.21.15.59:0/832436166 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f281c0523c0 con 0x7f282413ec20 2023-09-30T02:07:18.454 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.454+0000 7f281b7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:07:18.454 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.454+0000 7f281b7fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:07:18.710 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.710+0000 7f2829c1a700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:07:18.710 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.710+0000 7f2829c1a700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:07:18.711 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.710+0000 7f2829c1a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:07:18.711 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.710+0000 7f2829c1a700 1 -- 172.21.15.59:0/832436166 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f282413dab0 con 0x7f282413ec20 2023-09-30T02:07:18.716 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.716+0000 7f281b7fe700 1 -- 172.21.15.59:0/832436166 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f281c022080 con 0x7f282413ec20 2023-09-30T02:07:18.716 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.716+0000 7f281b7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:07:18.716 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.716+0000 7f281b7fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:07:18.716 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:07:18.717 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:07:18.719 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.718+0000 7f2829c1a700 10 librados: watch_flush enter 2023-09-30T02:07:18.719 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.718+0000 7f2829c1a700 10 librados: watch_flush exit 2023-09-30T02:07:18.719 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.718+0000 7f2829c1a700 20 client.4461.objecter shutdown clearing up homeless session... 2023-09-30T02:07:18.719 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.718+0000 7f2829c1a700 10 client.4461.objecter successfully canceled tick 2023-09-30T02:07:18.720 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.718+0000 7f2829c1a700 1 -- 172.21.15.59:0/832436166 >> v2:172.21.15.110:6800/115870 conn(0x7f28040688c0 msgr2=0x7f280406ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:18.720 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.718+0000 7f2829c1a700 1 --2- 172.21.15.59:0/832436166 >> v2:172.21.15.110:6800/115870 conn(0x7f28040688c0 0x7f280406ad70 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2814006fd0 tx=0x7f2814006db0 comp rx=0 tx=0).stop 2023-09-30T02:07:18.720 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.718+0000 7f2829c1a700 10 monclient: shutdown 2023-09-30T02:07:18.720 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.718+0000 7f2829c1a700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:07:18.720 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.718+0000 7f2829c1a700 1 -- 172.21.15.59:0/832436166 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f282413ec20 msgr2=0x7f2824150e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:18.720 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.718+0000 7f2829c1a700 1 --2- 172.21.15.59:0/832436166 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f282413ec20 0x7f2824150e10 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f281c001350 tx=0x7f281c009040 comp rx=0 tx=0).stop 2023-09-30T02:07:18.720 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.718+0000 7f2829c1a700 1 -- 172.21.15.59:0/832436166 shutdown_connections 2023-09-30T02:07:18.720 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.718+0000 7f2829c1a700 1 --2- 172.21.15.59:0/832436166 >> v2:172.21.15.110:6800/115870 conn(0x7f28040688c0 0x7f280406ad70 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:18.720 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.718+0000 7f2829c1a700 1 --2- 172.21.15.59:0/832436166 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2810004810 0x7f282413ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:18.720 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.718+0000 7f2829c1a700 1 --2- 172.21.15.59:0/832436166 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f282413ec20 0x7f2824150e10 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:18.720 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.718+0000 7f2829c1a700 1 --2- 172.21.15.59:0/832436166 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f282413e200 0x7f282414d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:18.721 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.718+0000 7f2829c1a700 1 -- 172.21.15.59:0/832436166 >> 172.21.15.59:0/832436166 conn(0x7f28240631f0 msgr2=0x7f2824144d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:07:18.721 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.718+0000 7f2829c1a700 1 -- 172.21.15.59:0/832436166 shutdown_connections 2023-09-30T02:07:18.721 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.718+0000 7f2829c1a700 1 -- 172.21.15.59:0/832436166 wait complete. 2023-09-30T02:07:18.721 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:18.719+0000 7f2829c1a700 1 librados: shutdown 2023-09-30T02:07:21.734 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:07:21.872 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.872+0000 7fec1ceff700 10 monclient: get_monmap_and_config 2023-09-30T02:07:21.872 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.872+0000 7fec1ceff700 10 monclient: build_initial_monmap 2023-09-30T02:07:21.872 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.872+0000 7fec1ceff700 10 monclient: monmap: 2023-09-30T02:07:21.872 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:07:21.872 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:21.872 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:07:21.873272+0000 2023-09-30T02:07:21.872 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:07:21.873272+0000 2023-09-30T02:07:21.873 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:07:21.873 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:21.873 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:07:21.873 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:07:21.873 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:07:21.873 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:07:21.873 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:07:21.873 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:21.875 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.873+0000 7fec1ceff700 1 Processor -- start 2023-09-30T02:07:21.875 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.873+0000 7fec1ceff700 1 -- start start 2023-09-30T02:07:21.875 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.873+0000 7fec1ceff700 10 monclient: init 2023-09-30T02:07:21.875 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.874+0000 7fec1ceff700 10 monclient: _reopen_session rank -1 2023-09-30T02:07:21.875 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.874+0000 7fec1ceff700 10 monclient: _add_conns ranks=[1,2,0,4,3] 2023-09-30T02:07:21.875 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.874+0000 7fec1ceff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fec180a5b70 addr v1:172.21.15.110:6789/0 2023-09-30T02:07:21.875 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.874+0000 7fec1ceff700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fec18142290 0x7fec180b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:21.875 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.874+0000 7fec1ceff700 10 monclient(hunting): picked mon.noname-b con 0x7fec18142290 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:07:21.875 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.874+0000 7fec1ceff700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fec180a5f40 addr v1:172.21.15.59:6789/0 2023-09-30T02:07:21.876 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.874+0000 7fec1ceff700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec180a6380 con 0x7fec180a5f40 2023-09-30T02:07:21.876 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.874+0000 7fec1ceff700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec180a75a0 con 0x7fec180a5b70 2023-09-30T02:07:21.877 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.874+0000 7fec1ceff700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:21.877 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.874+0000 7fec1ceff700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec180af3f0 con 0x7fec18142290 2023-09-30T02:07:21.877 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.874+0000 7fec1ceff700 10 monclient(hunting): _renew_subs 2023-09-30T02:07:21.877 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.874+0000 7fec1ceff700 10 monclient(hunting): authenticate will time out at 1887.176552s 2023-09-30T02:07:21.877 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.875+0000 7fec15d9b700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fec18142290 0x7fec180b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:21.877 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.875+0000 7fec1659c700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fec180a5b70 0x7fec18141b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.59:41226/0 (socket says 172.21.15.59:41226) 2023-09-30T02:07:21.877 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.875+0000 7fec1659c700 1 -- 172.21.15.59:0/966399949 learned_addr learned my addr 172.21.15.59:0/966399949 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:07:21.877 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.875+0000 7fec15d9b700 10 monclient(hunting): get_auth_request con 0x7fec18142290 auth_method 0 2023-09-30T02:07:21.877 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.875+0000 7fec15d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:07:21.877 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.875+0000 7fec15d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:07:21.877 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.875+0000 7fec15d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:07:21.878 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.876+0000 7fec15d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:07:21.878 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.876+0000 7fec15d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:07:21.879 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.876+0000 7fec15d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:07:21.879 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.878+0000 7fec1559a700 1 -- 172.21.15.59:0/966399949 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 551432673 0 0) 0x7fec180a75a0 con 0x7fec180a5b70 2023-09-30T02:07:21.879 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.878+0000 7fec1559a700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:07:21.879 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.878+0000 7fec1559a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:07:21.879 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.878+0000 7fec1559a700 10 monclient(hunting): my global_id is 4474 2023-09-30T02:07:21.879 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.878+0000 7fec1559a700 1 -- 172.21.15.59:0/966399949 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec00003570 con 0x7fec180a5b70 2023-09-30T02:07:21.879 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.878+0000 7fec15d9b700 10 monclient(hunting): handle_auth_done global_id 4415 payload 274 2023-09-30T02:07:21.879 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.878+0000 7fec1559a700 1 -- 172.21.15.59:0/966399949 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3646073671 0 0) 0x7fec180a6380 con 0x7fec180a5f40 2023-09-30T02:07:21.879 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.878+0000 7fec15d9b700 1 -- 172.21.15.59:0/966399949 >> v1:172.21.15.110:6789/0 conn(0x7fec180a5b70 legacy=0x7fec18141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:21.879 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.878+0000 7fec15d9b700 1 -- 172.21.15.59:0/966399949 >> v1:172.21.15.59:6789/0 conn(0x7fec180a5f40 legacy=0x7fec1813af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:21.879 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.878+0000 7fec15d9b700 10 monclient: _finish_hunting 0 2023-09-30T02:07:21.879 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.878+0000 7fec15d9b700 1 monclient: found mon.noname-b 2023-09-30T02:07:21.880 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.878+0000 7fec15d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:07:21.880 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.879+0000 7fec15d9b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.59:3301/0 2023-09-30T02:07:21.880 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.879+0000 7fec15d9b700 1 -- 172.21.15.59:0/966399949 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec1813b640 con 0x7fec18142290 2023-09-30T02:07:21.880 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.879+0000 7fec1559a700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-30T02:07:21.880 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.881+0000 7fec15d9b700 1 --2- 172.21.15.59:0/966399949 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fec18142290 0x7fec180b1500 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fec0c009910 tx=0x7fec0c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d5b8ad514822474 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:07:21.885 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.885+0000 7fec1559a700 1 -- 172.21.15.59:0/966399949 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec0c00b680 con 0x7fec18142290 2023-09-30T02:07:21.885 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.885+0000 7fec1559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:07:21.885 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.885+0000 7fec1559a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:07:21.885 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.885+0000 7fec1559a700 10 monclient: dump: 2023-09-30T02:07:21.886 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:07:21.886 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:21.886 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:21.886 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:21.886 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:07:21.886 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:21.886 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:07:21.886 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:07:21.886 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:07:21.886 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:21.886 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.885+0000 7fec1559a700 10 monclient: _finish_auth 0 2023-09-30T02:07:21.886 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.885+0000 7fec1559a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:07:21.887 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.885+0000 7fec1559a700 1 -- 172.21.15.59:0/966399949 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec0c00b7e0 con 0x7fec18142290 2023-09-30T02:07:21.887 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.885+0000 7fec1ceff700 5 monclient: authenticate success, global_id 4415 2023-09-30T02:07:21.887 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.885+0000 7fec1ceff700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:07:21.887 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.885+0000 7fec1559a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:07:21.887 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.885+0000 7fec1ceff700 10 monclient: get_monmap_and_config success 2023-09-30T02:07:21.887 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.885+0000 7fec1ceff700 4 set_mon_vals no callback set 2023-09-30T02:07:21.887 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.885+0000 7fec1559a700 1 -- 172.21.15.59:0/966399949 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec0c00bba0 con 0x7fec18142290 2023-09-30T02:07:21.887 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.885+0000 7fec1559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:07:21.887 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.885+0000 7fec1559a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:07:21.887 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.885+0000 7fec1559a700 10 monclient: dump: 2023-09-30T02:07:21.887 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:07:21.887 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:21.887 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:21.888 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:21.888 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:07:21.888 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:21.888 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:07:21.888 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:07:21.888 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:07:21.888 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:21.888 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.886+0000 7fec1ceff700 10 monclient: shutdown 2023-09-30T02:07:21.888 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.886+0000 7fec1ceff700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:07:21.888 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.886+0000 7fec1ceff700 1 -- 172.21.15.59:0/966399949 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fec18142290 msgr2=0x7fec180b1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:21.888 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.886+0000 7fec1ceff700 1 --2- 172.21.15.59:0/966399949 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fec18142290 0x7fec180b1500 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fec0c009910 tx=0x7fec0c0091a0 comp rx=0 tx=0).stop 2023-09-30T02:07:21.888 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.886+0000 7fec1ceff700 1 -- 172.21.15.59:0/966399949 shutdown_connections 2023-09-30T02:07:21.888 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.886+0000 7fec1ceff700 1 --2- 172.21.15.59:0/966399949 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fec18142290 0x7fec180b1500 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:21.889 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.886+0000 7fec1ceff700 1 -- 172.21.15.59:0/966399949 >> 172.21.15.59:0/966399949 conn(0x7fec180631f0 msgr2=0x7fec180635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:07:21.889 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.886+0000 7fec1ceff700 1 -- 172.21.15.59:0/966399949 shutdown_connections 2023-09-30T02:07:21.889 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.886+0000 7fec1ceff700 1 -- 172.21.15.59:0/966399949 wait complete. 2023-09-30T02:07:21.891 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.887+0000 7fec1ceff700 10 monclient: build_initial_monmap 2023-09-30T02:07:21.891 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.887+0000 7fec1ceff700 10 monclient: monmap: 2023-09-30T02:07:21.891 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:07:21.892 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:07:21.892 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:07:21.892 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:07:21.892 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:07:21.892 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:07:21.892 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:07:21.892 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:07:21.892 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:07:21.892 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:21.892 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.887+0000 7fec1ceff700 1 librados: starting msgr at 2023-09-30T02:07:21.892 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.887+0000 7fec1ceff700 1 librados: starting objecter 2023-09-30T02:07:21.892 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.887+0000 7fec1ceff700 1 Processor -- start 2023-09-30T02:07:21.893 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.887+0000 7fec1ceff700 1 -- start start 2023-09-30T02:07:21.893 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.887+0000 7fec1ceff700 1 librados: setting wanted keys 2023-09-30T02:07:21.893 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.887+0000 7fec1ceff700 1 librados: calling monclient init 2023-09-30T02:07:21.893 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.887+0000 7fec1ceff700 10 monclient: init 2023-09-30T02:07:21.893 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.888+0000 7fec1ceff700 10 monclient: _reopen_session rank -1 2023-09-30T02:07:21.893 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.888+0000 7fec1ceff700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T02:07:21.893 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.888+0000 7fec1ceff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec180a5b70 0x7fec18144c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:21.893 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.888+0000 7fec1ceff700 10 monclient(hunting): picked mon.noname-a con 0x7fec180a5b70 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:07:21.893 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.888+0000 7fec1ceff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fec180a5f40 0x7fec1814de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:21.893 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.888+0000 7fec1ceff700 10 monclient(hunting): picked mon.noname-b con 0x7fec180a5f40 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:07:21.893 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.888+0000 7fec1ceff700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fec18142290 0x7fec1814e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:21.893 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.888+0000 7fec1ceff700 10 monclient(hunting): picked mon.noname-c con 0x7fec18142290 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:07:21.894 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.888+0000 7fec1ceff700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:21.894 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.888+0000 7fec1ceff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec1814e890 con 0x7fec180a5b70 2023-09-30T02:07:21.894 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.888+0000 7fec1ceff700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:21.894 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.888+0000 7fec1ceff700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec1813c0d0 con 0x7fec180a5f40 2023-09-30T02:07:21.894 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.888+0000 7fec1ceff700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:21.894 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.888+0000 7fec1ceff700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec1813d950 con 0x7fec18142290 2023-09-30T02:07:21.894 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.888+0000 7fec1ceff700 10 monclient(hunting): _renew_subs 2023-09-30T02:07:21.894 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.888+0000 7fec1ceff700 10 monclient(hunting): authenticate will time out at 1887.190135s 2023-09-30T02:07:21.894 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.888+0000 7fec15d9b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fec180a5f40 0x7fec1814de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:21.894 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.888+0000 7fec1659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec180a5b70 0x7fec18144c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:21.894 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.889+0000 7fec15d9b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fec180a5f40 0x7fec1814de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:43414/0 (socket says 172.21.15.59:43414) 2023-09-30T02:07:21.894 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.889+0000 7fec15d9b700 1 -- 172.21.15.59:0/2866075842 learned_addr learned my addr 172.21.15.59:0/2866075842 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:07:21.895 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.889+0000 7fec15d9b700 10 monclient(hunting): get_auth_request con 0x7fec180a5f40 auth_method 0 2023-09-30T02:07:21.895 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.889+0000 7fec15d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:07:21.895 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.889+0000 7fec15d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:07:21.895 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.889+0000 7fec15d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:07:21.895 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.889+0000 7fec1659c700 10 monclient(hunting): get_auth_request con 0x7fec180a5b70 auth_method 0 2023-09-30T02:07:21.895 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.889+0000 7fec1659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:07:21.895 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.889+0000 7fec1659c700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:07:21.895 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.889+0000 7fec1659c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:07:21.895 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.889+0000 7fec15d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:07:21.895 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.889+0000 7fec15d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:07:21.895 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.889+0000 7fec15d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:07:21.895 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.890+0000 7fec16d9d700 1 --2- 172.21.15.59:0/2866075842 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fec18142290 0x7fec1814e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:21.895 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.890+0000 7fec1659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:07:21.896 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.890+0000 7fec1659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:07:21.896 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.890+0000 7fec1659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:07:21.896 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.890+0000 7fec16d9d700 10 monclient(hunting): get_auth_request con 0x7fec18142290 auth_method 0 2023-09-30T02:07:21.896 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.890+0000 7fec16d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:07:21.896 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.890+0000 7fec16d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:07:21.896 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.890+0000 7fec16d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:07:21.896 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.891+0000 7fec16d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:07:21.896 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.891+0000 7fec16d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:07:21.896 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.891+0000 7fec16d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:07:21.897 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.893+0000 7fec15d9b700 10 monclient(hunting): handle_auth_done global_id 4477 payload 824 2023-09-30T02:07:21.897 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.893+0000 7fec15d9b700 1 -- 172.21.15.59:0/2866075842 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fec18142290 msgr2=0x7fec1814e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:21.897 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.893+0000 7fec15d9b700 1 --2- 172.21.15.59:0/2866075842 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fec18142290 0x7fec1814e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:21.897 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.893+0000 7fec15d9b700 1 -- 172.21.15.59:0/2866075842 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec180a5b70 msgr2=0x7fec18144c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:21.897 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.893+0000 7fec15d9b700 1 --2- 172.21.15.59:0/2866075842 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec180a5b70 0x7fec18144c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:21.897 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.893+0000 7fec15d9b700 10 monclient: _finish_hunting 0 2023-09-30T02:07:21.897 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.893+0000 7fec15d9b700 1 monclient: found mon.noname-b 2023-09-30T02:07:21.897 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.893+0000 7fec15d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:07:21.898 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.893+0000 7fec15d9b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:07:21.898 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.893+0000 7fec15d9b700 1 -- 172.21.15.59:0/2866075842 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec0c009420 con 0x7fec180a5f40 2023-09-30T02:07:21.898 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.896+0000 7fec15d9b700 1 --2- 172.21.15.59:0/2866075842 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fec180a5f40 0x7fec1814de10 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fec180a6510 tx=0x7fec0c017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:07:21.898 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.896+0000 7fec067fc700 10 client.?.objecter ms_handle_connect 0x7fec180a5f40 2023-09-30T02:07:21.898 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.896+0000 7fec067fc700 10 client.?.objecter resend_mon_ops 2023-09-30T02:07:21.899 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.899+0000 7fec067fc700 1 -- 172.21.15.59:0/2866075842 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec0c00bb60 con 0x7fec180a5f40 2023-09-30T02:07:21.899 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.899+0000 7fec067fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:07:21.899 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.899+0000 7fec067fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:07:21.899 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.899+0000 7fec067fc700 10 monclient: dump: 2023-09-30T02:07:21.899 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:07:21.899 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:21.899 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:21.900 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:21.900 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:07:21.900 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:21.900 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:07:21.900 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:07:21.900 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:07:21.900 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:21.900 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.899+0000 7fec067fc700 10 monclient: _finish_auth 0 2023-09-30T02:07:21.900 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.899+0000 7fec067fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:07:21.900 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.899+0000 7fec1ceff700 5 monclient: authenticate success, global_id 4477 2023-09-30T02:07:21.900 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.899+0000 7fec1ceff700 10 monclient: _renew_subs 2023-09-30T02:07:21.901 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.899+0000 7fec1ceff700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:07:21.901 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.899+0000 7fec1ceff700 1 -- 172.21.15.59:0/2866075842 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec1813c5f0 con 0x7fec180a5f40 2023-09-30T02:07:21.901 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.899+0000 7fec1ceff700 10 client.4477.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:07:21.901 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.899+0000 7fec1ceff700 10 monclient: _renew_subs 2023-09-30T02:07:21.901 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.899+0000 7fec1ceff700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:07:21.901 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.899+0000 7fec1ceff700 1 -- 172.21.15.59:0/2866075842 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec1813ca10 con 0x7fec180a5f40 2023-09-30T02:07:21.901 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.899+0000 7fec1ceff700 1 librados: init done 2023-09-30T02:07:21.901 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.900+0000 7fec1ceff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:07:21.901 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.900+0000 7fec1ceff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:07:21.901 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.900+0000 7fec1ceff700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:07:21.902 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.900+0000 7fec1ceff700 1 -- 172.21.15.59:0/2866075842 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec18064a00 con 0x7fec180a5f40 2023-09-30T02:07:21.902 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.901+0000 7fec067fc700 1 -- 172.21.15.59:0/2866075842 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec0c006bc0 con 0x7fec180a5f40 2023-09-30T02:07:21.902 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.901+0000 7fec067fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:07:21.902 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.901+0000 7fec067fc700 1 -- 172.21.15.59:0/2866075842 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec0c006dd0 con 0x7fec180a5f40 2023-09-30T02:07:21.903 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.901+0000 7fec067fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:07:21.903 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.901+0000 7fec067fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:07:21.903 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.901+0000 7fec067fc700 10 monclient: dump: 2023-09-30T02:07:21.903 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:07:21.903 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:21.903 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:21.903 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:21.903 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:07:21.903 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:21.903 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:07:21.903 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:07:21.904 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:07:21.904 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:21.904 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:21.901+0000 7fec14d99700 4 set_mon_vals no callback set 2023-09-30T02:07:22.104 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:22.105+0000 7fec067fc700 1 -- 172.21.15.59:0/2866075842 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fec0c027660 con 0x7fec180a5f40 2023-09-30T02:07:22.105 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:22.105+0000 7fec067fc700 1 --2- 172.21.15.59:0/2866075842 >> v2:172.21.15.110:6800/115870 conn(0x7febf8068890 0x7febf806ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:22.105 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:22.105+0000 7fec067fc700 1 -- 172.21.15.59:0/2866075842 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fec0c085d80 con 0x7fec180a5f40 2023-09-30T02:07:22.105 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:22.105+0000 7fec067fc700 10 client.4477.objecter ms_dispatch 0x7fec18001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:07:22.105 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:22.105+0000 7fec067fc700 3 client.4477.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:07:22.106 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:22.105+0000 7fec067fc700 3 client.4477.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:07:22.106 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:22.105+0000 7fec067fc700 20 client.4477.objecter dump_active .. 0 homeless 2023-09-30T02:07:22.106 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:22.105+0000 7fec1659c700 1 --2- 172.21.15.59:0/2866075842 >> v2:172.21.15.110:6800/115870 conn(0x7febf8068890 0x7febf806ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:22.106 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:22.105+0000 7fec1659c700 10 monclient: get_auth_request con 0x7febf8068890 auth_method 0 2023-09-30T02:07:22.106 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:22.106+0000 7fec1659c700 1 --2- 172.21.15.59:0/2866075842 >> v2:172.21.15.110:6800/115870 conn(0x7febf8068890 0x7febf806ad40 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fec08007fd0 tx=0x7fec08009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:07:22.106 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:22.106+0000 7fec067fc700 10 client.4477.objecter ms_handle_connect 0x7febf8068890 2023-09-30T02:07:22.169 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:22.170+0000 7fec067fc700 1 -- 172.21.15.59:0/2866075842 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fec0c0535a0 con 0x7fec180a5f40 2023-09-30T02:07:22.169 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:22.170+0000 7fec067fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:07:22.169 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:22.170+0000 7fec067fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:07:22.425 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:22.426+0000 7fec1ceff700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:07:22.425 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:22.426+0000 7fec1ceff700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:07:22.425 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:22.426+0000 7fec1ceff700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:07:22.426 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:22.426+0000 7fec1ceff700 1 -- 172.21.15.59:0/2866075842 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fec1806a720 con 0x7fec180a5f40 2023-09-30T02:07:22.429 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:22.430+0000 7fec067fc700 1 -- 172.21.15.59:0/2866075842 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fec0c015100 con 0x7fec180a5f40 2023-09-30T02:07:22.429 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:22.430+0000 7fec067fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:07:22.429 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:22.430+0000 7fec067fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:07:22.430 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:07:22.430 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:07:22.432 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:22.432+0000 7fec1ceff700 10 librados: watch_flush enter 2023-09-30T02:07:22.432 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:22.432+0000 7fec1ceff700 10 librados: watch_flush exit 2023-09-30T02:07:22.432 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:22.432+0000 7fec1ceff700 20 client.4477.objecter shutdown clearing up homeless session... 2023-09-30T02:07:22.433 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:22.432+0000 7fec1ceff700 10 client.4477.objecter successfully canceled tick 2023-09-30T02:07:22.433 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:22.432+0000 7fec1ceff700 1 -- 172.21.15.59:0/2866075842 >> v2:172.21.15.110:6800/115870 conn(0x7febf8068890 msgr2=0x7febf806ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:22.433 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:22.432+0000 7fec1ceff700 1 --2- 172.21.15.59:0/2866075842 >> v2:172.21.15.110:6800/115870 conn(0x7febf8068890 0x7febf806ad40 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fec08007fd0 tx=0x7fec08009040 comp rx=0 tx=0).stop 2023-09-30T02:07:22.433 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:22.432+0000 7fec1ceff700 10 monclient: shutdown 2023-09-30T02:07:22.433 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:22.432+0000 7fec1ceff700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:07:22.433 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:22.432+0000 7fec1ceff700 1 -- 172.21.15.59:0/2866075842 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fec180a5f40 msgr2=0x7fec1814de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:22.433 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:22.432+0000 7fec1ceff700 1 --2- 172.21.15.59:0/2866075842 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fec180a5f40 0x7fec1814de10 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fec180a6510 tx=0x7fec0c017000 comp rx=0 tx=0).stop 2023-09-30T02:07:22.433 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:22.432+0000 7fec1ceff700 1 -- 172.21.15.59:0/2866075842 shutdown_connections 2023-09-30T02:07:22.433 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:22.432+0000 7fec1ceff700 1 --2- 172.21.15.59:0/2866075842 >> v2:172.21.15.110:6800/115870 conn(0x7febf8068890 0x7febf806ad40 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:22.433 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:22.432+0000 7fec1ceff700 1 --2- 172.21.15.59:0/2866075842 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fec18142290 0x7fec1814e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:22.433 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:22.432+0000 7fec1ceff700 1 --2- 172.21.15.59:0/2866075842 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec180a5b70 0x7fec18144c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:22.434 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:22.432+0000 7fec1ceff700 1 --2- 172.21.15.59:0/2866075842 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fec180a5f40 0x7fec1814de10 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:22.434 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:22.433+0000 7fec1ceff700 1 -- 172.21.15.59:0/2866075842 >> 172.21.15.59:0/2866075842 conn(0x7fec180631f0 msgr2=0x7fec18070f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:07:22.434 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:22.433+0000 7fec1ceff700 1 -- 172.21.15.59:0/2866075842 shutdown_connections 2023-09-30T02:07:22.434 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:22.433+0000 7fec1ceff700 1 -- 172.21.15.59:0/2866075842 wait complete. 2023-09-30T02:07:22.434 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:22.433+0000 7fec1ceff700 1 librados: shutdown 2023-09-30T02:07:25.447 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:07:25.585 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.586+0000 7f8962bef700 10 monclient: get_monmap_and_config 2023-09-30T02:07:25.586 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.586+0000 7f8962bef700 10 monclient: build_initial_monmap 2023-09-30T02:07:25.586 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.586+0000 7f8962bef700 10 monclient: monmap: 2023-09-30T02:07:25.586 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:07:25.586 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:25.586 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:07:25.586647+0000 2023-09-30T02:07:25.586 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:07:25.586647+0000 2023-09-30T02:07:25.586 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:07:25.586 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:25.586 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:07:25.586 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:07:25.586 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:07:25.587 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:07:25.587 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:07:25.587 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:25.589 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.587+0000 7f8962bef700 1 Processor -- start 2023-09-30T02:07:25.589 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.587+0000 7f8962bef700 1 -- start start 2023-09-30T02:07:25.589 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.587+0000 7f8962bef700 10 monclient: init 2023-09-30T02:07:25.589 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.587+0000 7f8962bef700 10 monclient: _reopen_session rank -1 2023-09-30T02:07:25.589 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.587+0000 7f8962bef700 10 monclient: _add_conns ranks=[3,4,2,1,0] 2023-09-30T02:07:25.589 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.587+0000 7f8962bef700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f895c0a5b70 0x7f895c0b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:25.589 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.587+0000 7f8962bef700 10 monclient(hunting): picked mon.noname-a con 0x7f895c0a5b70 addr v2:172.21.15.59:3300/0 2023-09-30T02:07:25.589 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.587+0000 7f8962bef700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f895c0a5f40 0x7f895c141b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:25.589 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.587+0000 7f8962bef700 10 monclient(hunting): picked mon.noname-c con 0x7f895c0a5f40 addr v2:172.21.15.110:3300/0 2023-09-30T02:07:25.590 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.587+0000 7f8962bef700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f895c1420c0 0x7f895c13e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:25.590 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.587+0000 7f8962bef700 10 monclient(hunting): picked mon.noname-b con 0x7f895c1420c0 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:07:25.590 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.587+0000 7f8962bef700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:25.590 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.587+0000 7f8962bef700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f895c0a63e0 con 0x7f895c0a5b70 2023-09-30T02:07:25.590 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.587+0000 7f8962bef700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:25.590 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.587+0000 7f8962bef700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f895c0a75a0 con 0x7f895c0a5f40 2023-09-30T02:07:25.590 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.587+0000 7f8962bef700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:25.590 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.587+0000 7f8962bef700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f895c0af3f0 con 0x7f895c1420c0 2023-09-30T02:07:25.590 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.587+0000 7f8962bef700 10 monclient(hunting): _renew_subs 2023-09-30T02:07:25.590 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.587+0000 7f8962bef700 10 monclient(hunting): authenticate will time out at 1890.889835s 2023-09-30T02:07:25.590 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.588+0000 7f896118c700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f895c1420c0 0x7f895c13e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:25.591 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.588+0000 7f896098b700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f895c0a5b70 0x7f895c0b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:25.591 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.588+0000 7f895bfff700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f895c0a5f40 0x7f895c141b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:25.591 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.589+0000 7f896118c700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f895c1420c0 0x7f895c13e230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:51668/0 (socket says 172.21.15.59:51668) 2023-09-30T02:07:25.591 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.589+0000 7f896118c700 1 -- 172.21.15.59:0/3896027563 learned_addr learned my addr 172.21.15.59:0/3896027563 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:07:25.591 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.589+0000 7f896118c700 10 monclient(hunting): get_auth_request con 0x7f895c1420c0 auth_method 0 2023-09-30T02:07:25.591 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.589+0000 7f896118c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:07:25.591 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.589+0000 7f896118c700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:07:25.591 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.589+0000 7f896118c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:07:25.591 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.589+0000 7f896098b700 10 monclient(hunting): get_auth_request con 0x7f895c0a5b70 auth_method 0 2023-09-30T02:07:25.591 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.589+0000 7f896098b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:07:25.591 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.589+0000 7f896098b700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:07:25.591 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.589+0000 7f896098b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:07:25.591 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.589+0000 7f895bfff700 10 monclient(hunting): get_auth_request con 0x7f895c0a5f40 auth_method 0 2023-09-30T02:07:25.592 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.589+0000 7f895bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:07:25.592 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.589+0000 7f895bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:07:25.592 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.589+0000 7f895bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:07:25.592 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.589+0000 7f896118c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:07:25.592 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.589+0000 7f896118c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:07:25.592 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.589+0000 7f896118c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:07:25.592 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.589+0000 7f896098b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:07:25.592 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.589+0000 7f896098b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:07:25.592 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.589+0000 7f896098b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:07:25.592 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.590+0000 7f895bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:07:25.592 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.590+0000 7f895bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:07:25.592 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.590+0000 7f895bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:07:25.593 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.591+0000 7f896118c700 10 monclient(hunting): handle_auth_done global_id 4421 payload 274 2023-09-30T02:07:25.593 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.591+0000 7f896118c700 1 -- 172.21.15.59:0/3896027563 >> v2:172.21.15.110:3300/0 conn(0x7f895c0a5f40 msgr2=0x7f895c141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:07:25.593 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.591+0000 7f896118c700 1 --2- 172.21.15.59:0/3896027563 >> v2:172.21.15.110:3300/0 conn(0x7f895c0a5f40 0x7f895c141b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:25.594 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.591+0000 7f896118c700 1 -- 172.21.15.59:0/3896027563 >> v2:172.21.15.59:3300/0 conn(0x7f895c0a5b70 msgr2=0x7f895c0b1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:07:25.594 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.592+0000 7f896118c700 1 --2- 172.21.15.59:0/3896027563 >> v2:172.21.15.59:3300/0 conn(0x7f895c0a5b70 0x7f895c0b1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:25.594 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.592+0000 7f896118c700 10 monclient: _finish_hunting 0 2023-09-30T02:07:25.594 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.592+0000 7f896118c700 1 monclient: found mon.noname-b 2023-09-30T02:07:25.594 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.592+0000 7f896118c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:07:25.594 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.592+0000 7f896118c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.59:3301/0 2023-09-30T02:07:25.594 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.592+0000 7f896118c700 1 -- 172.21.15.59:0/3896027563 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f895c142500 con 0x7f895c1420c0 2023-09-30T02:07:25.594 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.593+0000 7f896118c700 1 --2- 172.21.15.59:0/3896027563 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f895c1420c0 0x7f895c13e230 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f895c0b1fc0 tx=0x7f8954009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=771555bdd70cc551 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:07:25.597 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.598+0000 7f895b7fe700 1 -- 172.21.15.59:0/3896027563 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f895400e070 con 0x7f895c1420c0 2023-09-30T02:07:25.597 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.598+0000 7f895b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:07:25.597 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.598+0000 7f895b7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:07:25.598 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.598+0000 7f895b7fe700 10 monclient: dump: 2023-09-30T02:07:25.598 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:07:25.598 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:25.598 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:25.598 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:25.598 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:07:25.598 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:25.598 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:07:25.598 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:07:25.598 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:07:25.598 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:25.598 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.598+0000 7f895b7fe700 10 monclient: _finish_auth 0 2023-09-30T02:07:25.599 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.598+0000 7f895b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:07:25.599 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.598+0000 7f895b7fe700 1 -- 172.21.15.59:0/3896027563 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f895400b970 con 0x7f895c1420c0 2023-09-30T02:07:25.599 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.598+0000 7f895b7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:07:25.599 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.598+0000 7f895b7fe700 1 -- 172.21.15.59:0/3896027563 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89540092e0 con 0x7f895c1420c0 2023-09-30T02:07:25.599 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.598+0000 7f8962bef700 5 monclient: authenticate success, global_id 4421 2023-09-30T02:07:25.599 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.598+0000 7f8962bef700 10 monclient: get_monmap_and_config success 2023-09-30T02:07:25.599 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.598+0000 7f895b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:07:25.599 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.598+0000 7f8962bef700 4 set_mon_vals no callback set 2023-09-30T02:07:25.599 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.598+0000 7f895b7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:07:25.599 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.598+0000 7f895b7fe700 10 monclient: dump: 2023-09-30T02:07:25.599 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:07:25.599 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:25.600 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:25.600 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:25.600 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:07:25.600 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:25.600 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:07:25.600 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:07:25.600 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:07:25.600 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:25.603 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.599+0000 7f8962bef700 10 monclient: shutdown 2023-09-30T02:07:25.603 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.599+0000 7f8962bef700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:07:25.603 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.599+0000 7f8962bef700 1 -- 172.21.15.59:0/3896027563 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f895c1420c0 msgr2=0x7f895c13e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:25.603 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.599+0000 7f8962bef700 1 --2- 172.21.15.59:0/3896027563 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f895c1420c0 0x7f895c13e230 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f895c0b1fc0 tx=0x7f8954009840 comp rx=0 tx=0).stop 2023-09-30T02:07:25.603 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.599+0000 7f8962bef700 1 -- 172.21.15.59:0/3896027563 shutdown_connections 2023-09-30T02:07:25.603 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.599+0000 7f8962bef700 1 --2- 172.21.15.59:0/3896027563 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f895c1420c0 0x7f895c13e230 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:25.603 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.599+0000 7f8962bef700 1 --2- 172.21.15.59:0/3896027563 >> v2:172.21.15.59:3300/0 conn(0x7f895c0a5b70 0x7f895c0b1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:25.603 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.599+0000 7f8962bef700 1 --2- 172.21.15.59:0/3896027563 >> v2:172.21.15.110:3300/0 conn(0x7f895c0a5f40 0x7f895c141b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:25.603 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.599+0000 7f8962bef700 1 -- 172.21.15.59:0/3896027563 >> 172.21.15.59:0/3896027563 conn(0x7f895c0631f0 msgr2=0x7f895c0635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:07:25.603 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.599+0000 7f8962bef700 1 -- 172.21.15.59:0/3896027563 shutdown_connections 2023-09-30T02:07:25.603 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.599+0000 7f8962bef700 1 -- 172.21.15.59:0/3896027563 wait complete. 2023-09-30T02:07:25.604 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.599+0000 7f8962bef700 10 monclient: build_initial_monmap 2023-09-30T02:07:25.604 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.599+0000 7f8962bef700 10 monclient: monmap: 2023-09-30T02:07:25.604 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:07:25.604 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:07:25.604 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:07:25.604 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:07:25.604 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:07:25.604 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:07:25.604 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:07:25.604 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:07:25.604 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:07:25.604 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:25.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.600+0000 7f8962bef700 1 librados: starting msgr at 2023-09-30T02:07:25.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.600+0000 7f8962bef700 1 librados: starting objecter 2023-09-30T02:07:25.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.600+0000 7f8962bef700 1 Processor -- start 2023-09-30T02:07:25.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.600+0000 7f8962bef700 1 -- start start 2023-09-30T02:07:25.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.600+0000 7f8962bef700 1 librados: setting wanted keys 2023-09-30T02:07:25.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.600+0000 7f8962bef700 1 librados: calling monclient init 2023-09-30T02:07:25.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.600+0000 7f8962bef700 10 monclient: init 2023-09-30T02:07:25.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.601+0000 7f8962bef700 10 monclient: _reopen_session rank -1 2023-09-30T02:07:25.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.601+0000 7f8962bef700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T02:07:25.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.601+0000 7f8962bef700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f895c0a5b70 0x7f895c13ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:25.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.601+0000 7f8962bef700 10 monclient(hunting): picked mon.noname-b con 0x7f895c0a5b70 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:07:25.606 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.601+0000 7f8962bef700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f895c0a5f40 0x7f895c0a9290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:25.606 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.601+0000 7f8962bef700 10 monclient(hunting): picked mon.noname-a con 0x7f895c0a5f40 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:07:25.606 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.601+0000 7f8962bef700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f895c1420c0 0x7f895c0adc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:25.606 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.601+0000 7f8962bef700 10 monclient(hunting): picked mon.noname-c con 0x7f895c1420c0 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:07:25.606 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.601+0000 7f8962bef700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:25.606 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.601+0000 7f8962bef700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f895c13ba00 con 0x7f895c0a5f40 2023-09-30T02:07:25.606 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.601+0000 7f8962bef700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:25.606 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.601+0000 7f8962bef700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f895c0aa3b0 con 0x7f895c0a5b70 2023-09-30T02:07:25.606 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.601+0000 7f8962bef700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:25.606 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.601+0000 7f8962bef700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f895c0a9f20 con 0x7f895c1420c0 2023-09-30T02:07:25.606 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.601+0000 7f8962bef700 10 monclient(hunting): _renew_subs 2023-09-30T02:07:25.607 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.601+0000 7f8962bef700 10 monclient(hunting): authenticate will time out at 1890.903776s 2023-09-30T02:07:25.607 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.602+0000 7f896118c700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f895c1420c0 0x7f895c0adc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:25.607 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.602+0000 7f896118c700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f895c1420c0 0x7f895c0adc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:51674/0 (socket says 172.21.15.59:51674) 2023-09-30T02:07:25.607 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.602+0000 7f896118c700 1 -- 172.21.15.59:0/1095505447 learned_addr learned my addr 172.21.15.59:0/1095505447 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:07:25.607 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.602+0000 7f896118c700 10 monclient(hunting): get_auth_request con 0x7f895c1420c0 auth_method 0 2023-09-30T02:07:25.607 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.602+0000 7f896118c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:07:25.607 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.602+0000 7f896118c700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:07:25.607 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.602+0000 7f896118c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:07:25.607 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.603+0000 7f896098b700 1 --2- 172.21.15.59:0/1095505447 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f895c0a5b70 0x7f895c13ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:25.607 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.603+0000 7f896118c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:07:25.607 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.603+0000 7f896118c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:07:25.607 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.603+0000 7f896118c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:07:25.608 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.603+0000 7f896098b700 10 monclient(hunting): get_auth_request con 0x7f895c0a5b70 auth_method 0 2023-09-30T02:07:25.608 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.603+0000 7f896098b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:07:25.608 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.603+0000 7f896098b700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:07:25.608 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.603+0000 7f896098b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:07:25.608 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.603+0000 7f896098b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:07:25.608 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.603+0000 7f896098b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:07:25.608 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.604+0000 7f896098b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:07:25.609 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.606+0000 7f896118c700 10 monclient(hunting): handle_auth_done global_id 4424 payload 824 2023-09-30T02:07:25.609 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.606+0000 7f896118c700 1 -- 172.21.15.59:0/1095505447 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f895c0a5b70 msgr2=0x7f895c13ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:25.609 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.606+0000 7f896118c700 1 --2- 172.21.15.59:0/1095505447 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f895c0a5b70 0x7f895c13ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:25.609 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.607+0000 7f896118c700 1 -- 172.21.15.59:0/1095505447 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f895c0a5f40 msgr2=0x7f895c0a9290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:25.609 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.607+0000 7f896118c700 1 --2- 172.21.15.59:0/1095505447 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f895c0a5f40 0x7f895c0a9290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:25.609 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.607+0000 7f896118c700 10 monclient: _finish_hunting 0 2023-09-30T02:07:25.609 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.607+0000 7f896118c700 1 monclient: found mon.noname-c 2023-09-30T02:07:25.610 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.607+0000 7f896118c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:07:25.610 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.607+0000 7f896118c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.59:3301/0 2023-09-30T02:07:25.610 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.607+0000 7f896118c700 1 -- 172.21.15.59:0/1095505447 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8954009ae0 con 0x7f895c1420c0 2023-09-30T02:07:25.610 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.609+0000 7f896118c700 1 --2- 172.21.15.59:0/1095505447 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f895c1420c0 0x7f895c0adc00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f895c0af530 tx=0x7f8954018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:07:25.610 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.609+0000 7f8958ff9700 10 client.?.objecter ms_handle_connect 0x7f895c1420c0 2023-09-30T02:07:25.610 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.609+0000 7f8958ff9700 10 client.?.objecter resend_mon_ops 2023-09-30T02:07:25.614 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.614+0000 7f8958ff9700 1 -- 172.21.15.59:0/1095505447 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89540094e0 con 0x7f895c1420c0 2023-09-30T02:07:25.614 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.614+0000 7f8958ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:07:25.614 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.614+0000 7f8958ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:07:25.614 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.614+0000 7f8958ff9700 10 monclient: dump: 2023-09-30T02:07:25.614 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:07:25.614 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:25.614 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:25.614 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:25.614 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:07:25.614 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:25.614 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:07:25.615 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:07:25.615 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:07:25.615 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:25.615 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.614+0000 7f8958ff9700 10 monclient: _finish_auth 0 2023-09-30T02:07:25.615 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.614+0000 7f8958ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:07:25.615 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.614+0000 7f8962bef700 5 monclient: authenticate success, global_id 4424 2023-09-30T02:07:25.615 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.614+0000 7f8958ff9700 1 -- 172.21.15.59:0/1095505447 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89540032f0 con 0x7f895c1420c0 2023-09-30T02:07:25.615 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.614+0000 7f8958ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:07:25.615 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.614+0000 7f8958ff9700 1 -- 172.21.15.59:0/1095505447 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8954008e10 con 0x7f895c1420c0 2023-09-30T02:07:25.615 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.614+0000 7f8958ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:07:25.615 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.614+0000 7f8958ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:07:25.615 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.614+0000 7f8958ff9700 10 monclient: dump: 2023-09-30T02:07:25.616 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:07:25.616 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:25.616 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:25.616 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:25.616 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:07:25.616 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:25.616 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:07:25.616 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:07:25.616 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:07:25.616 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:25.616 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.614+0000 7f895affd700 4 set_mon_vals no callback set 2023-09-30T02:07:25.617 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.614+0000 7f8962bef700 10 monclient: _renew_subs 2023-09-30T02:07:25.617 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.614+0000 7f8962bef700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:07:25.617 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.614+0000 7f8962bef700 1 -- 172.21.15.59:0/1095505447 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f895c0a9950 con 0x7f895c1420c0 2023-09-30T02:07:25.617 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.614+0000 7f8962bef700 10 client.4424.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:07:25.617 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.614+0000 7f8962bef700 10 monclient: _renew_subs 2023-09-30T02:07:25.617 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.614+0000 7f8962bef700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:07:25.617 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.614+0000 7f8962bef700 1 -- 172.21.15.59:0/1095505447 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f895c0ae140 con 0x7f895c1420c0 2023-09-30T02:07:25.617 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.614+0000 7f8962bef700 1 librados: init done 2023-09-30T02:07:25.618 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.615+0000 7f8962bef700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:07:25.618 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.615+0000 7f8962bef700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:07:25.618 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.615+0000 7f8962bef700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:07:25.618 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.615+0000 7f8962bef700 1 -- 172.21.15.59:0/1095505447 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f895c0aa0b0 con 0x7f895c1420c0 2023-09-30T02:07:25.914 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.914+0000 7f8958ff9700 1 -- 172.21.15.59:0/1095505447 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f8954003040 con 0x7f895c1420c0 2023-09-30T02:07:25.914 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.914+0000 7f8958ff9700 1 --2- 172.21.15.59:0/1095505447 >> v2:172.21.15.110:6800/115870 conn(0x7f8948068960 0x7f894806ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:25.914 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.914+0000 7f8958ff9700 1 -- 172.21.15.59:0/1095505447 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f8954085c80 con 0x7f895c1420c0 2023-09-30T02:07:25.914 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.914+0000 7f8958ff9700 10 client.4424.objecter ms_dispatch 0x7f895c001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:07:25.914 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.914+0000 7f8958ff9700 3 client.4424.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:07:25.915 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.914+0000 7f8958ff9700 3 client.4424.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:07:25.915 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.914+0000 7f8958ff9700 20 client.4424.objecter dump_active .. 0 homeless 2023-09-30T02:07:25.915 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.914+0000 7f8958ff9700 1 -- 172.21.15.59:0/1095505447 <== mon.2 v2:172.21.15.59:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f89540860d0 con 0x7f895c1420c0 2023-09-30T02:07:25.915 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.915+0000 7f8958ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:07:25.915 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.915+0000 7f8958ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:07:25.915 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.915+0000 7f896098b700 1 --2- 172.21.15.59:0/1095505447 >> v2:172.21.15.110:6800/115870 conn(0x7f8948068960 0x7f894806ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:25.915 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.915+0000 7f896098b700 10 monclient: get_auth_request con 0x7f8948068960 auth_method 0 2023-09-30T02:07:25.915 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.915+0000 7f896098b700 1 --2- 172.21.15.59:0/1095505447 >> v2:172.21.15.110:6800/115870 conn(0x7f8948068960 0x7f894806ae10 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f894c009db0 tx=0x7f894c0091a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:07:25.915 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:25.915+0000 7f8958ff9700 10 client.4424.objecter ms_handle_connect 0x7f8948068960 2023-09-30T02:07:26.177 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:26.178+0000 7f8962bef700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:07:26.178 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:26.178+0000 7f8962bef700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:07:26.178 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:26.178+0000 7f8962bef700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:07:26.178 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:26.178+0000 7f8962bef700 1 -- 172.21.15.59:0/1095505447 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f895c064a00 con 0x7f895c1420c0 2023-09-30T02:07:26.183 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:26.183+0000 7f8958ff9700 1 -- 172.21.15.59:0/1095505447 <== mon.2 v2:172.21.15.59:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f89540535a0 con 0x7f895c1420c0 2023-09-30T02:07:26.183 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:26.183+0000 7f8958ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:07:26.183 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:26.183+0000 7f8958ff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:07:26.183 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:07:26.183 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:07:26.186 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:26.186+0000 7f8962bef700 10 librados: watch_flush enter 2023-09-30T02:07:26.186 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:26.186+0000 7f8962bef700 10 librados: watch_flush exit 2023-09-30T02:07:26.186 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:26.186+0000 7f8962bef700 20 client.4424.objecter shutdown clearing up homeless session... 2023-09-30T02:07:26.186 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:26.186+0000 7f8962bef700 10 client.4424.objecter successfully canceled tick 2023-09-30T02:07:26.186 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:26.186+0000 7f8962bef700 1 -- 172.21.15.59:0/1095505447 >> v2:172.21.15.110:6800/115870 conn(0x7f8948068960 msgr2=0x7f894806ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:26.186 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:26.186+0000 7f8962bef700 1 --2- 172.21.15.59:0/1095505447 >> v2:172.21.15.110:6800/115870 conn(0x7f8948068960 0x7f894806ae10 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f894c009db0 tx=0x7f894c0091a0 comp rx=0 tx=0).stop 2023-09-30T02:07:26.187 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:26.186+0000 7f8962bef700 10 monclient: shutdown 2023-09-30T02:07:26.187 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:26.186+0000 7f8962bef700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:07:26.187 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:26.186+0000 7f8962bef700 1 -- 172.21.15.59:0/1095505447 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f895c1420c0 msgr2=0x7f895c0adc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:26.187 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:26.186+0000 7f8962bef700 1 --2- 172.21.15.59:0/1095505447 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f895c1420c0 0x7f895c0adc00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f895c0af530 tx=0x7f8954018000 comp rx=0 tx=0).stop 2023-09-30T02:07:26.187 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:26.186+0000 7f8962bef700 1 -- 172.21.15.59:0/1095505447 shutdown_connections 2023-09-30T02:07:26.187 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:26.186+0000 7f8962bef700 1 --2- 172.21.15.59:0/1095505447 >> v2:172.21.15.110:6800/115870 conn(0x7f8948068960 0x7f894806ae10 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:26.187 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:26.186+0000 7f8962bef700 1 --2- 172.21.15.59:0/1095505447 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f895c0a5b70 0x7f895c13ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:26.187 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:26.186+0000 7f8962bef700 1 --2- 172.21.15.59:0/1095505447 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f895c1420c0 0x7f895c0adc00 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:26.187 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:26.186+0000 7f8962bef700 1 --2- 172.21.15.59:0/1095505447 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f895c0a5f40 0x7f895c0a9290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:26.187 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:26.186+0000 7f8962bef700 1 -- 172.21.15.59:0/1095505447 >> 172.21.15.59:0/1095505447 conn(0x7f895c0631f0 msgr2=0x7f895c0b0be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:07:26.187 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:26.186+0000 7f8962bef700 1 -- 172.21.15.59:0/1095505447 shutdown_connections 2023-09-30T02:07:26.187 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:26.186+0000 7f8962bef700 1 -- 172.21.15.59:0/1095505447 wait complete. 2023-09-30T02:07:26.187 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:26.186+0000 7f8962bef700 1 librados: shutdown 2023-09-30T02:07:29.201 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:07:29.339 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.339+0000 7fb903623700 10 monclient: get_monmap_and_config 2023-09-30T02:07:29.339 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.339+0000 7fb903623700 10 monclient: build_initial_monmap 2023-09-30T02:07:29.339 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.339+0000 7fb903623700 10 monclient: monmap: 2023-09-30T02:07:29.339 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:07:29.339 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:29.340 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:07:29.340254+0000 2023-09-30T02:07:29.340 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:07:29.340254+0000 2023-09-30T02:07:29.340 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:07:29.340 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:29.340 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:07:29.340 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:07:29.340 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:07:29.340 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:07:29.340 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:07:29.340 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:29.340 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.340+0000 7fb903623700 1 Processor -- start 2023-09-30T02:07:29.342 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.340+0000 7fb903623700 1 -- start start 2023-09-30T02:07:29.342 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.340+0000 7fb903623700 10 monclient: init 2023-09-30T02:07:29.342 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.341+0000 7fb903623700 10 monclient: _reopen_session rank -1 2023-09-30T02:07:29.342 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.341+0000 7fb903623700 10 monclient: _add_conns ranks=[0,3,2,4,1] 2023-09-30T02:07:29.342 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.341+0000 7fb903623700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb8fc0a5b70 addr v1:172.21.15.59:6789/0 2023-09-30T02:07:29.342 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.341+0000 7fb903623700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fb8fc142290 0x7fb8fc0b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:29.342 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.341+0000 7fb903623700 10 monclient(hunting): picked mon.noname-a con 0x7fb8fc142290 addr v2:172.21.15.59:3300/0 2023-09-30T02:07:29.343 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.341+0000 7fb903623700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb8fc0a5f40 0x7fb8fc13af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:29.343 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.341+0000 7fb903623700 10 monclient(hunting): picked mon.noname-b con 0x7fb8fc0a5f40 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:07:29.343 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.341+0000 7fb903623700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8fc0a6380 con 0x7fb8fc0a5b70 2023-09-30T02:07:29.343 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.341+0000 7fb903623700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:29.343 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.341+0000 7fb903623700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8fc0a75a0 con 0x7fb8fc142290 2023-09-30T02:07:29.343 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.341+0000 7fb903623700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:29.343 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.341+0000 7fb903623700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8fc0af3f0 con 0x7fb8fc0a5f40 2023-09-30T02:07:29.343 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.341+0000 7fb903623700 10 monclient(hunting): _renew_subs 2023-09-30T02:07:29.343 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.341+0000 7fb903623700 10 monclient(hunting): authenticate will time out at 1894.643425s 2023-09-30T02:07:29.343 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.342+0000 7fb900bbe700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fb8fc142290 0x7fb8fc0b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:29.343 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.342+0000 7fb901bc0700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb8fc0a5f40 0x7fb8fc13af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:29.344 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.342+0000 7fb901bc0700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb8fc0a5f40 0x7fb8fc13af30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:55272/0 (socket says 172.21.15.59:55272) 2023-09-30T02:07:29.344 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.342+0000 7fb901bc0700 1 -- 172.21.15.59:0/2674535298 learned_addr learned my addr 172.21.15.59:0/2674535298 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:07:29.344 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.342+0000 7fb901bc0700 10 monclient(hunting): get_auth_request con 0x7fb8fc0a5f40 auth_method 0 2023-09-30T02:07:29.344 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.342+0000 7fb901bc0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:07:29.344 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.342+0000 7fb901bc0700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:07:29.344 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.342+0000 7fb901bc0700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:07:29.344 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.343+0000 7fb900bbe700 10 monclient(hunting): get_auth_request con 0x7fb8fc142290 auth_method 0 2023-09-30T02:07:29.344 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.343+0000 7fb900bbe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:07:29.344 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.343+0000 7fb900bbe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:07:29.344 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.343+0000 7fb900bbe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:07:29.345 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.343+0000 7fb901bc0700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:07:29.345 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.343+0000 7fb901bc0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:07:29.346 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.343+0000 7fb901bc0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:07:29.346 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.343+0000 7fb900bbe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:07:29.346 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.343+0000 7fb900bbe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:07:29.346 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.344+0000 7fb900bbe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:07:29.346 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.346+0000 7fb901bc0700 10 monclient(hunting): handle_auth_done global_id 4427 payload 274 2023-09-30T02:07:29.346 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.346+0000 7fb901bc0700 1 -- 172.21.15.59:0/2674535298 >> v2:172.21.15.59:3300/0 conn(0x7fb8fc142290 msgr2=0x7fb8fc0b1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:07:29.346 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.346+0000 7fb901bc0700 1 --2- 172.21.15.59:0/2674535298 >> v2:172.21.15.59:3300/0 conn(0x7fb8fc142290 0x7fb8fc0b1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:29.346 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.346+0000 7fb901bc0700 1 -- 172.21.15.59:0/2674535298 >> v1:172.21.15.59:6789/0 conn(0x7fb8fc0a5b70 legacy=0x7fb8fc141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:29.346 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.346+0000 7fb901bc0700 10 monclient: _finish_hunting 0 2023-09-30T02:07:29.346 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.346+0000 7fb901bc0700 1 monclient: found mon.noname-b 2023-09-30T02:07:29.346 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.346+0000 7fb901bc0700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:07:29.347 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.346+0000 7fb901bc0700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.59:3301/0 2023-09-30T02:07:29.347 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.346+0000 7fb901bc0700 1 -- 172.21.15.59:0/2674535298 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8fc13b470 con 0x7fb8fc0a5f40 2023-09-30T02:07:29.348 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.348+0000 7fb901bc0700 1 --2- 172.21.15.59:0/2674535298 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb8fc0a5f40 0x7fb8fc13af30 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb8f4009f20 tx=0x7fb8f4009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=238fe9a2ec28583f server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:07:29.353 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.353+0000 7fb8f3fff700 1 -- 172.21.15.59:0/2674535298 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8f400e070 con 0x7fb8fc0a5f40 2023-09-30T02:07:29.353 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.353+0000 7fb8f3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:07:29.353 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.353+0000 7fb8f3fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:07:29.353 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.353+0000 7fb8f3fff700 10 monclient: dump: 2023-09-30T02:07:29.353 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:07:29.353 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:29.353 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:29.353 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:29.353 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:07:29.354 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:29.354 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:07:29.354 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:07:29.354 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:07:29.354 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:29.354 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.353+0000 7fb8f3fff700 10 monclient: _finish_auth 0 2023-09-30T02:07:29.354 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.353+0000 7fb8f3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:07:29.354 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.353+0000 7fb8f3fff700 1 -- 172.21.15.59:0/2674535298 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8f400b9d0 con 0x7fb8fc0a5f40 2023-09-30T02:07:29.354 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.353+0000 7fb903623700 5 monclient: authenticate success, global_id 4427 2023-09-30T02:07:29.354 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.353+0000 7fb903623700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:07:29.354 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.353+0000 7fb8f3fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:07:29.354 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.353+0000 7fb903623700 10 monclient: get_monmap_and_config success 2023-09-30T02:07:29.355 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.353+0000 7fb8f3fff700 1 -- 172.21.15.59:0/2674535298 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8f40092e0 con 0x7fb8fc0a5f40 2023-09-30T02:07:29.355 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.353+0000 7fb903623700 4 set_mon_vals no callback set 2023-09-30T02:07:29.355 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.353+0000 7fb8f3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:07:29.355 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.353+0000 7fb8f3fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:07:29.355 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.353+0000 7fb8f3fff700 10 monclient: dump: 2023-09-30T02:07:29.355 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:07:29.355 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:29.355 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:29.355 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:29.355 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:07:29.355 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:29.355 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:07:29.355 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:07:29.356 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:07:29.356 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:29.359 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.354+0000 7fb903623700 10 monclient: shutdown 2023-09-30T02:07:29.359 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.354+0000 7fb903623700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:07:29.359 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.354+0000 7fb903623700 1 -- 172.21.15.59:0/2674535298 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb8fc0a5f40 msgr2=0x7fb8fc13af30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:29.359 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.354+0000 7fb903623700 1 --2- 172.21.15.59:0/2674535298 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb8fc0a5f40 0x7fb8fc13af30 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb8f4009f20 tx=0x7fb8f4009840 comp rx=0 tx=0).stop 2023-09-30T02:07:29.359 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.354+0000 7fb903623700 1 -- 172.21.15.59:0/2674535298 shutdown_connections 2023-09-30T02:07:29.359 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.354+0000 7fb903623700 1 --2- 172.21.15.59:0/2674535298 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb8fc0a5f40 0x7fb8fc13af30 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:29.359 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.354+0000 7fb903623700 1 --2- 172.21.15.59:0/2674535298 >> v2:172.21.15.59:3300/0 conn(0x7fb8fc142290 0x7fb8fc0b1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:29.359 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.354+0000 7fb903623700 1 -- 172.21.15.59:0/2674535298 >> 172.21.15.59:0/2674535298 conn(0x7fb8fc0631f0 msgr2=0x7fb8fc0635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:07:29.359 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.354+0000 7fb903623700 1 -- 172.21.15.59:0/2674535298 shutdown_connections 2023-09-30T02:07:29.359 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.354+0000 7fb903623700 1 -- 172.21.15.59:0/2674535298 wait complete. 2023-09-30T02:07:29.360 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.355+0000 7fb903623700 10 monclient: build_initial_monmap 2023-09-30T02:07:29.360 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.355+0000 7fb903623700 10 monclient: monmap: 2023-09-30T02:07:29.360 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:07:29.360 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:07:29.360 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:07:29.360 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:07:29.360 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:07:29.360 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:07:29.360 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:07:29.360 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:07:29.360 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:07:29.360 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:29.360 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.355+0000 7fb903623700 1 librados: starting msgr at 2023-09-30T02:07:29.361 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.355+0000 7fb903623700 1 librados: starting objecter 2023-09-30T02:07:29.361 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.355+0000 7fb903623700 1 Processor -- start 2023-09-30T02:07:29.361 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.355+0000 7fb903623700 1 -- start start 2023-09-30T02:07:29.361 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.355+0000 7fb903623700 1 librados: setting wanted keys 2023-09-30T02:07:29.361 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.355+0000 7fb903623700 1 librados: calling monclient init 2023-09-30T02:07:29.361 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.355+0000 7fb903623700 10 monclient: init 2023-09-30T02:07:29.361 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.356+0000 7fb903623700 10 monclient: _reopen_session rank -1 2023-09-30T02:07:29.361 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.356+0000 7fb903623700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T02:07:29.361 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.356+0000 7fb903623700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb8fc0a5b70 0x7fb8fc13ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:29.361 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.356+0000 7fb903623700 10 monclient(hunting): picked mon.noname-a con 0x7fb8fc0a5b70 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:07:29.361 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.356+0000 7fb903623700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb8fc0a5f40 0x7fb8fc140970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:29.361 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.356+0000 7fb903623700 10 monclient(hunting): picked mon.noname-c con 0x7fb8fc0a5f40 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:07:29.362 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.356+0000 7fb903623700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8fc142290 0x7fb8fc14d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:29.362 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.356+0000 7fb903623700 10 monclient(hunting): picked mon.noname-b con 0x7fb8fc142290 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:07:29.362 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.356+0000 7fb903623700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:29.362 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.356+0000 7fb903623700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8fc13e7f0 con 0x7fb8fc0a5b70 2023-09-30T02:07:29.362 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.356+0000 7fb903623700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:29.362 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.356+0000 7fb903623700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8fc141460 con 0x7fb8fc142290 2023-09-30T02:07:29.362 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.356+0000 7fb903623700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:29.362 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.356+0000 7fb903623700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8fc141620 con 0x7fb8fc0a5f40 2023-09-30T02:07:29.362 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.356+0000 7fb903623700 10 monclient(hunting): _renew_subs 2023-09-30T02:07:29.362 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.356+0000 7fb903623700 10 monclient(hunting): authenticate will time out at 1894.658550s 2023-09-30T02:07:29.362 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.357+0000 7fb9013bf700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb8fc0a5b70 0x7fb8fc13ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:29.362 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.357+0000 7fb900bbe700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb8fc0a5f40 0x7fb8fc140970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:29.363 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.357+0000 7fb901bc0700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8fc142290 0x7fb8fc14d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:29.363 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.357+0000 7fb900bbe700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb8fc0a5f40 0x7fb8fc140970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:55276/0 (socket says 172.21.15.59:55276) 2023-09-30T02:07:29.363 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.357+0000 7fb900bbe700 1 -- 172.21.15.59:0/1608723783 learned_addr learned my addr 172.21.15.59:0/1608723783 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:07:29.363 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.357+0000 7fb900bbe700 10 monclient(hunting): get_auth_request con 0x7fb8fc0a5f40 auth_method 0 2023-09-30T02:07:29.363 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.357+0000 7fb900bbe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:07:29.363 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.357+0000 7fb900bbe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:07:29.363 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.357+0000 7fb900bbe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:07:29.363 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.357+0000 7fb9013bf700 10 monclient(hunting): get_auth_request con 0x7fb8fc0a5b70 auth_method 0 2023-09-30T02:07:29.363 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.357+0000 7fb9013bf700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:07:29.363 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.357+0000 7fb9013bf700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:07:29.363 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.357+0000 7fb9013bf700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:07:29.363 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.357+0000 7fb901bc0700 10 monclient(hunting): get_auth_request con 0x7fb8fc142290 auth_method 0 2023-09-30T02:07:29.364 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.357+0000 7fb901bc0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:07:29.364 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.357+0000 7fb901bc0700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:07:29.364 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.357+0000 7fb901bc0700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:07:29.364 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.358+0000 7fb900bbe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:07:29.364 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.358+0000 7fb900bbe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:07:29.364 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.358+0000 7fb900bbe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:07:29.364 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.358+0000 7fb901bc0700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:07:29.364 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.358+0000 7fb901bc0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:07:29.364 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.358+0000 7fb901bc0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:07:29.365 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.360+0000 7fb9013bf700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:07:29.365 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.360+0000 7fb9013bf700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:07:29.365 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.360+0000 7fb9013bf700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:07:29.366 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.362+0000 7fb900bbe700 10 monclient(hunting): handle_auth_done global_id 4430 payload 824 2023-09-30T02:07:29.366 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.362+0000 7fb900bbe700 1 -- 172.21.15.59:0/1608723783 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8fc142290 msgr2=0x7fb8fc14d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:29.366 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.362+0000 7fb900bbe700 1 --2- 172.21.15.59:0/1608723783 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8fc142290 0x7fb8fc14d6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:29.366 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.362+0000 7fb900bbe700 1 -- 172.21.15.59:0/1608723783 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb8fc0a5b70 msgr2=0x7fb8fc13ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:29.366 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.362+0000 7fb900bbe700 1 --2- 172.21.15.59:0/1608723783 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb8fc0a5b70 0x7fb8fc13ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:29.366 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.362+0000 7fb900bbe700 10 monclient: _finish_hunting 0 2023-09-30T02:07:29.366 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.362+0000 7fb900bbe700 1 monclient: found mon.noname-c 2023-09-30T02:07:29.366 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.362+0000 7fb900bbe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:07:29.366 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.362+0000 7fb900bbe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.59:3301/0 2023-09-30T02:07:29.366 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.362+0000 7fb900bbe700 1 -- 172.21.15.59:0/1608723783 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8f4009ae0 con 0x7fb8fc0a5f40 2023-09-30T02:07:29.366 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.365+0000 7fb900bbe700 1 --2- 172.21.15.59:0/1608723783 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb8fc0a5f40 0x7fb8fc140970 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb8ec00e930 tx=0x7fb8ec0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:07:29.366 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.365+0000 7fb8f17fa700 10 client.?.objecter ms_handle_connect 0x7fb8fc0a5f40 2023-09-30T02:07:29.367 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.365+0000 7fb8f17fa700 10 client.?.objecter resend_mon_ops 2023-09-30T02:07:29.369 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.369+0000 7fb8f17fa700 1 -- 172.21.15.59:0/1608723783 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8ec013680 con 0x7fb8fc0a5f40 2023-09-30T02:07:29.369 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.369+0000 7fb8f17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:07:29.369 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.369+0000 7fb8f17fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:07:29.369 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.369+0000 7fb8f17fa700 10 monclient: dump: 2023-09-30T02:07:29.369 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:07:29.369 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:29.370 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:29.370 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:29.370 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:07:29.370 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:29.370 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:07:29.370 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:07:29.370 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:07:29.370 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:29.370 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.369+0000 7fb8f17fa700 10 monclient: _finish_auth 0 2023-09-30T02:07:29.370 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.369+0000 7fb8f17fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:07:29.370 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.369+0000 7fb8f17fa700 1 -- 172.21.15.59:0/1608723783 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8ec0137e0 con 0x7fb8fc0a5f40 2023-09-30T02:07:29.370 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.369+0000 7fb8f17fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:07:29.371 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.369+0000 7fb8f17fa700 1 -- 172.21.15.59:0/1608723783 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8ec013b80 con 0x7fb8fc0a5f40 2023-09-30T02:07:29.371 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.369+0000 7fb903623700 5 monclient: authenticate success, global_id 4430 2023-09-30T02:07:29.371 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.369+0000 7fb903623700 10 monclient: _renew_subs 2023-09-30T02:07:29.371 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.369+0000 7fb903623700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:07:29.371 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.369+0000 7fb903623700 1 -- 172.21.15.59:0/1608723783 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8fc13d990 con 0x7fb8fc0a5f40 2023-09-30T02:07:29.371 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.369+0000 7fb8f37fe700 4 set_mon_vals no callback set 2023-09-30T02:07:29.371 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.369+0000 7fb8f17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:07:29.371 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.369+0000 7fb8f17fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:07:29.371 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.369+0000 7fb8f17fa700 10 monclient: dump: 2023-09-30T02:07:29.371 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:07:29.371 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:29.371 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:29.371 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:29.372 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:07:29.372 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:29.372 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:07:29.372 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:07:29.372 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:07:29.372 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:29.372 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.369+0000 7fb903623700 10 client.4430.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:07:29.372 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.369+0000 7fb903623700 10 monclient: _renew_subs 2023-09-30T02:07:29.372 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.369+0000 7fb903623700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:07:29.372 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.369+0000 7fb903623700 1 -- 172.21.15.59:0/1608723783 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8fc13df20 con 0x7fb8fc0a5f40 2023-09-30T02:07:29.372 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.369+0000 7fb903623700 1 librados: init done 2023-09-30T02:07:29.373 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.370+0000 7fb903623700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:07:29.373 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.371+0000 7fb903623700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:07:29.373 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.371+0000 7fb903623700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:07:29.373 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.371+0000 7fb903623700 1 -- 172.21.15.59:0/1608723783 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8fc064a00 con 0x7fb8fc0a5f40 2023-09-30T02:07:29.510 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.510+0000 7fb8f17fa700 1 -- 172.21.15.59:0/1608723783 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fb8ec00c3e0 con 0x7fb8fc0a5f40 2023-09-30T02:07:29.511 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.511+0000 7fb8f17fa700 1 --2- 172.21.15.59:0/1608723783 >> v2:172.21.15.110:6800/115870 conn(0x7fb8e00689e0 0x7fb8e006ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:29.511 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.511+0000 7fb8f17fa700 1 -- 172.21.15.59:0/1608723783 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fb8ec084720 con 0x7fb8fc0a5f40 2023-09-30T02:07:29.511 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.511+0000 7fb8f17fa700 10 client.4430.objecter ms_dispatch 0x7fb8fc001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:07:29.511 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.511+0000 7fb8f17fa700 3 client.4430.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:07:29.511 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.511+0000 7fb8f17fa700 3 client.4430.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:07:29.511 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.511+0000 7fb8f17fa700 20 client.4430.objecter dump_active .. 0 homeless 2023-09-30T02:07:29.511 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.511+0000 7fb9013bf700 1 --2- 172.21.15.59:0/1608723783 >> v2:172.21.15.110:6800/115870 conn(0x7fb8e00689e0 0x7fb8e006ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:29.511 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.511+0000 7fb9013bf700 10 monclient: get_auth_request con 0x7fb8e00689e0 auth_method 0 2023-09-30T02:07:29.512 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.512+0000 7fb8f17fa700 10 client.4430.objecter ms_handle_connect 0x7fb8e00689e0 2023-09-30T02:07:29.512 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.512+0000 7fb9013bf700 1 --2- 172.21.15.59:0/1608723783 >> v2:172.21.15.110:6800/115870 conn(0x7fb8e00689e0 0x7fb8e006ae90 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fb8f8003a80 tx=0x7fb8f8007ec0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:07:29.669 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.669+0000 7fb8f17fa700 1 -- 172.21.15.59:0/1608723783 <== mon.2 v2:172.21.15.59:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb8ec052040 con 0x7fb8fc0a5f40 2023-09-30T02:07:29.669 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.669+0000 7fb8f17fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:07:29.669 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.669+0000 7fb8f17fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:07:29.919 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.920+0000 7fb903623700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:07:29.919 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.920+0000 7fb903623700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:07:29.919 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.920+0000 7fb903623700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:07:29.919 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.920+0000 7fb903623700 1 -- 172.21.15.59:0/1608723783 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fb8fc06a720 con 0x7fb8fc0a5f40 2023-09-30T02:07:29.924 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.925+0000 7fb8f17fa700 1 -- 172.21.15.59:0/1608723783 <== mon.2 v2:172.21.15.59:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fb8ec055660 con 0x7fb8fc0a5f40 2023-09-30T02:07:29.924 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.925+0000 7fb8f17fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:07:29.924 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.925+0000 7fb8f17fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:07:29.925 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:07:29.925 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:07:29.927 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.927+0000 7fb903623700 10 librados: watch_flush enter 2023-09-30T02:07:29.927 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.927+0000 7fb903623700 10 librados: watch_flush exit 2023-09-30T02:07:29.927 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.927+0000 7fb903623700 20 client.4430.objecter shutdown clearing up homeless session... 2023-09-30T02:07:29.928 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.927+0000 7fb903623700 10 client.4430.objecter successfully canceled tick 2023-09-30T02:07:29.928 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.927+0000 7fb903623700 1 -- 172.21.15.59:0/1608723783 >> v2:172.21.15.110:6800/115870 conn(0x7fb8e00689e0 msgr2=0x7fb8e006ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:29.928 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.927+0000 7fb903623700 1 --2- 172.21.15.59:0/1608723783 >> v2:172.21.15.110:6800/115870 conn(0x7fb8e00689e0 0x7fb8e006ae90 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fb8f8003a80 tx=0x7fb8f8007ec0 comp rx=0 tx=0).stop 2023-09-30T02:07:29.928 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.927+0000 7fb903623700 10 monclient: shutdown 2023-09-30T02:07:29.928 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.927+0000 7fb903623700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:07:29.928 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.927+0000 7fb903623700 1 -- 172.21.15.59:0/1608723783 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb8fc0a5f40 msgr2=0x7fb8fc140970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:29.928 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.927+0000 7fb903623700 1 --2- 172.21.15.59:0/1608723783 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb8fc0a5f40 0x7fb8fc140970 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb8ec00e930 tx=0x7fb8ec0091a0 comp rx=0 tx=0).stop 2023-09-30T02:07:29.928 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.928+0000 7fb903623700 1 -- 172.21.15.59:0/1608723783 shutdown_connections 2023-09-30T02:07:29.928 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.928+0000 7fb903623700 1 --2- 172.21.15.59:0/1608723783 >> v2:172.21.15.110:6800/115870 conn(0x7fb8e00689e0 0x7fb8e006ae90 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:29.928 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.928+0000 7fb903623700 1 --2- 172.21.15.59:0/1608723783 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb8fc142290 0x7fb8fc14d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:29.928 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.928+0000 7fb903623700 1 --2- 172.21.15.59:0/1608723783 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb8fc0a5b70 0x7fb8fc13ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:29.928 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.928+0000 7fb903623700 1 --2- 172.21.15.59:0/1608723783 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb8fc0a5f40 0x7fb8fc140970 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:29.929 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.928+0000 7fb903623700 1 -- 172.21.15.59:0/1608723783 >> 172.21.15.59:0/1608723783 conn(0x7fb8fc0631f0 msgr2=0x7fb8fc0b3360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:07:29.929 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.928+0000 7fb903623700 1 -- 172.21.15.59:0/1608723783 shutdown_connections 2023-09-30T02:07:29.929 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.928+0000 7fb903623700 1 -- 172.21.15.59:0/1608723783 wait complete. 2023-09-30T02:07:29.929 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:29.928+0000 7fb903623700 1 librados: shutdown 2023-09-30T02:07:32.943 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:07:33.085 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.085+0000 7f1376250700 10 monclient: get_monmap_and_config 2023-09-30T02:07:33.085 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.085+0000 7f1376250700 10 monclient: build_initial_monmap 2023-09-30T02:07:33.085 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.085+0000 7f1376250700 10 monclient: monmap: 2023-09-30T02:07:33.085 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:07:33.085 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:33.085 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:07:33.085882+0000 2023-09-30T02:07:33.085 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:07:33.085882+0000 2023-09-30T02:07:33.085 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:07:33.085 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:33.086 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:07:33.086 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:07:33.086 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:07:33.086 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:07:33.086 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:07:33.086 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:33.087 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.086+0000 7f1376250700 1 Processor -- start 2023-09-30T02:07:33.088 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.086+0000 7f1376250700 1 -- start start 2023-09-30T02:07:33.088 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.086+0000 7f1376250700 10 monclient: init 2023-09-30T02:07:33.088 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.087+0000 7f1376250700 10 monclient: _reopen_session rank -1 2023-09-30T02:07:33.088 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.087+0000 7f1376250700 10 monclient: _add_conns ranks=[4,1,2,3,0] 2023-09-30T02:07:33.088 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.087+0000 7f1376250700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f13700a5b70 0x7f13700b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:33.088 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.087+0000 7f1376250700 10 monclient(hunting): picked mon.noname-c con 0x7f13700a5b70 addr v2:172.21.15.110:3300/0 2023-09-30T02:07:33.088 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.087+0000 7f1376250700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f13700a5f40 addr v1:172.21.15.110:6789/0 2023-09-30T02:07:33.088 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.087+0000 7f1376250700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1370142290 0x7f137013af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:33.088 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.087+0000 7f1376250700 10 monclient(hunting): picked mon.noname-b con 0x7f1370142290 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:07:33.088 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.087+0000 7f1376250700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13700a75a0 con 0x7f13700a5f40 2023-09-30T02:07:33.088 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.087+0000 7f1376250700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:33.089 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.087+0000 7f1376250700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13701426d0 con 0x7f13700a5b70 2023-09-30T02:07:33.089 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.087+0000 7f1376250700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:33.089 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.087+0000 7f1376250700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13700a63e0 con 0x7f1370142290 2023-09-30T02:07:33.089 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.087+0000 7f1376250700 10 monclient(hunting): _renew_subs 2023-09-30T02:07:33.089 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.087+0000 7f1376250700 10 monclient(hunting): authenticate will time out at 1898.389182s 2023-09-30T02:07:33.089 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.087+0000 7f136ffff700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1370142290 0x7f137013af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:33.089 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.088+0000 7f136f7fe700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f13700a5b70 0x7f13700b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:33.089 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.088+0000 7f136ffff700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1370142290 0x7f137013af30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:55292/0 (socket says 172.21.15.59:55292) 2023-09-30T02:07:33.089 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.088+0000 7f136ffff700 1 -- 172.21.15.59:0/3268588111 learned_addr learned my addr 172.21.15.59:0/3268588111 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:07:33.089 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.088+0000 7f136ffff700 10 monclient(hunting): get_auth_request con 0x7f1370142290 auth_method 0 2023-09-30T02:07:33.089 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.088+0000 7f136ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:07:33.089 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.088+0000 7f136ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:07:33.090 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.088+0000 7f136ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:07:33.090 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.088+0000 7f136f7fe700 10 monclient(hunting): get_auth_request con 0x7f13700a5b70 auth_method 0 2023-09-30T02:07:33.090 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.088+0000 7f136f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:07:33.090 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.088+0000 7f136f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:07:33.090 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.088+0000 7f136f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:07:33.091 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.089+0000 7f136ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:07:33.091 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.089+0000 7f136ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:07:33.091 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.089+0000 7f136ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:07:33.091 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.089+0000 7f136f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:07:33.091 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.089+0000 7f136f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:07:33.091 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.089+0000 7f136f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:07:33.091 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.091+0000 7f136ffff700 10 monclient(hunting): handle_auth_done global_id 4433 payload 274 2023-09-30T02:07:33.091 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.091+0000 7f136ffff700 1 -- 172.21.15.59:0/3268588111 >> v2:172.21.15.110:3300/0 conn(0x7f13700a5b70 msgr2=0x7f13700b1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:07:33.091 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.091+0000 7f136ffff700 1 --2- 172.21.15.59:0/3268588111 >> v2:172.21.15.110:3300/0 conn(0x7f13700a5b70 0x7f13700b1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:33.091 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.091+0000 7f136ffff700 1 -- 172.21.15.59:0/3268588111 >> v1:172.21.15.110:6789/0 conn(0x7f13700a5f40 legacy=0x7f1370141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:33.092 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.091+0000 7f136ffff700 10 monclient: _finish_hunting 0 2023-09-30T02:07:33.092 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.091+0000 7f136ffff700 1 monclient: found mon.noname-b 2023-09-30T02:07:33.092 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.091+0000 7f136ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:07:33.092 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.091+0000 7f136ffff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.59:3301/0 2023-09-30T02:07:33.092 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.091+0000 7f136ffff700 1 -- 172.21.15.59:0/3268588111 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13700af3f0 con 0x7f1370142290 2023-09-30T02:07:33.092 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.093+0000 7f136ffff700 1 --2- 172.21.15.59:0/3268588111 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1370142290 0x7f137013af30 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f1368009f20 tx=0x7f1368009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b187dce1575000fa server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:07:33.097 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.098+0000 7f136e7fc700 1 -- 172.21.15.59:0/3268588111 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f136800e070 con 0x7f1370142290 2023-09-30T02:07:33.097 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.098+0000 7f136e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:07:33.097 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.098+0000 7f136e7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:07:33.098 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.098+0000 7f136e7fc700 10 monclient: dump: 2023-09-30T02:07:33.098 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:07:33.098 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:33.098 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:33.098 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:33.098 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:07:33.098 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:33.098 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:07:33.098 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:07:33.098 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:07:33.098 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:33.099 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.098+0000 7f136e7fc700 10 monclient: _finish_auth 0 2023-09-30T02:07:33.099 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.098+0000 7f136e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:07:33.099 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.098+0000 7f136e7fc700 1 -- 172.21.15.59:0/3268588111 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f136800b9d0 con 0x7f1370142290 2023-09-30T02:07:33.099 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.098+0000 7f136e7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:07:33.099 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.098+0000 7f136e7fc700 1 -- 172.21.15.59:0/3268588111 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13680092e0 con 0x7f1370142290 2023-09-30T02:07:33.099 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.098+0000 7f1376250700 5 monclient: authenticate success, global_id 4433 2023-09-30T02:07:33.099 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.098+0000 7f1376250700 10 monclient: get_monmap_and_config success 2023-09-30T02:07:33.099 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.098+0000 7f1376250700 4 set_mon_vals no callback set 2023-09-30T02:07:33.099 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.098+0000 7f136e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:07:33.099 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.098+0000 7f136e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:07:33.099 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.098+0000 7f136e7fc700 10 monclient: dump: 2023-09-30T02:07:33.099 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:07:33.099 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:33.100 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:33.100 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:33.100 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:07:33.100 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:33.100 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:07:33.100 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:07:33.100 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:07:33.100 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:33.103 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.098+0000 7f1376250700 10 monclient: shutdown 2023-09-30T02:07:33.103 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.098+0000 7f1376250700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:07:33.103 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.098+0000 7f1376250700 1 -- 172.21.15.59:0/3268588111 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1370142290 msgr2=0x7f137013af30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:33.104 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.098+0000 7f1376250700 1 --2- 172.21.15.59:0/3268588111 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1370142290 0x7f137013af30 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f1368009f20 tx=0x7f1368009840 comp rx=0 tx=0).stop 2023-09-30T02:07:33.104 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.099+0000 7f1376250700 1 -- 172.21.15.59:0/3268588111 shutdown_connections 2023-09-30T02:07:33.104 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.099+0000 7f1376250700 1 --2- 172.21.15.59:0/3268588111 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1370142290 0x7f137013af30 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:33.104 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.099+0000 7f1376250700 1 --2- 172.21.15.59:0/3268588111 >> v2:172.21.15.110:3300/0 conn(0x7f13700a5b70 0x7f13700b1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:33.104 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.099+0000 7f1376250700 1 -- 172.21.15.59:0/3268588111 >> 172.21.15.59:0/3268588111 conn(0x7f13700631f0 msgr2=0x7f13700635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:07:33.104 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.099+0000 7f1376250700 1 -- 172.21.15.59:0/3268588111 shutdown_connections 2023-09-30T02:07:33.104 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.099+0000 7f1376250700 1 -- 172.21.15.59:0/3268588111 wait complete. 2023-09-30T02:07:33.104 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.100+0000 7f1376250700 10 monclient: build_initial_monmap 2023-09-30T02:07:33.104 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.100+0000 7f1376250700 10 monclient: monmap: 2023-09-30T02:07:33.104 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:07:33.104 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:07:33.104 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:07:33.105 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:07:33.105 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:07:33.105 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:07:33.105 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:07:33.105 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:07:33.105 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:07:33.105 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:33.105 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.100+0000 7f1376250700 1 librados: starting msgr at 2023-09-30T02:07:33.105 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.100+0000 7f1376250700 1 librados: starting objecter 2023-09-30T02:07:33.105 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.100+0000 7f1376250700 1 Processor -- start 2023-09-30T02:07:33.105 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.100+0000 7f1376250700 1 -- start start 2023-09-30T02:07:33.106 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.100+0000 7f1376250700 1 librados: setting wanted keys 2023-09-30T02:07:33.106 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.100+0000 7f1376250700 1 librados: calling monclient init 2023-09-30T02:07:33.106 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.100+0000 7f1376250700 10 monclient: init 2023-09-30T02:07:33.106 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.101+0000 7f1376250700 10 monclient: _reopen_session rank -1 2023-09-30T02:07:33.106 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.101+0000 7f1376250700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T02:07:33.106 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.101+0000 7f1376250700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f13700a5b70 0x7f1370144e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:33.106 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.101+0000 7f1376250700 10 monclient(hunting): picked mon.noname-c con 0x7f13700a5b70 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:07:33.106 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.101+0000 7f1376250700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f13700a5f40 0x7f137013c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:33.106 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.101+0000 7f1376250700 10 monclient(hunting): picked mon.noname-b con 0x7f13700a5f40 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:07:33.106 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.101+0000 7f1376250700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1370142290 0x7f137013c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:33.106 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.101+0000 7f1376250700 10 monclient(hunting): picked mon.noname-a con 0x7f1370142290 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:07:33.106 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.101+0000 7f1376250700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:33.107 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.101+0000 7f1376250700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f137013ea50 con 0x7f1370142290 2023-09-30T02:07:33.107 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.101+0000 7f1376250700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:33.107 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.101+0000 7f1376250700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13700a8490 con 0x7f13700a5f40 2023-09-30T02:07:33.107 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.101+0000 7f1376250700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:33.107 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.101+0000 7f1376250700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f137013e700 con 0x7f13700a5b70 2023-09-30T02:07:33.107 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.101+0000 7f1376250700 10 monclient(hunting): _renew_subs 2023-09-30T02:07:33.107 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.101+0000 7f1376250700 10 monclient(hunting): authenticate will time out at 1898.403715s 2023-09-30T02:07:33.107 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.102+0000 7f136f7fe700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f13700a5b70 0x7f1370144e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:33.107 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.102+0000 7f136effd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f13700a5f40 0x7f137013c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:33.107 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.102+0000 7f136ffff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1370142290 0x7f137013c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:33.107 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.102+0000 7f136effd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f13700a5f40 0x7f137013c030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:46938/0 (socket says 172.21.15.59:46938) 2023-09-30T02:07:33.107 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.102+0000 7f136effd700 1 -- 172.21.15.59:0/1530272669 learned_addr learned my addr 172.21.15.59:0/1530272669 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:07:33.108 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.102+0000 7f136effd700 10 monclient(hunting): get_auth_request con 0x7f13700a5f40 auth_method 0 2023-09-30T02:07:33.108 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.102+0000 7f136effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:07:33.108 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.102+0000 7f136effd700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:07:33.108 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.102+0000 7f136effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:07:33.108 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.102+0000 7f136f7fe700 10 monclient(hunting): get_auth_request con 0x7f13700a5b70 auth_method 0 2023-09-30T02:07:33.108 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.102+0000 7f136f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:07:33.108 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.102+0000 7f136f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:07:33.108 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.102+0000 7f136f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:07:33.108 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.103+0000 7f136ffff700 10 monclient(hunting): get_auth_request con 0x7f1370142290 auth_method 0 2023-09-30T02:07:33.108 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.103+0000 7f136ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:07:33.108 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.103+0000 7f136ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:07:33.108 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.103+0000 7f136ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:07:33.108 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.103+0000 7f136effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:07:33.109 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.103+0000 7f136effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:07:33.109 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.103+0000 7f136effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:07:33.109 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.103+0000 7f136f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:07:33.109 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.103+0000 7f136f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:07:33.109 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.103+0000 7f136f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:07:33.109 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.103+0000 7f136ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:07:33.109 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.103+0000 7f136ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:07:33.109 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.103+0000 7f136ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:07:33.110 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.106+0000 7f136effd700 10 monclient(hunting): handle_auth_done global_id 4495 payload 824 2023-09-30T02:07:33.110 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.106+0000 7f136effd700 1 -- 172.21.15.59:0/1530272669 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f13700a5b70 msgr2=0x7f1370144e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:33.110 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.106+0000 7f136effd700 1 --2- 172.21.15.59:0/1530272669 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f13700a5b70 0x7f1370144e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:33.110 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.106+0000 7f136effd700 1 -- 172.21.15.59:0/1530272669 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1370142290 msgr2=0x7f137013c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:33.110 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.106+0000 7f136effd700 1 --2- 172.21.15.59:0/1530272669 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1370142290 0x7f137013c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:33.111 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.106+0000 7f136effd700 10 monclient: _finish_hunting 0 2023-09-30T02:07:33.111 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.106+0000 7f136effd700 1 monclient: found mon.noname-b 2023-09-30T02:07:33.111 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.106+0000 7f136effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:07:33.111 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.106+0000 7f136effd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:07:33.111 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.106+0000 7f136effd700 1 -- 172.21.15.59:0/1530272669 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1368009ae0 con 0x7f13700a5f40 2023-09-30T02:07:33.111 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.108+0000 7f136effd700 1 --2- 172.21.15.59:0/1530272669 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f13700a5f40 0x7f137013c030 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f136400c590 tx=0x7f136400c8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:07:33.111 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.108+0000 7f13577fe700 10 client.?.objecter ms_handle_connect 0x7f13700a5f40 2023-09-30T02:07:33.111 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.108+0000 7f13577fe700 10 client.?.objecter resend_mon_ops 2023-09-30T02:07:33.112 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.112+0000 7f13577fe700 1 -- 172.21.15.59:0/1530272669 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1364011590 con 0x7f13700a5f40 2023-09-30T02:07:33.113 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.112+0000 7f13577fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:07:33.113 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.113+0000 7f13577fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:07:33.113 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.113+0000 7f13577fe700 10 monclient: dump: 2023-09-30T02:07:33.113 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:07:33.113 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:33.113 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:33.113 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:33.113 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:07:33.114 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:33.114 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:07:33.114 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:07:33.114 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:07:33.114 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:33.114 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.113+0000 7f13577fe700 10 monclient: _finish_auth 0 2023-09-30T02:07:33.114 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.113+0000 7f13577fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:07:33.115 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.113+0000 7f13577fe700 1 -- 172.21.15.59:0/1530272669 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1364011710 con 0x7f13700a5f40 2023-09-30T02:07:33.115 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.113+0000 7f1376250700 5 monclient: authenticate success, global_id 4495 2023-09-30T02:07:33.115 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.113+0000 7f1376250700 10 monclient: _renew_subs 2023-09-30T02:07:33.115 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.113+0000 7f1376250700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:07:33.115 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.113+0000 7f1376250700 1 -- 172.21.15.59:0/1530272669 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13700a8760 con 0x7f13700a5f40 2023-09-30T02:07:33.115 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.113+0000 7f13577fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:07:33.115 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.113+0000 7f13577fe700 1 -- 172.21.15.59:0/1530272669 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1364011ad0 con 0x7f13700a5f40 2023-09-30T02:07:33.115 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.113+0000 7f13577fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:07:33.116 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.113+0000 7f13577fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:07:33.116 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.113+0000 7f13577fe700 10 monclient: dump: 2023-09-30T02:07:33.116 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:07:33.116 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:33.116 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:33.116 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:33.116 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:07:33.117 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:33.117 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:07:33.117 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:07:33.117 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:07:33.117 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:33.117 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.113+0000 7f136dffb700 4 set_mon_vals no callback set 2023-09-30T02:07:33.117 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.113+0000 7f1376250700 10 client.4495.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:07:33.117 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.113+0000 7f1376250700 10 monclient: _renew_subs 2023-09-30T02:07:33.117 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.113+0000 7f1376250700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:07:33.117 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.113+0000 7f1376250700 1 -- 172.21.15.59:0/1530272669 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f137013e340 con 0x7f13700a5f40 2023-09-30T02:07:33.118 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.113+0000 7f1376250700 1 librados: init done 2023-09-30T02:07:33.118 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.114+0000 7f1376250700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:07:33.118 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.114+0000 7f1376250700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:07:33.118 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.114+0000 7f1376250700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:07:33.119 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.114+0000 7f1376250700 1 -- 172.21.15.59:0/1530272669 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f137013b200 con 0x7f13700a5f40 2023-09-30T02:07:33.365 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.364+0000 7f13577fe700 1 -- 172.21.15.59:0/1530272669 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f136401c440 con 0x7f13700a5f40 2023-09-30T02:07:33.365 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.364+0000 7f13577fe700 1 --2- 172.21.15.59:0/1530272669 >> v2:172.21.15.110:6800/115870 conn(0x7f135c068910 0x7f135c06adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:33.365 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.365+0000 7f13577fe700 1 -- 172.21.15.59:0/1530272669 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f1364083aa0 con 0x7f13700a5f40 2023-09-30T02:07:33.365 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.365+0000 7f13577fe700 10 client.4495.objecter ms_dispatch 0x7f1370001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:07:33.365 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.365+0000 7f13577fe700 3 client.4495.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:07:33.366 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.365+0000 7f13577fe700 3 client.4495.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:07:33.366 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.365+0000 7f13577fe700 20 client.4495.objecter dump_active .. 0 homeless 2023-09-30T02:07:33.366 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.365+0000 7f136f7fe700 1 --2- 172.21.15.59:0/1530272669 >> v2:172.21.15.110:6800/115870 conn(0x7f135c068910 0x7f135c06adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:33.366 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.365+0000 7f136f7fe700 10 monclient: get_auth_request con 0x7f135c068910 auth_method 0 2023-09-30T02:07:33.366 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.365+0000 7f13577fe700 10 client.4495.objecter ms_handle_connect 0x7f135c068910 2023-09-30T02:07:33.366 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.365+0000 7f136f7fe700 1 --2- 172.21.15.59:0/1530272669 >> v2:172.21.15.110:6800/115870 conn(0x7f135c068910 0x7f135c06adc0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f1360009db0 tx=0x7f13600091a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:07:33.372 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.373+0000 7f13577fe700 1 -- 172.21.15.59:0/1530272669 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1364051340 con 0x7f13700a5f40 2023-09-30T02:07:33.373 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.373+0000 7f13577fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:07:33.373 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.373+0000 7f13577fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:07:33.595 INFO:tasks.ceph.mon.a.smithi059.stderr:==00:00:02:43.460 116256== Warning: unimplemented fcntl command: 1036 2023-09-30T02:07:33.601 INFO:tasks.ceph.mon.a.smithi059.stderr:==00:00:02:43.467 116256== Warning: unimplemented fcntl command: 1036 2023-09-30T02:07:33.634 INFO:tasks.ceph.mon.a.smithi059.stderr:==00:00:02:43.499 116256== Warning: unimplemented fcntl command: 1036 2023-09-30T02:07:33.652 INFO:tasks.ceph.mon.b.smithi110.stderr:==00:00:02:43.499 115869== Warning: unimplemented fcntl command: 1036 2023-09-30T02:07:33.656 INFO:tasks.ceph.mon.b.smithi110.stderr:==00:00:02:43.503 115869== Warning: unimplemented fcntl command: 1036 2023-09-30T02:07:33.660 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:02:43.526 116257== Warning: unimplemented fcntl command: 1036 2023-09-30T02:07:33.663 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.664+0000 7f1376250700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:07:33.664 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.664+0000 7f1376250700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:07:33.664 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.664+0000 7f1376250700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:07:33.664 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.664+0000 7f1376250700 1 -- 172.21.15.59:0/1530272669 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f1370064a00 con 0x7f13700a5f40 2023-09-30T02:07:33.665 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:02:43.531 116257== Warning: unimplemented fcntl command: 1036 2023-09-30T02:07:33.681 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.682+0000 7f13577fe700 1 -- 172.21.15.59:0/1530272669 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f1364017080 con 0x7f13700a5f40 2023-09-30T02:07:33.681 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.682+0000 7f13577fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:07:33.681 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.682+0000 7f13577fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:07:33.681 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:07:33.681 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:07:33.685 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.685+0000 7f1376250700 10 librados: watch_flush enter 2023-09-30T02:07:33.685 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.685+0000 7f1376250700 10 librados: watch_flush exit 2023-09-30T02:07:33.685 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.685+0000 7f1376250700 20 client.4495.objecter shutdown clearing up homeless session... 2023-09-30T02:07:33.685 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.685+0000 7f1376250700 10 client.4495.objecter successfully canceled tick 2023-09-30T02:07:33.685 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.685+0000 7f1376250700 1 -- 172.21.15.59:0/1530272669 >> v2:172.21.15.110:6800/115870 conn(0x7f135c068910 msgr2=0x7f135c06adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:33.685 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.685+0000 7f1376250700 1 --2- 172.21.15.59:0/1530272669 >> v2:172.21.15.110:6800/115870 conn(0x7f135c068910 0x7f135c06adc0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f1360009db0 tx=0x7f13600091a0 comp rx=0 tx=0).stop 2023-09-30T02:07:33.685 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.685+0000 7f1376250700 10 monclient: shutdown 2023-09-30T02:07:33.685 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.685+0000 7f1376250700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:07:33.686 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.685+0000 7f1376250700 1 -- 172.21.15.59:0/1530272669 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f13700a5f40 msgr2=0x7f137013c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:33.686 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.685+0000 7f1376250700 1 --2- 172.21.15.59:0/1530272669 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f13700a5f40 0x7f137013c030 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f136400c590 tx=0x7f136400c8a0 comp rx=0 tx=0).stop 2023-09-30T02:07:33.686 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.685+0000 7f1376250700 1 -- 172.21.15.59:0/1530272669 shutdown_connections 2023-09-30T02:07:33.686 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.685+0000 7f1376250700 1 --2- 172.21.15.59:0/1530272669 >> v2:172.21.15.110:6800/115870 conn(0x7f135c068910 0x7f135c06adc0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:33.686 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.685+0000 7f1376250700 1 --2- 172.21.15.59:0/1530272669 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1370142290 0x7f137013c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:33.686 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.685+0000 7f1376250700 1 --2- 172.21.15.59:0/1530272669 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f13700a5b70 0x7f1370144e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:33.686 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.685+0000 7f1376250700 1 --2- 172.21.15.59:0/1530272669 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f13700a5f40 0x7f137013c030 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:33.686 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.685+0000 7f1376250700 1 -- 172.21.15.59:0/1530272669 >> 172.21.15.59:0/1530272669 conn(0x7f13700631f0 msgr2=0x7f1370137ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:07:33.686 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.685+0000 7f1376250700 1 -- 172.21.15.59:0/1530272669 shutdown_connections 2023-09-30T02:07:33.686 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.685+0000 7f1376250700 1 -- 172.21.15.59:0/1530272669 wait complete. 2023-09-30T02:07:33.686 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:33.685+0000 7f1376250700 1 librados: shutdown 2023-09-30T02:07:33.690 INFO:tasks.ceph.mon.b.smithi110.stderr:==00:00:02:43.537 115869== Warning: unimplemented fcntl command: 1036 2023-09-30T02:07:33.700 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:02:43.566 116257== Warning: unimplemented fcntl command: 1036 2023-09-30T02:07:36.704 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:07:36.845 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.845+0000 7f1d3d8b6700 10 monclient: get_monmap_and_config 2023-09-30T02:07:36.845 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.845+0000 7f1d3d8b6700 10 monclient: build_initial_monmap 2023-09-30T02:07:36.845 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.845+0000 7f1d3d8b6700 10 monclient: monmap: 2023-09-30T02:07:36.845 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:07:36.845 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:36.845 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:07:36.846177+0000 2023-09-30T02:07:36.845 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:07:36.846177+0000 2023-09-30T02:07:36.846 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:07:36.846 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:36.846 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:07:36.846 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:07:36.846 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:07:36.846 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:07:36.846 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:07:36.846 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:36.847 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.846+0000 7f1d3d8b6700 1 Processor -- start 2023-09-30T02:07:36.848 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.846+0000 7f1d3d8b6700 1 -- start start 2023-09-30T02:07:36.848 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.846+0000 7f1d3d8b6700 10 monclient: init 2023-09-30T02:07:36.848 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.847+0000 7f1d3d8b6700 10 monclient: _reopen_session rank -1 2023-09-30T02:07:36.848 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.847+0000 7f1d3d8b6700 10 monclient: _add_conns ranks=[0,4,1,2,3] 2023-09-30T02:07:36.848 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.847+0000 7f1d3d8b6700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1d380ac9b0 addr v1:172.21.15.59:6789/0 2023-09-30T02:07:36.848 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.847+0000 7f1d3d8b6700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f1d380acd80 0x7f1d380ad9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:36.848 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.847+0000 7f1d3d8b6700 10 monclient(hunting): picked mon.noname-c con 0x7f1d380acd80 addr v2:172.21.15.110:3300/0 2023-09-30T02:07:36.848 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.847+0000 7f1d3d8b6700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1d380ae430 addr v1:172.21.15.110:6789/0 2023-09-30T02:07:36.848 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.847+0000 7f1d3d8b6700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d380ad560 con 0x7f1d380ac9b0 2023-09-30T02:07:36.848 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.847+0000 7f1d3d8b6700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d380aa260 con 0x7f1d380ae430 2023-09-30T02:07:36.848 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.847+0000 7f1d3d8b6700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:36.849 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.847+0000 7f1d3d8b6700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d380ad2c0 con 0x7f1d380acd80 2023-09-30T02:07:36.849 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.847+0000 7f1d3d8b6700 10 monclient(hunting): _renew_subs 2023-09-30T02:07:36.849 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.847+0000 7f1d3d8b6700 10 monclient(hunting): authenticate will time out at 1902.149616s 2023-09-30T02:07:36.849 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.848+0000 7f1d36ffd700 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f1d380ac9b0 0x7f1d380aa960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:47580/0 (socket says 172.21.15.59:47580) 2023-09-30T02:07:36.849 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.848+0000 7f1d36ffd700 1 -- 172.21.15.59:0/236705422 learned_addr learned my addr 172.21.15.59:0/236705422 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:07:36.849 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.848+0000 7f1d367fc700 1 --2- 172.21.15.59:0/236705422 >> v2:172.21.15.110:3300/0 conn(0x7f1d380acd80 0x7f1d380ad9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:36.849 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.849+0000 7f1d367fc700 10 monclient(hunting): get_auth_request con 0x7f1d380acd80 auth_method 0 2023-09-30T02:07:36.849 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.849+0000 7f1d367fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:07:36.850 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.849+0000 7f1d367fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:07:36.850 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.849+0000 7f1d367fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:07:36.850 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.850+0000 7f1d367fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:07:36.850 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.850+0000 7f1d367fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:07:36.850 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.850+0000 7f1d367fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:07:36.851 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.851+0000 7f1d35ffb700 1 -- 172.21.15.59:0/236705422 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2036613584 0 0) 0x7f1d380ad560 con 0x7f1d380ac9b0 2023-09-30T02:07:36.851 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.852+0000 7f1d35ffb700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:07:36.851 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.852+0000 7f1d35ffb700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:07:36.851 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.852+0000 7f1d35ffb700 10 monclient(hunting): my global_id is 4485 2023-09-30T02:07:36.851 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.852+0000 7f1d35ffb700 1 -- 172.21.15.59:0/236705422 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d20003570 con 0x7f1d380ac9b0 2023-09-30T02:07:36.852 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.852+0000 7f1d367fc700 10 monclient(hunting): handle_auth_done global_id 4498 payload 274 2023-09-30T02:07:36.852 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.852+0000 7f1d367fc700 1 -- 172.21.15.59:0/236705422 >> v1:172.21.15.110:6789/0 conn(0x7f1d380ae430 legacy=0x7f1d3813e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:36.852 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.852+0000 7f1d367fc700 1 -- 172.21.15.59:0/236705422 >> v1:172.21.15.59:6789/0 conn(0x7f1d380ac9b0 legacy=0x7f1d380aa960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:36.852 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.853+0000 7f1d367fc700 10 monclient: _finish_hunting 0 2023-09-30T02:07:36.852 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.853+0000 7f1d367fc700 1 monclient: found mon.noname-c 2023-09-30T02:07:36.852 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.853+0000 7f1d367fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:07:36.852 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.853+0000 7f1d367fc700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3300/0 2023-09-30T02:07:36.852 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.853+0000 7f1d367fc700 1 -- 172.21.15.59:0/236705422 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d380ab070 con 0x7f1d380acd80 2023-09-30T02:07:36.855 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.855+0000 7f1d367fc700 1 --2- 172.21.15.59:0/236705422 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1d380acd80 0x7f1d380ad9f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c0098e0 tx=0x7f1d2c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=190726c664e92434 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:07:36.863 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.864+0000 7f1d35ffb700 1 -- 172.21.15.59:0/236705422 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d2c00b640 con 0x7f1d380acd80 2023-09-30T02:07:36.863 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.864+0000 7f1d35ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:07:36.864 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.864+0000 7f1d35ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:07:36.864 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.864+0000 7f1d35ffb700 10 monclient: dump: 2023-09-30T02:07:36.864 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:07:36.864 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:36.864 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:36.864 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:36.864 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:07:36.864 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:36.865 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:07:36.865 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:07:36.865 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:07:36.865 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:36.865 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.864+0000 7f1d35ffb700 10 monclient: _finish_auth 0 2023-09-30T02:07:36.865 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.864+0000 7f1d35ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:07:36.865 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.864+0000 7f1d35ffb700 1 -- 172.21.15.59:0/236705422 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d2c00b7a0 con 0x7f1d380acd80 2023-09-30T02:07:36.865 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.864+0000 7f1d35ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:07:36.865 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.864+0000 7f1d35ffb700 1 -- 172.21.15.59:0/236705422 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d2c00bb40 con 0x7f1d380acd80 2023-09-30T02:07:36.866 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.864+0000 7f1d3d8b6700 5 monclient: authenticate success, global_id 4498 2023-09-30T02:07:36.866 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.864+0000 7f1d3d8b6700 10 monclient: get_monmap_and_config success 2023-09-30T02:07:36.866 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.864+0000 7f1d3d8b6700 4 set_mon_vals no callback set 2023-09-30T02:07:36.866 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.864+0000 7f1d35ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:07:36.866 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.864+0000 7f1d35ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:07:36.866 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.864+0000 7f1d35ffb700 10 monclient: dump: 2023-09-30T02:07:36.866 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:07:36.866 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:36.866 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:36.867 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:36.867 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:07:36.867 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:36.867 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:07:36.867 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:07:36.867 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:07:36.867 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:36.871 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.864+0000 7f1d3d8b6700 10 monclient: shutdown 2023-09-30T02:07:36.871 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.864+0000 7f1d3d8b6700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:07:36.871 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.864+0000 7f1d3d8b6700 1 -- 172.21.15.59:0/236705422 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1d380acd80 msgr2=0x7f1d380ad9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:36.871 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.864+0000 7f1d3d8b6700 1 --2- 172.21.15.59:0/236705422 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1d380acd80 0x7f1d380ad9f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c0098e0 tx=0x7f1d2c0091a0 comp rx=0 tx=0).stop 2023-09-30T02:07:36.871 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.865+0000 7f1d3d8b6700 1 -- 172.21.15.59:0/236705422 shutdown_connections 2023-09-30T02:07:36.872 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.865+0000 7f1d3d8b6700 1 --2- 172.21.15.59:0/236705422 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1d380acd80 0x7f1d380ad9f0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:36.872 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.865+0000 7f1d3d8b6700 1 -- 172.21.15.59:0/236705422 >> 172.21.15.59:0/236705422 conn(0x7f1d38069f90 msgr2=0x7f1d38068730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:07:36.872 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.865+0000 7f1d3d8b6700 1 -- 172.21.15.59:0/236705422 shutdown_connections 2023-09-30T02:07:36.872 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.865+0000 7f1d3d8b6700 1 -- 172.21.15.59:0/236705422 wait complete. 2023-09-30T02:07:36.872 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.866+0000 7f1d3d8b6700 10 monclient: build_initial_monmap 2023-09-30T02:07:36.872 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.866+0000 7f1d3d8b6700 10 monclient: monmap: 2023-09-30T02:07:36.872 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:07:36.872 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:07:36.873 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:07:36.873 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:07:36.873 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:07:36.873 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:07:36.873 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:07:36.873 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:07:36.873 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:07:36.873 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:36.873 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.866+0000 7f1d3d8b6700 1 librados: starting msgr at 2023-09-30T02:07:36.874 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.866+0000 7f1d3d8b6700 1 librados: starting objecter 2023-09-30T02:07:36.874 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.866+0000 7f1d3d8b6700 1 Processor -- start 2023-09-30T02:07:36.874 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.866+0000 7f1d3d8b6700 1 -- start start 2023-09-30T02:07:36.874 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.866+0000 7f1d3d8b6700 1 librados: setting wanted keys 2023-09-30T02:07:36.874 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.866+0000 7f1d3d8b6700 1 librados: calling monclient init 2023-09-30T02:07:36.874 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.866+0000 7f1d3d8b6700 10 monclient: init 2023-09-30T02:07:36.874 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.867+0000 7f1d3d8b6700 10 monclient: _reopen_session rank -1 2023-09-30T02:07:36.874 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.867+0000 7f1d3d8b6700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T02:07:36.874 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.867+0000 7f1d3d8b6700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d380ac9b0 0x7f1d3813fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:36.875 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.867+0000 7f1d3d8b6700 10 monclient(hunting): picked mon.noname-a con 0x7f1d380ac9b0 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:07:36.875 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.867+0000 7f1d3d8b6700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1d380acd80 0x7f1d38151250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:36.875 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.867+0000 7f1d3d8b6700 10 monclient(hunting): picked mon.noname-c con 0x7f1d380acd80 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:07:36.875 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.867+0000 7f1d3d8b6700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1d380ae430 0x7f1d38119630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:36.875 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.867+0000 7f1d3d8b6700 10 monclient(hunting): picked mon.noname-b con 0x7f1d380ae430 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:07:36.875 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.867+0000 7f1d3d8b6700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:36.875 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.867+0000 7f1d3d8b6700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d38119e90 con 0x7f1d380ac9b0 2023-09-30T02:07:36.875 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.867+0000 7f1d3d8b6700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:36.875 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.867+0000 7f1d3d8b6700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d38119b70 con 0x7f1d380ae430 2023-09-30T02:07:36.876 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.867+0000 7f1d3d8b6700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:36.876 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.867+0000 7f1d3d8b6700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d38151d80 con 0x7f1d380acd80 2023-09-30T02:07:36.876 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.867+0000 7f1d3d8b6700 10 monclient(hunting): _renew_subs 2023-09-30T02:07:36.876 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.867+0000 7f1d3d8b6700 10 monclient(hunting): authenticate will time out at 1902.169388s 2023-09-30T02:07:36.876 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.867+0000 7f1d36ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d380ac9b0 0x7f1d3813fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:36.876 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.868+0000 7f1d367fc700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1d380acd80 0x7f1d38151250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:36.876 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.868+0000 7f1d36ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d380ac9b0 0x7f1d3813fc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:55598/0 (socket says 172.21.15.59:55598) 2023-09-30T02:07:36.876 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.868+0000 7f1d36ffd700 1 -- 172.21.15.59:0/4218767204 learned_addr learned my addr 172.21.15.59:0/4218767204 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:07:36.876 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.868+0000 7f1d36ffd700 10 monclient(hunting): get_auth_request con 0x7f1d380ac9b0 auth_method 0 2023-09-30T02:07:36.877 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.868+0000 7f1d36ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:07:36.877 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.868+0000 7f1d36ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:07:36.877 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.868+0000 7f1d36ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:07:36.877 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.868+0000 7f1d367fc700 10 monclient(hunting): get_auth_request con 0x7f1d380acd80 auth_method 0 2023-09-30T02:07:36.877 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.868+0000 7f1d367fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:07:36.877 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.868+0000 7f1d367fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:07:36.877 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.868+0000 7f1d367fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:07:36.877 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.868+0000 7f1d377fe700 1 --2- 172.21.15.59:0/4218767204 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1d380ae430 0x7f1d38119630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:36.877 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.869+0000 7f1d377fe700 10 monclient(hunting): get_auth_request con 0x7f1d380ae430 auth_method 0 2023-09-30T02:07:36.877 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.869+0000 7f1d377fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:07:36.878 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.869+0000 7f1d377fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:07:36.878 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.869+0000 7f1d377fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:07:36.878 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.869+0000 7f1d36ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:07:36.878 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.869+0000 7f1d36ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:07:36.878 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.869+0000 7f1d36ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:07:36.878 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.869+0000 7f1d367fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:07:36.878 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.869+0000 7f1d367fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:07:36.878 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.869+0000 7f1d367fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:07:36.878 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.870+0000 7f1d377fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:07:36.879 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.870+0000 7f1d377fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:07:36.879 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.870+0000 7f1d377fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:07:36.881 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.873+0000 7f1d377fe700 10 monclient(hunting): handle_auth_done global_id 4501 payload 824 2023-09-30T02:07:36.881 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.873+0000 7f1d377fe700 1 -- 172.21.15.59:0/4218767204 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1d380acd80 msgr2=0x7f1d38151250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:36.881 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.873+0000 7f1d377fe700 1 --2- 172.21.15.59:0/4218767204 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1d380acd80 0x7f1d38151250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:36.881 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.873+0000 7f1d377fe700 1 -- 172.21.15.59:0/4218767204 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d380ac9b0 msgr2=0x7f1d3813fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:36.881 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.873+0000 7f1d377fe700 1 --2- 172.21.15.59:0/4218767204 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d380ac9b0 0x7f1d3813fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:36.881 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.873+0000 7f1d377fe700 10 monclient: _finish_hunting 0 2023-09-30T02:07:36.881 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.873+0000 7f1d377fe700 1 monclient: found mon.noname-b 2023-09-30T02:07:36.882 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.873+0000 7f1d377fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:07:36.882 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.873+0000 7f1d377fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:07:36.882 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.873+0000 7f1d377fe700 1 -- 172.21.15.59:0/4218767204 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d2c009420 con 0x7f1d380ae430 2023-09-30T02:07:36.882 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.876+0000 7f1d377fe700 1 --2- 172.21.15.59:0/4218767204 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1d380ae430 0x7f1d38119630 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f1d3000c7c0 tx=0x7f1d30001560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:07:36.882 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.876+0000 7f1d1effd700 10 client.?.objecter ms_handle_connect 0x7f1d380ae430 2023-09-30T02:07:36.882 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.876+0000 7f1d1effd700 10 client.?.objecter resend_mon_ops 2023-09-30T02:07:36.882 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.879+0000 7f1d1effd700 1 -- 172.21.15.59:0/4218767204 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d30011590 con 0x7f1d380ae430 2023-09-30T02:07:36.882 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.879+0000 7f1d1effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:07:36.882 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.879+0000 7f1d1effd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:07:36.883 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.879+0000 7f1d1effd700 10 monclient: dump: 2023-09-30T02:07:36.883 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:07:36.883 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:36.883 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:36.883 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:36.883 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:07:36.883 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:36.883 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:07:36.883 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:07:36.883 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:07:36.883 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:36.884 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.879+0000 7f1d1effd700 10 monclient: _finish_auth 0 2023-09-30T02:07:36.884 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.879+0000 7f1d1effd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:07:36.884 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.879+0000 7f1d3d8b6700 5 monclient: authenticate success, global_id 4501 2023-09-30T02:07:36.884 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.879+0000 7f1d3d8b6700 10 monclient: _renew_subs 2023-09-30T02:07:36.884 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.879+0000 7f1d3d8b6700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:07:36.884 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.879+0000 7f1d3d8b6700 1 -- 172.21.15.59:0/4218767204 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d38151fa0 con 0x7f1d380ae430 2023-09-30T02:07:36.884 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.879+0000 7f1d3d8b6700 10 client.4501.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:07:36.884 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.879+0000 7f1d3d8b6700 10 monclient: _renew_subs 2023-09-30T02:07:36.884 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.879+0000 7f1d3d8b6700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:07:36.884 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.879+0000 7f1d3d8b6700 1 -- 172.21.15.59:0/4218767204 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d3811a5f0 con 0x7f1d380ae430 2023-09-30T02:07:36.885 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.879+0000 7f1d3d8b6700 1 librados: init done 2023-09-30T02:07:36.885 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.879+0000 7f1d1effd700 1 -- 172.21.15.59:0/4218767204 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d30011900 con 0x7f1d380ae430 2023-09-30T02:07:36.885 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.879+0000 7f1d1effd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:07:36.885 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.880+0000 7f1d357fa700 4 set_mon_vals no callback set 2023-09-30T02:07:36.885 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.880+0000 7f1d1effd700 1 -- 172.21.15.59:0/4218767204 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d300015d0 con 0x7f1d380ae430 2023-09-30T02:07:36.885 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.880+0000 7f1d1effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:07:36.885 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.880+0000 7f1d1effd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:07:36.885 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.880+0000 7f1d1effd700 10 monclient: dump: 2023-09-30T02:07:36.885 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:07:36.886 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:36.886 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:36.886 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:36.886 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:07:36.886 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:36.886 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:07:36.886 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:07:36.886 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:07:36.886 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:36.887 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.880+0000 7f1d3d8b6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:07:36.887 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.880+0000 7f1d3d8b6700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:07:36.887 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.880+0000 7f1d3d8b6700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:07:36.887 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.880+0000 7f1d3d8b6700 1 -- 172.21.15.59:0/4218767204 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d180050b0 con 0x7f1d380ae430 2023-09-30T02:07:36.901 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.902+0000 7f1d1effd700 1 -- 172.21.15.59:0/4218767204 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f1d30003030 con 0x7f1d380ae430 2023-09-30T02:07:36.902 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.902+0000 7f1d1effd700 1 --2- 172.21.15.59:0/4218767204 >> v2:172.21.15.110:6800/115870 conn(0x7f1d24068890 0x7f1d2406ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:36.902 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.903+0000 7f1d36ffd700 1 --2- 172.21.15.59:0/4218767204 >> v2:172.21.15.110:6800/115870 conn(0x7f1d24068890 0x7f1d2406ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:36.902 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.903+0000 7f1d36ffd700 10 monclient: get_auth_request con 0x7f1d24068890 auth_method 0 2023-09-30T02:07:36.902 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.903+0000 7f1d36ffd700 1 --2- 172.21.15.59:0/4218767204 >> v2:172.21.15.110:6800/115870 conn(0x7f1d24068890 0x7f1d2406ad40 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1d28001ad0 tx=0x7f1d28009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:07:36.903 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:36.903+0000 7f1d1effd700 10 client.4501.objecter ms_handle_connect 0x7f1d24068890 2023-09-30T02:07:37.134 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:37.135+0000 7f1d1effd700 1 -- 172.21.15.59:0/4218767204 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f1d30055ab0 con 0x7f1d380ae430 2023-09-30T02:07:37.134 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:37.135+0000 7f1d1effd700 10 client.4501.objecter ms_dispatch 0x7f1d38001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:07:37.134 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:37.135+0000 7f1d1effd700 3 client.4501.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:07:37.135 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:37.135+0000 7f1d1effd700 3 client.4501.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:07:37.135 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:37.135+0000 7f1d1effd700 20 client.4501.objecter dump_active .. 0 homeless 2023-09-30T02:07:37.144 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:37.145+0000 7f1d1effd700 1 -- 172.21.15.59:0/4218767204 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1d30051df0 con 0x7f1d380ae430 2023-09-30T02:07:37.145 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:37.145+0000 7f1d1effd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:07:37.145 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:37.145+0000 7f1d1effd700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:07:37.394 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:37.395+0000 7f1d3d8b6700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:07:37.395 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:37.395+0000 7f1d3d8b6700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:07:37.395 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:37.395+0000 7f1d3d8b6700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:07:37.395 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:37.395+0000 7f1d3d8b6700 1 -- 172.21.15.59:0/4218767204 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f1d18005630 con 0x7f1d380ae430 2023-09-30T02:07:37.402 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:37.403+0000 7f1d1effd700 1 -- 172.21.15.59:0/4218767204 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f1d30055410 con 0x7f1d380ae430 2023-09-30T02:07:37.403 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:37.403+0000 7f1d1effd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:07:37.403 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:37.403+0000 7f1d1effd700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:07:37.403 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:07:37.403 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:07:37.405 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:37.405+0000 7f1d3d8b6700 10 librados: watch_flush enter 2023-09-30T02:07:37.406 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:37.405+0000 7f1d3d8b6700 10 librados: watch_flush exit 2023-09-30T02:07:37.406 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:37.405+0000 7f1d3d8b6700 20 client.4501.objecter shutdown clearing up homeless session... 2023-09-30T02:07:37.406 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:37.405+0000 7f1d3d8b6700 10 client.4501.objecter successfully canceled tick 2023-09-30T02:07:37.406 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:37.405+0000 7f1d3d8b6700 1 -- 172.21.15.59:0/4218767204 >> v2:172.21.15.110:6800/115870 conn(0x7f1d24068890 msgr2=0x7f1d2406ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:37.406 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:37.405+0000 7f1d3d8b6700 1 --2- 172.21.15.59:0/4218767204 >> v2:172.21.15.110:6800/115870 conn(0x7f1d24068890 0x7f1d2406ad40 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1d28001ad0 tx=0x7f1d28009040 comp rx=0 tx=0).stop 2023-09-30T02:07:37.406 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:37.405+0000 7f1d3d8b6700 10 monclient: shutdown 2023-09-30T02:07:37.406 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:37.405+0000 7f1d3d8b6700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:07:37.406 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:37.405+0000 7f1d3d8b6700 1 -- 172.21.15.59:0/4218767204 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1d380ae430 msgr2=0x7f1d38119630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:37.406 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:37.405+0000 7f1d3d8b6700 1 --2- 172.21.15.59:0/4218767204 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1d380ae430 0x7f1d38119630 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f1d3000c7c0 tx=0x7f1d30001560 comp rx=0 tx=0).stop 2023-09-30T02:07:37.407 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:37.405+0000 7f1d3d8b6700 1 -- 172.21.15.59:0/4218767204 shutdown_connections 2023-09-30T02:07:37.407 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:37.405+0000 7f1d3d8b6700 1 --2- 172.21.15.59:0/4218767204 >> v2:172.21.15.110:6800/115870 conn(0x7f1d24068890 0x7f1d2406ad40 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:37.407 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:37.406+0000 7f1d3d8b6700 1 --2- 172.21.15.59:0/4218767204 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1d380ae430 0x7f1d38119630 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:37.407 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:37.406+0000 7f1d3d8b6700 1 --2- 172.21.15.59:0/4218767204 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1d380ac9b0 0x7f1d3813fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:37.407 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:37.406+0000 7f1d3d8b6700 1 --2- 172.21.15.59:0/4218767204 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1d380acd80 0x7f1d38151250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:37.407 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:37.406+0000 7f1d3d8b6700 1 -- 172.21.15.59:0/4218767204 >> 172.21.15.59:0/4218767204 conn(0x7f1d38069f90 msgr2=0x7f1d380ac030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:07:37.407 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:37.406+0000 7f1d3d8b6700 1 -- 172.21.15.59:0/4218767204 shutdown_connections 2023-09-30T02:07:37.407 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:37.406+0000 7f1d3d8b6700 1 -- 172.21.15.59:0/4218767204 wait complete. 2023-09-30T02:07:37.407 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:37.406+0000 7f1d3d8b6700 1 librados: shutdown 2023-09-30T02:07:40.419 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:07:40.558 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.558+0000 7fadbdd3b700 10 monclient: get_monmap_and_config 2023-09-30T02:07:40.558 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.558+0000 7fadbdd3b700 10 monclient: build_initial_monmap 2023-09-30T02:07:40.558 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.558+0000 7fadbdd3b700 10 monclient: monmap: 2023-09-30T02:07:40.558 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:07:40.558 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:40.558 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:07:40.559060+0000 2023-09-30T02:07:40.558 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:07:40.559060+0000 2023-09-30T02:07:40.559 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:07:40.559 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:40.559 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:07:40.559 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:07:40.559 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:07:40.559 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:07:40.559 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:07:40.559 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:40.561 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.559+0000 7fadbdd3b700 1 Processor -- start 2023-09-30T02:07:40.561 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.559+0000 7fadbdd3b700 1 -- start start 2023-09-30T02:07:40.561 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.559+0000 7fadbdd3b700 10 monclient: init 2023-09-30T02:07:40.561 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.560+0000 7fadbdd3b700 10 monclient: _reopen_session rank -1 2023-09-30T02:07:40.561 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.560+0000 7fadbdd3b700 10 monclient: _add_conns ranks=[3,4,0,1,2] 2023-09-30T02:07:40.561 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.560+0000 7fadbdd3b700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fadb80a5b70 0x7fadb80b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:40.561 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.560+0000 7fadbdd3b700 10 monclient(hunting): picked mon.noname-a con 0x7fadb80a5b70 addr v2:172.21.15.59:3300/0 2023-09-30T02:07:40.561 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.560+0000 7fadbdd3b700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fadb80a5f40 0x7fadb8141b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:40.561 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.560+0000 7fadbdd3b700 10 monclient(hunting): picked mon.noname-c con 0x7fadb80a5f40 addr v2:172.21.15.110:3300/0 2023-09-30T02:07:40.561 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.560+0000 7fadbdd3b700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fadb81420c0 addr v1:172.21.15.59:6789/0 2023-09-30T02:07:40.561 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.560+0000 7fadbdd3b700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadb80a75a0 con 0x7fadb81420c0 2023-09-30T02:07:40.562 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.560+0000 7fadbdd3b700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:40.562 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.560+0000 7fadbdd3b700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadb80a63e0 con 0x7fadb80a5b70 2023-09-30T02:07:40.562 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.560+0000 7fadbdd3b700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:40.562 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.560+0000 7fadbdd3b700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadb80af3f0 con 0x7fadb80a5f40 2023-09-30T02:07:40.562 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.560+0000 7fadbdd3b700 10 monclient(hunting): _renew_subs 2023-09-30T02:07:40.562 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.560+0000 7fadbdd3b700 10 monclient(hunting): authenticate will time out at 1905.862255s 2023-09-30T02:07:40.562 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.560+0000 7fadb77fe700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fadb80a5b70 0x7fadb80b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:40.562 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.561+0000 7fadb6ffd700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fadb80a5f40 0x7fadb8141b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:40.562 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.561+0000 7fadb77fe700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fadb80a5b70 0x7fadb80b1500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:41204/0 (socket says 172.21.15.59:41204) 2023-09-30T02:07:40.562 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.561+0000 7fadb77fe700 1 -- 172.21.15.59:0/623167309 learned_addr learned my addr 172.21.15.59:0/623167309 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:07:40.562 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.561+0000 7fadb77fe700 10 monclient(hunting): get_auth_request con 0x7fadb80a5b70 auth_method 0 2023-09-30T02:07:40.563 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.561+0000 7fadb77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:07:40.563 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.561+0000 7fadb77fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:07:40.563 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.561+0000 7fadb77fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:07:40.563 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.561+0000 7fadb6ffd700 10 monclient(hunting): get_auth_request con 0x7fadb80a5f40 auth_method 0 2023-09-30T02:07:40.563 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.561+0000 7fadb6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:07:40.563 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.561+0000 7fadb6ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:07:40.563 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.561+0000 7fadb6ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:07:40.564 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.562+0000 7fadb6ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:07:40.564 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.562+0000 7fadb6ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:07:40.564 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.562+0000 7fadb6ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:07:40.564 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.562+0000 7fadb77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:07:40.564 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.562+0000 7fadb77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:07:40.564 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.562+0000 7fadb77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:07:40.564 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.564+0000 7fadb6ffd700 10 monclient(hunting): handle_auth_done global_id 4504 payload 274 2023-09-30T02:07:40.564 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.564+0000 7fadb6ffd700 1 -- 172.21.15.59:0/623167309 >> v2:172.21.15.59:3300/0 conn(0x7fadb80a5b70 msgr2=0x7fadb80b1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:07:40.565 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.565+0000 7fadb6ffd700 1 --2- 172.21.15.59:0/623167309 >> v2:172.21.15.59:3300/0 conn(0x7fadb80a5b70 0x7fadb80b1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:40.565 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.565+0000 7fadb6ffd700 1 -- 172.21.15.59:0/623167309 >> v1:172.21.15.59:6789/0 conn(0x7fadb81420c0 legacy=0x7fadb813e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:40.565 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.565+0000 7fadb6ffd700 10 monclient: _finish_hunting 0 2023-09-30T02:07:40.565 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.565+0000 7fadb6ffd700 1 monclient: found mon.noname-c 2023-09-30T02:07:40.565 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.565+0000 7fadb6ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:07:40.565 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.565+0000 7fadb6ffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3300/0 2023-09-30T02:07:40.565 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.565+0000 7fadb6ffd700 1 -- 172.21.15.59:0/623167309 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadb81424e0 con 0x7fadb80a5f40 2023-09-30T02:07:40.566 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.567+0000 7fadb6ffd700 1 --2- 172.21.15.59:0/623167309 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fadb80a5f40 0x7fadb8141b80 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fadb80b1fc0 tx=0x7fada00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cccbdb37041de575 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:07:40.571 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.571+0000 7fadb67fc700 1 -- 172.21.15.59:0/623167309 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fada0011630 con 0x7fadb80a5f40 2023-09-30T02:07:40.571 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.571+0000 7fadb67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:07:40.571 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.571+0000 7fadb67fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:07:40.571 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.571+0000 7fadb67fc700 10 monclient: dump: 2023-09-30T02:07:40.571 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:07:40.571 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:40.571 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:40.571 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:40.571 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:07:40.571 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:40.571 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:07:40.571 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:07:40.572 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:07:40.572 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:40.572 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.571+0000 7fadb67fc700 10 monclient: _finish_auth 0 2023-09-30T02:07:40.572 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.571+0000 7fadb67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:07:40.572 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.571+0000 7fadb67fc700 1 -- 172.21.15.59:0/623167309 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fada0011790 con 0x7fadb80a5f40 2023-09-30T02:07:40.572 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.571+0000 7fadb67fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:07:40.572 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.571+0000 7fadb67fc700 1 -- 172.21.15.59:0/623167309 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fada0011b30 con 0x7fadb80a5f40 2023-09-30T02:07:40.572 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.571+0000 7fadb67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:07:40.572 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.571+0000 7fadb67fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:07:40.572 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.571+0000 7fadb67fc700 10 monclient: dump: 2023-09-30T02:07:40.572 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:07:40.572 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:40.573 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:40.573 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:40.573 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:07:40.573 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:40.573 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:07:40.573 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:07:40.573 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:07:40.573 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:40.573 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.571+0000 7fadbdd3b700 5 monclient: authenticate success, global_id 4504 2023-09-30T02:07:40.573 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.571+0000 7fadbdd3b700 10 monclient: get_monmap_and_config success 2023-09-30T02:07:40.573 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.571+0000 7fadbdd3b700 4 set_mon_vals no callback set 2023-09-30T02:07:40.576 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.572+0000 7fadbdd3b700 10 monclient: shutdown 2023-09-30T02:07:40.576 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.572+0000 7fadbdd3b700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:07:40.576 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.572+0000 7fadbdd3b700 1 -- 172.21.15.59:0/623167309 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fadb80a5f40 msgr2=0x7fadb8141b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:40.576 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.572+0000 7fadbdd3b700 1 --2- 172.21.15.59:0/623167309 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fadb80a5f40 0x7fadb8141b80 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fadb80b1fc0 tx=0x7fada00091a0 comp rx=0 tx=0).stop 2023-09-30T02:07:40.576 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.572+0000 7fadbdd3b700 1 -- 172.21.15.59:0/623167309 shutdown_connections 2023-09-30T02:07:40.576 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.572+0000 7fadbdd3b700 1 --2- 172.21.15.59:0/623167309 >> v2:172.21.15.59:3300/0 conn(0x7fadb80a5b70 0x7fadb80b1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:40.577 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.572+0000 7fadbdd3b700 1 --2- 172.21.15.59:0/623167309 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fadb80a5f40 0x7fadb8141b80 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:40.577 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.572+0000 7fadbdd3b700 1 -- 172.21.15.59:0/623167309 >> 172.21.15.59:0/623167309 conn(0x7fadb80631f0 msgr2=0x7fadb80635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:07:40.577 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.572+0000 7fadbdd3b700 1 -- 172.21.15.59:0/623167309 shutdown_connections 2023-09-30T02:07:40.577 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.572+0000 7fadbdd3b700 1 -- 172.21.15.59:0/623167309 wait complete. 2023-09-30T02:07:40.577 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.573+0000 7fadbdd3b700 10 monclient: build_initial_monmap 2023-09-30T02:07:40.577 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.573+0000 7fadbdd3b700 10 monclient: monmap: 2023-09-30T02:07:40.577 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:07:40.577 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:07:40.577 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:07:40.577 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:07:40.577 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:07:40.577 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:07:40.578 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:07:40.578 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:07:40.578 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:07:40.578 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:40.578 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.573+0000 7fadbdd3b700 1 librados: starting msgr at 2023-09-30T02:07:40.578 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.573+0000 7fadbdd3b700 1 librados: starting objecter 2023-09-30T02:07:40.578 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.573+0000 7fadbdd3b700 1 Processor -- start 2023-09-30T02:07:40.578 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.573+0000 7fadbdd3b700 1 -- start start 2023-09-30T02:07:40.578 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.573+0000 7fadbdd3b700 1 librados: setting wanted keys 2023-09-30T02:07:40.578 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.573+0000 7fadbdd3b700 1 librados: calling monclient init 2023-09-30T02:07:40.578 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.573+0000 7fadbdd3b700 10 monclient: init 2023-09-30T02:07:40.579 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.574+0000 7fadbdd3b700 10 monclient: _reopen_session rank -1 2023-09-30T02:07:40.579 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.574+0000 7fadbdd3b700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T02:07:40.579 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.574+0000 7fadbdd3b700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fadb80a5b70 0x7fadb8136930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:40.579 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.574+0000 7fadbdd3b700 10 monclient(hunting): picked mon.noname-c con 0x7fadb80a5b70 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:07:40.579 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.574+0000 7fadbdd3b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fadb80a5f40 0x7fadb8138c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:40.579 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.574+0000 7fadbdd3b700 10 monclient(hunting): picked mon.noname-a con 0x7fadb80a5f40 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:07:40.579 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.574+0000 7fadbdd3b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fadb81420c0 0x7fadb813cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:40.579 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.574+0000 7fadbdd3b700 10 monclient(hunting): picked mon.noname-b con 0x7fadb81420c0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:07:40.579 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.574+0000 7fadbdd3b700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:40.579 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.574+0000 7fadbdd3b700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadb813caf0 con 0x7fadb80a5f40 2023-09-30T02:07:40.579 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.574+0000 7fadbdd3b700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:40.579 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.574+0000 7fadbdd3b700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadb81391c0 con 0x7fadb81420c0 2023-09-30T02:07:40.579 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.574+0000 7fadbdd3b700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:40.580 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.574+0000 7fadbdd3b700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadb813f9e0 con 0x7fadb80a5b70 2023-09-30T02:07:40.580 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.574+0000 7fadbdd3b700 10 monclient(hunting): _renew_subs 2023-09-30T02:07:40.580 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.574+0000 7fadbdd3b700 10 monclient(hunting): authenticate will time out at 1905.876390s 2023-09-30T02:07:40.580 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.574+0000 7fadb7fff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fadb81420c0 0x7fadb813cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:40.580 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.575+0000 7fadb77fe700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fadb80a5b70 0x7fadb8136930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:40.580 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.575+0000 7fadb6ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fadb80a5f40 0x7fadb8138c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:40.580 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.575+0000 7fadb7fff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fadb81420c0 0x7fadb813cf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:54654/0 (socket says 172.21.15.59:54654) 2023-09-30T02:07:40.580 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.575+0000 7fadb7fff700 1 -- 172.21.15.59:0/2423947224 learned_addr learned my addr 172.21.15.59:0/2423947224 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:07:40.580 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.575+0000 7fadb7fff700 10 monclient(hunting): get_auth_request con 0x7fadb81420c0 auth_method 0 2023-09-30T02:07:40.580 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.575+0000 7fadb7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:07:40.580 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.575+0000 7fadb7fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:07:40.580 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.575+0000 7fadb7fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:07:40.581 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.575+0000 7fadb6ffd700 10 monclient(hunting): get_auth_request con 0x7fadb80a5f40 auth_method 0 2023-09-30T02:07:40.581 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.575+0000 7fadb6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:07:40.581 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.575+0000 7fadb6ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:07:40.581 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.575+0000 7fadb6ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:07:40.581 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.575+0000 7fadb77fe700 10 monclient(hunting): get_auth_request con 0x7fadb80a5b70 auth_method 0 2023-09-30T02:07:40.581 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.575+0000 7fadb77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:07:40.581 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.575+0000 7fadb77fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:07:40.581 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.575+0000 7fadb77fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:07:40.581 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.575+0000 7fadb7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:07:40.581 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.575+0000 7fadb7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:07:40.581 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.575+0000 7fadb7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:07:40.581 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.576+0000 7fadb6ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:07:40.582 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.576+0000 7fadb6ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:07:40.582 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.576+0000 7fadb6ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:07:40.582 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.576+0000 7fadb77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:07:40.582 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.576+0000 7fadb77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:07:40.582 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.576+0000 7fadb77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:07:40.583 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.579+0000 7fadb7fff700 10 monclient(hunting): handle_auth_done global_id 4507 payload 824 2023-09-30T02:07:40.583 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.579+0000 7fadb7fff700 1 -- 172.21.15.59:0/2423947224 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fadb80a5b70 msgr2=0x7fadb8136930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:40.583 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.579+0000 7fadb7fff700 1 --2- 172.21.15.59:0/2423947224 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fadb80a5b70 0x7fadb8136930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:40.583 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.579+0000 7fadb7fff700 1 -- 172.21.15.59:0/2423947224 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fadb80a5f40 msgr2=0x7fadb8138c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:40.583 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.579+0000 7fadb7fff700 1 --2- 172.21.15.59:0/2423947224 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fadb80a5f40 0x7fadb8138c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:40.583 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.579+0000 7fadb7fff700 10 monclient: _finish_hunting 0 2023-09-30T02:07:40.583 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.579+0000 7fadb7fff700 1 monclient: found mon.noname-b 2023-09-30T02:07:40.584 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.579+0000 7fadb7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:07:40.584 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.579+0000 7fadb7fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:07:40.584 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.579+0000 7fadb7fff700 1 -- 172.21.15.59:0/2423947224 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fada0009420 con 0x7fadb81420c0 2023-09-30T02:07:40.584 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.581+0000 7fadb7fff700 1 --2- 172.21.15.59:0/2423947224 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fadb81420c0 0x7fadb813cf80 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fadb000c870 tx=0x7fadb0001670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:07:40.584 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.581+0000 7fadaf7fe700 10 client.?.objecter ms_handle_connect 0x7fadb81420c0 2023-09-30T02:07:40.584 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.581+0000 7fadaf7fe700 10 client.?.objecter resend_mon_ops 2023-09-30T02:07:40.585 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.585+0000 7fadaf7fe700 1 -- 172.21.15.59:0/2423947224 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadb0012630 con 0x7fadb81420c0 2023-09-30T02:07:40.585 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.585+0000 7fadaf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:07:40.586 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.585+0000 7fadaf7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:07:40.586 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.585+0000 7fadaf7fe700 10 monclient: dump: 2023-09-30T02:07:40.586 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:07:40.586 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:40.586 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:40.586 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:40.586 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:07:40.586 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:40.586 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:07:40.586 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:07:40.586 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:07:40.587 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:40.587 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.585+0000 7fadaf7fe700 10 monclient: _finish_auth 0 2023-09-30T02:07:40.587 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.585+0000 7fadaf7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:07:40.587 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.585+0000 7fadaf7fe700 1 -- 172.21.15.59:0/2423947224 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadb00127b0 con 0x7fadb81420c0 2023-09-30T02:07:40.587 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.585+0000 7fadbdd3b700 5 monclient: authenticate success, global_id 4507 2023-09-30T02:07:40.587 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.585+0000 7fadbdd3b700 10 monclient: _renew_subs 2023-09-30T02:07:40.587 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.585+0000 7fadbdd3b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:07:40.587 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.585+0000 7fadbdd3b700 1 -- 172.21.15.59:0/2423947224 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadb813d580 con 0x7fadb81420c0 2023-09-30T02:07:40.587 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.585+0000 7fadaf7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:07:40.587 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.585+0000 7fadaf7fe700 1 -- 172.21.15.59:0/2423947224 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadb0012b70 con 0x7fadb81420c0 2023-09-30T02:07:40.587 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.585+0000 7fadaf7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:07:40.587 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.585+0000 7fadaf7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:07:40.588 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.585+0000 7fadaf7fe700 10 monclient: dump: 2023-09-30T02:07:40.588 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:07:40.588 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:40.588 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:40.588 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:40.588 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:07:40.588 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:40.588 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:07:40.588 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:07:40.588 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:07:40.588 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:40.588 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.585+0000 7fadb5ffb700 4 set_mon_vals no callback set 2023-09-30T02:07:40.589 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.585+0000 7fadbdd3b700 10 client.4507.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:07:40.589 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.585+0000 7fadbdd3b700 10 monclient: _renew_subs 2023-09-30T02:07:40.589 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.585+0000 7fadbdd3b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:07:40.589 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.585+0000 7fadbdd3b700 1 -- 172.21.15.59:0/2423947224 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadb8134d20 con 0x7fadb81420c0 2023-09-30T02:07:40.589 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.585+0000 7fadbdd3b700 1 librados: init done 2023-09-30T02:07:40.589 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.586+0000 7fadbdd3b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:07:40.589 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.586+0000 7fadbdd3b700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:07:40.589 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.586+0000 7fadbdd3b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:07:40.589 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.586+0000 7fadbdd3b700 1 -- 172.21.15.59:0/2423947224 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadb813cca0 con 0x7fadb81420c0 2023-09-30T02:07:40.833 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.834+0000 7fadaf7fe700 1 -- 172.21.15.59:0/2423947224 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fadb0003030 con 0x7fadb81420c0 2023-09-30T02:07:40.834 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.834+0000 7fadaf7fe700 1 --2- 172.21.15.59:0/2423947224 >> v2:172.21.15.110:6800/115870 conn(0x7fad98068890 0x7fad9806ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:40.834 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.835+0000 7fadb77fe700 1 --2- 172.21.15.59:0/2423947224 >> v2:172.21.15.110:6800/115870 conn(0x7fad98068890 0x7fad9806ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:40.834 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.835+0000 7fadb77fe700 10 monclient: get_auth_request con 0x7fad98068890 auth_method 0 2023-09-30T02:07:40.835 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.835+0000 7fadb77fe700 1 --2- 172.21.15.59:0/2423947224 >> v2:172.21.15.110:6800/115870 conn(0x7fad98068890 0x7fad9806ad40 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fada8006d40 tx=0x7fada8009250 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:07:40.835 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.835+0000 7fadaf7fe700 10 client.4507.objecter ms_handle_connect 0x7fad98068890 2023-09-30T02:07:40.838 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.839+0000 7fadaf7fe700 1 -- 172.21.15.59:0/2423947224 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fadb0056020 con 0x7fadb81420c0 2023-09-30T02:07:40.839 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.839+0000 7fadaf7fe700 10 client.4507.objecter ms_dispatch 0x7fadb8001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:07:40.839 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.839+0000 7fadaf7fe700 3 client.4507.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:07:40.839 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.839+0000 7fadaf7fe700 3 client.4507.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:07:40.839 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.839+0000 7fadaf7fe700 20 client.4507.objecter dump_active .. 0 homeless 2023-09-30T02:07:40.849 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.849+0000 7fadaf7fe700 1 -- 172.21.15.59:0/2423947224 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fadb0051c60 con 0x7fadb81420c0 2023-09-30T02:07:40.849 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.849+0000 7fadaf7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:07:40.849 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:40.849+0000 7fadaf7fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:07:41.109 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:41.110+0000 7fadbdd3b700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:07:41.110 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:41.110+0000 7fadbdd3b700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:07:41.110 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:41.110+0000 7fadbdd3b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:07:41.110 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:41.110+0000 7fadbdd3b700 1 -- 172.21.15.59:0/2423947224 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fadb813cca0 con 0x7fadb81420c0 2023-09-30T02:07:41.114 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:41.115+0000 7fadaf7fe700 1 -- 172.21.15.59:0/2423947224 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fadb813cca0 con 0x7fadb81420c0 2023-09-30T02:07:41.114 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:41.115+0000 7fadaf7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:07:41.114 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:41.115+0000 7fadaf7fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:07:41.115 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:07:41.115 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:07:41.117 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:41.117+0000 7fadbdd3b700 10 librados: watch_flush enter 2023-09-30T02:07:41.117 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:41.117+0000 7fadbdd3b700 10 librados: watch_flush exit 2023-09-30T02:07:41.117 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:41.117+0000 7fadbdd3b700 20 client.4507.objecter shutdown clearing up homeless session... 2023-09-30T02:07:41.118 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:41.117+0000 7fadbdd3b700 10 client.4507.objecter successfully canceled tick 2023-09-30T02:07:41.118 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:41.117+0000 7fadbdd3b700 1 -- 172.21.15.59:0/2423947224 >> v2:172.21.15.110:6800/115870 conn(0x7fad98068890 msgr2=0x7fad9806ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:41.118 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:41.117+0000 7fadbdd3b700 1 --2- 172.21.15.59:0/2423947224 >> v2:172.21.15.110:6800/115870 conn(0x7fad98068890 0x7fad9806ad40 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fada8006d40 tx=0x7fada8009250 comp rx=0 tx=0).stop 2023-09-30T02:07:41.118 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:41.117+0000 7fadbdd3b700 10 monclient: shutdown 2023-09-30T02:07:41.118 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:41.117+0000 7fadbdd3b700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:07:41.118 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:41.117+0000 7fadbdd3b700 1 -- 172.21.15.59:0/2423947224 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fadb81420c0 msgr2=0x7fadb813cf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:41.118 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:41.117+0000 7fadbdd3b700 1 --2- 172.21.15.59:0/2423947224 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fadb81420c0 0x7fadb813cf80 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fadb000c870 tx=0x7fadb0001670 comp rx=0 tx=0).stop 2023-09-30T02:07:41.118 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:41.117+0000 7fadbdd3b700 1 -- 172.21.15.59:0/2423947224 shutdown_connections 2023-09-30T02:07:41.118 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:41.117+0000 7fadbdd3b700 1 --2- 172.21.15.59:0/2423947224 >> v2:172.21.15.110:6800/115870 conn(0x7fad98068890 0x7fad9806ad40 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:41.118 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:41.117+0000 7fadbdd3b700 1 --2- 172.21.15.59:0/2423947224 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fadb81420c0 0x7fadb813cf80 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:41.118 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:41.117+0000 7fadbdd3b700 1 --2- 172.21.15.59:0/2423947224 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fadb80a5b70 0x7fadb8136930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:41.118 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:41.118+0000 7fadbdd3b700 1 --2- 172.21.15.59:0/2423947224 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fadb80a5f40 0x7fadb8138c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:41.119 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:41.118+0000 7fadbdd3b700 1 -- 172.21.15.59:0/2423947224 >> 172.21.15.59:0/2423947224 conn(0x7fadb80631f0 msgr2=0x7fadb80af560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:07:41.119 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:41.118+0000 7fadbdd3b700 1 -- 172.21.15.59:0/2423947224 shutdown_connections 2023-09-30T02:07:41.119 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:41.118+0000 7fadbdd3b700 1 -- 172.21.15.59:0/2423947224 wait complete. 2023-09-30T02:07:41.119 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:41.118+0000 7fadbdd3b700 1 librados: shutdown 2023-09-30T02:07:44.133 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:07:44.275 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.275+0000 7fb733d1e700 10 monclient: get_monmap_and_config 2023-09-30T02:07:44.275 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.275+0000 7fb733d1e700 10 monclient: build_initial_monmap 2023-09-30T02:07:44.276 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.276+0000 7fb733d1e700 10 monclient: monmap: 2023-09-30T02:07:44.276 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:07:44.276 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:44.276 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:07:44.276395+0000 2023-09-30T02:07:44.276 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:07:44.276395+0000 2023-09-30T02:07:44.276 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:07:44.276 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:44.276 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:07:44.276 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:07:44.276 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:07:44.276 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:07:44.277 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:07:44.277 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:44.277 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.276+0000 7fb733d1e700 1 Processor -- start 2023-09-30T02:07:44.278 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.276+0000 7fb733d1e700 1 -- start start 2023-09-30T02:07:44.278 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.276+0000 7fb733d1e700 10 monclient: init 2023-09-30T02:07:44.278 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.277+0000 7fb733d1e700 10 monclient: _reopen_session rank -1 2023-09-30T02:07:44.279 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.277+0000 7fb733d1e700 10 monclient: _add_conns ranks=[0,4,3,2,1] 2023-09-30T02:07:44.279 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.277+0000 7fb733d1e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb72c0a5b70 addr v1:172.21.15.59:6789/0 2023-09-30T02:07:44.279 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.277+0000 7fb733d1e700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fb72c142290 0x7fb72c0b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:44.279 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.277+0000 7fb733d1e700 10 monclient(hunting): picked mon.noname-c con 0x7fb72c142290 addr v2:172.21.15.110:3300/0 2023-09-30T02:07:44.279 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.277+0000 7fb733d1e700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fb72c0a5f40 0x7fb72c13af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:44.279 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.277+0000 7fb733d1e700 10 monclient(hunting): picked mon.noname-a con 0x7fb72c0a5f40 addr v2:172.21.15.59:3300/0 2023-09-30T02:07:44.279 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.277+0000 7fb733d1e700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb72c0a75a0 con 0x7fb72c0a5b70 2023-09-30T02:07:44.279 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.277+0000 7fb733d1e700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:44.279 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.277+0000 7fb733d1e700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb72c0a6360 con 0x7fb72c0a5f40 2023-09-30T02:07:44.279 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.277+0000 7fb733d1e700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:44.279 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.277+0000 7fb733d1e700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb72c0af3f0 con 0x7fb72c142290 2023-09-30T02:07:44.279 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.277+0000 7fb733d1e700 10 monclient(hunting): _renew_subs 2023-09-30T02:07:44.280 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.277+0000 7fb733d1e700 10 monclient(hunting): authenticate will time out at 1909.579603s 2023-09-30T02:07:44.280 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.278+0000 7fb7312b9700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fb72c142290 0x7fb72c0b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:44.280 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.278+0000 7fb7312b9700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fb72c142290 0x7fb72c0b1500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:54670/0 (socket says 172.21.15.59:54670) 2023-09-30T02:07:44.280 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.278+0000 7fb7322bb700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fb72c0a5f40 0x7fb72c13af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:44.280 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.278+0000 7fb7312b9700 1 -- 172.21.15.59:0/2420626141 learned_addr learned my addr 172.21.15.59:0/2420626141 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:07:44.280 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.278+0000 7fb7312b9700 10 monclient(hunting): get_auth_request con 0x7fb72c142290 auth_method 0 2023-09-30T02:07:44.280 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.278+0000 7fb7312b9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:07:44.280 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.278+0000 7fb7312b9700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:07:44.280 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.278+0000 7fb7312b9700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:07:44.280 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.279+0000 7fb7322bb700 10 monclient(hunting): get_auth_request con 0x7fb72c0a5f40 auth_method 0 2023-09-30T02:07:44.280 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.279+0000 7fb7322bb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:07:44.281 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.279+0000 7fb7322bb700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:07:44.281 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.279+0000 7fb7322bb700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:07:44.281 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.279+0000 7fb7312b9700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:07:44.281 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.279+0000 7fb7312b9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:07:44.281 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.279+0000 7fb7312b9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:07:44.282 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.280+0000 7fb7322bb700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:07:44.282 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.280+0000 7fb7322bb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:07:44.282 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.280+0000 7fb7322bb700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:07:44.282 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.281+0000 7fb7312b9700 10 monclient(hunting): handle_auth_done global_id 4510 payload 274 2023-09-30T02:07:44.282 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.281+0000 7fb7312b9700 1 -- 172.21.15.59:0/2420626141 >> v2:172.21.15.59:3300/0 conn(0x7fb72c0a5f40 msgr2=0x7fb72c13af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:07:44.282 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.281+0000 7fb7312b9700 1 --2- 172.21.15.59:0/2420626141 >> v2:172.21.15.59:3300/0 conn(0x7fb72c0a5f40 0x7fb72c13af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:44.282 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.281+0000 7fb7312b9700 1 -- 172.21.15.59:0/2420626141 >> v1:172.21.15.59:6789/0 conn(0x7fb72c0a5b70 legacy=0x7fb72c141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:44.282 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.281+0000 7fb7312b9700 10 monclient: _finish_hunting 0 2023-09-30T02:07:44.282 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.281+0000 7fb7312b9700 1 monclient: found mon.noname-c 2023-09-30T02:07:44.283 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.281+0000 7fb7312b9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:07:44.283 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.281+0000 7fb7312b9700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3300/0 2023-09-30T02:07:44.283 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.281+0000 7fb7312b9700 1 -- 172.21.15.59:0/2420626141 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb72c13b470 con 0x7fb72c142290 2023-09-30T02:07:44.283 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.283+0000 7fb7312b9700 1 --2- 172.21.15.59:0/2420626141 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb72c142290 0x7fb72c0b1500 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb72c0b1fc0 tx=0x7fb720009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=51dc2d60d4df1b9b server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:07:44.287 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.287+0000 7fb730ab8700 1 -- 172.21.15.59:0/2420626141 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb720009e70 con 0x7fb72c142290 2023-09-30T02:07:44.287 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.287+0000 7fb730ab8700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:07:44.287 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.287+0000 7fb730ab8700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:07:44.287 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.287+0000 7fb730ab8700 10 monclient: dump: 2023-09-30T02:07:44.287 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:07:44.287 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:44.287 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:44.287 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:44.287 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:07:44.287 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:44.288 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:07:44.288 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:07:44.288 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:07:44.288 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:44.288 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.287+0000 7fb730ab8700 10 monclient: _finish_auth 0 2023-09-30T02:07:44.288 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.287+0000 7fb730ab8700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:07:44.288 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.287+0000 7fb730ab8700 1 -- 172.21.15.59:0/2420626141 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb72000b860 con 0x7fb72c142290 2023-09-30T02:07:44.288 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.287+0000 7fb730ab8700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:07:44.288 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.287+0000 7fb730ab8700 1 -- 172.21.15.59:0/2420626141 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb72000bc00 con 0x7fb72c142290 2023-09-30T02:07:44.288 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.287+0000 7fb733d1e700 5 monclient: authenticate success, global_id 4510 2023-09-30T02:07:44.288 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.287+0000 7fb733d1e700 10 monclient: get_monmap_and_config success 2023-09-30T02:07:44.288 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.287+0000 7fb730ab8700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:07:44.289 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.287+0000 7fb733d1e700 4 set_mon_vals no callback set 2023-09-30T02:07:44.289 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.287+0000 7fb730ab8700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:07:44.289 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.287+0000 7fb730ab8700 10 monclient: dump: 2023-09-30T02:07:44.289 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:07:44.289 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:44.289 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:44.289 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:44.289 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:07:44.289 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:44.289 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:07:44.289 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:07:44.289 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:07:44.289 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:44.292 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.288+0000 7fb733d1e700 10 monclient: shutdown 2023-09-30T02:07:44.293 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.288+0000 7fb733d1e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:07:44.293 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.288+0000 7fb733d1e700 1 -- 172.21.15.59:0/2420626141 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb72c142290 msgr2=0x7fb72c0b1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:44.293 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.288+0000 7fb733d1e700 1 --2- 172.21.15.59:0/2420626141 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb72c142290 0x7fb72c0b1500 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb72c0b1fc0 tx=0x7fb720009250 comp rx=0 tx=0).stop 2023-09-30T02:07:44.293 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.288+0000 7fb733d1e700 1 -- 172.21.15.59:0/2420626141 shutdown_connections 2023-09-30T02:07:44.293 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.288+0000 7fb733d1e700 1 --2- 172.21.15.59:0/2420626141 >> v2:172.21.15.59:3300/0 conn(0x7fb72c0a5f40 0x7fb72c13af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:44.293 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.288+0000 7fb733d1e700 1 --2- 172.21.15.59:0/2420626141 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb72c142290 0x7fb72c0b1500 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:44.293 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.288+0000 7fb733d1e700 1 -- 172.21.15.59:0/2420626141 >> 172.21.15.59:0/2420626141 conn(0x7fb72c0631f0 msgr2=0x7fb72c0635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:07:44.293 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.289+0000 7fb733d1e700 1 -- 172.21.15.59:0/2420626141 shutdown_connections 2023-09-30T02:07:44.293 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.289+0000 7fb733d1e700 1 -- 172.21.15.59:0/2420626141 wait complete. 2023-09-30T02:07:44.293 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.289+0000 7fb733d1e700 10 monclient: build_initial_monmap 2023-09-30T02:07:44.293 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.289+0000 7fb733d1e700 10 monclient: monmap: 2023-09-30T02:07:44.294 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:07:44.294 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:07:44.294 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:07:44.294 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:07:44.294 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:07:44.294 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:07:44.294 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:07:44.294 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:07:44.295 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:07:44.295 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:44.295 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.289+0000 7fb733d1e700 1 librados: starting msgr at 2023-09-30T02:07:44.295 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.290+0000 7fb733d1e700 1 librados: starting objecter 2023-09-30T02:07:44.295 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.290+0000 7fb733d1e700 1 Processor -- start 2023-09-30T02:07:44.295 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.290+0000 7fb733d1e700 1 -- start start 2023-09-30T02:07:44.295 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.290+0000 7fb733d1e700 1 librados: setting wanted keys 2023-09-30T02:07:44.295 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.290+0000 7fb733d1e700 1 librados: calling monclient init 2023-09-30T02:07:44.295 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.290+0000 7fb733d1e700 10 monclient: init 2023-09-30T02:07:44.295 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.290+0000 7fb733d1e700 10 monclient: _reopen_session rank -1 2023-09-30T02:07:44.295 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.290+0000 7fb733d1e700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T02:07:44.295 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.290+0000 7fb733d1e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb72c0a5b70 0x7fb72c0ae3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:44.296 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.290+0000 7fb733d1e700 10 monclient(hunting): picked mon.noname-a con 0x7fb72c0a5b70 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:07:44.296 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.290+0000 7fb733d1e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb72c0a5f40 0x7fb72c140d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:44.296 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.290+0000 7fb733d1e700 10 monclient(hunting): picked mon.noname-b con 0x7fb72c0a5f40 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:07:44.296 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.290+0000 7fb733d1e700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb72c142290 0x7fb72c13e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:44.296 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.290+0000 7fb733d1e700 10 monclient(hunting): picked mon.noname-c con 0x7fb72c142290 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:07:44.296 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.290+0000 7fb733d1e700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:44.296 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.290+0000 7fb733d1e700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb72c13eb40 con 0x7fb72c0a5b70 2023-09-30T02:07:44.296 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.290+0000 7fb733d1e700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:44.296 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.290+0000 7fb733d1e700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb72c13de90 con 0x7fb72c0a5f40 2023-09-30T02:07:44.296 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.290+0000 7fb733d1e700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:44.296 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.290+0000 7fb733d1e700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb72c13e170 con 0x7fb72c142290 2023-09-30T02:07:44.296 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.290+0000 7fb733d1e700 10 monclient(hunting): _renew_subs 2023-09-30T02:07:44.297 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.290+0000 7fb733d1e700 10 monclient(hunting): authenticate will time out at 1909.592859s 2023-09-30T02:07:44.297 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.291+0000 7fb7312b9700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb72c0a5f40 0x7fb72c140d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:44.297 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.291+0000 7fb731aba700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb72c0a5b70 0x7fb72c0ae3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:44.297 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.291+0000 7fb7322bb700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb72c142290 0x7fb72c13e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:44.297 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.291+0000 7fb7312b9700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb72c0a5f40 0x7fb72c140d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:54672/0 (socket says 172.21.15.59:54672) 2023-09-30T02:07:44.297 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.291+0000 7fb7312b9700 1 -- 172.21.15.59:0/3496042629 learned_addr learned my addr 172.21.15.59:0/3496042629 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:07:44.297 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.291+0000 7fb7312b9700 10 monclient(hunting): get_auth_request con 0x7fb72c0a5f40 auth_method 0 2023-09-30T02:07:44.297 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.291+0000 7fb7312b9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:07:44.297 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.291+0000 7fb7312b9700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:07:44.297 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.291+0000 7fb7312b9700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:07:44.297 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.291+0000 7fb731aba700 10 monclient(hunting): get_auth_request con 0x7fb72c0a5b70 auth_method 0 2023-09-30T02:07:44.297 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.291+0000 7fb731aba700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:07:44.298 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.291+0000 7fb731aba700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:07:44.298 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.291+0000 7fb731aba700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:07:44.298 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.292+0000 7fb7322bb700 10 monclient(hunting): get_auth_request con 0x7fb72c142290 auth_method 0 2023-09-30T02:07:44.298 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.292+0000 7fb7322bb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:07:44.298 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.292+0000 7fb7322bb700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:07:44.298 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.292+0000 7fb7322bb700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:07:44.298 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.292+0000 7fb7312b9700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:07:44.298 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.292+0000 7fb7312b9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:07:44.298 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.292+0000 7fb7312b9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:07:44.298 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.292+0000 7fb731aba700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:07:44.298 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.292+0000 7fb731aba700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:07:44.299 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.292+0000 7fb731aba700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:07:44.299 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.292+0000 7fb7322bb700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:07:44.299 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.292+0000 7fb7322bb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:07:44.299 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.292+0000 7fb7322bb700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:07:44.299 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.295+0000 7fb7312b9700 10 monclient(hunting): handle_auth_done global_id 4513 payload 824 2023-09-30T02:07:44.300 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.295+0000 7fb7312b9700 1 -- 172.21.15.59:0/3496042629 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb72c142290 msgr2=0x7fb72c13e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:44.300 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.295+0000 7fb7312b9700 1 --2- 172.21.15.59:0/3496042629 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb72c142290 0x7fb72c13e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:44.300 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.295+0000 7fb7312b9700 1 -- 172.21.15.59:0/3496042629 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb72c0a5b70 msgr2=0x7fb72c0ae3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:44.300 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.295+0000 7fb7312b9700 1 --2- 172.21.15.59:0/3496042629 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb72c0a5b70 0x7fb72c0ae3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:44.300 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.295+0000 7fb7312b9700 10 monclient: _finish_hunting 0 2023-09-30T02:07:44.300 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.295+0000 7fb7312b9700 1 monclient: found mon.noname-b 2023-09-30T02:07:44.300 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.295+0000 7fb7312b9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:07:44.300 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.295+0000 7fb7312b9700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:07:44.300 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.295+0000 7fb7312b9700 1 -- 172.21.15.59:0/3496042629 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7200094f0 con 0x7fb72c0a5f40 2023-09-30T02:07:44.300 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.297+0000 7fb7312b9700 1 --2- 172.21.15.59:0/3496042629 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb72c0a5f40 0x7fb72c140d20 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb72000b3c0 tx=0x7fb720017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:07:44.300 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.297+0000 7fb71dffb700 10 client.?.objecter ms_handle_connect 0x7fb72c0a5f40 2023-09-30T02:07:44.301 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.297+0000 7fb71dffb700 10 client.?.objecter resend_mon_ops 2023-09-30T02:07:44.301 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.301+0000 7fb71dffb700 1 -- 172.21.15.59:0/3496042629 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb72000bbe0 con 0x7fb72c0a5f40 2023-09-30T02:07:44.302 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.301+0000 7fb71dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:07:44.302 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.301+0000 7fb71dffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:07:44.302 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.301+0000 7fb71dffb700 10 monclient: dump: 2023-09-30T02:07:44.302 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:07:44.302 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:44.302 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:44.302 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:44.302 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:07:44.302 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:44.302 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:07:44.303 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:07:44.303 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:07:44.303 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:44.303 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.301+0000 7fb71dffb700 10 monclient: _finish_auth 0 2023-09-30T02:07:44.303 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.301+0000 7fb71dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:07:44.303 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.301+0000 7fb71dffb700 1 -- 172.21.15.59:0/3496042629 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb720000c90 con 0x7fb72c0a5f40 2023-09-30T02:07:44.303 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.301+0000 7fb733d1e700 5 monclient: authenticate success, global_id 4513 2023-09-30T02:07:44.303 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.301+0000 7fb733d1e700 10 monclient: _renew_subs 2023-09-30T02:07:44.303 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.301+0000 7fb733d1e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:07:44.303 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.301+0000 7fb733d1e700 1 -- 172.21.15.59:0/3496042629 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb72c0a78a0 con 0x7fb72c0a5f40 2023-09-30T02:07:44.304 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.301+0000 7fb71dffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:07:44.304 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.301+0000 7fb71dffb700 1 -- 172.21.15.59:0/3496042629 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb720011670 con 0x7fb72c0a5f40 2023-09-30T02:07:44.304 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.301+0000 7fb71dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:07:44.304 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.301+0000 7fb71dffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:07:44.304 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.301+0000 7fb71dffb700 10 monclient: dump: 2023-09-30T02:07:44.304 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:07:44.304 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:44.304 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:44.304 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:44.304 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:07:44.304 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:44.304 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:07:44.305 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:07:44.305 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:07:44.305 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:44.305 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.302+0000 7fb733d1e700 10 client.4513.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:07:44.305 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.302+0000 7fb733d1e700 10 monclient: _renew_subs 2023-09-30T02:07:44.305 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.302+0000 7fb71ffff700 4 set_mon_vals no callback set 2023-09-30T02:07:44.305 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.302+0000 7fb733d1e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:07:44.305 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.302+0000 7fb733d1e700 1 -- 172.21.15.59:0/3496042629 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb72c151880 con 0x7fb72c0a5f40 2023-09-30T02:07:44.305 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.302+0000 7fb733d1e700 1 librados: init done 2023-09-30T02:07:44.306 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.303+0000 7fb733d1e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:07:44.306 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.303+0000 7fb733d1e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:07:44.306 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.303+0000 7fb733d1e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:07:44.306 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.303+0000 7fb733d1e700 1 -- 172.21.15.59:0/3496042629 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb72c064a00 con 0x7fb72c0a5f40 2023-09-30T02:07:44.545 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.546+0000 7fb71dffb700 1 -- 172.21.15.59:0/3496042629 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fb720011a60 con 0x7fb72c0a5f40 2023-09-30T02:07:44.546 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.547+0000 7fb71dffb700 1 --2- 172.21.15.59:0/3496042629 >> v2:172.21.15.110:6800/115870 conn(0x7fb710068960 0x7fb71006ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:44.546 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.547+0000 7fb731aba700 1 --2- 172.21.15.59:0/3496042629 >> v2:172.21.15.110:6800/115870 conn(0x7fb710068960 0x7fb71006ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:44.546 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.547+0000 7fb731aba700 10 monclient: get_auth_request con 0x7fb710068960 auth_method 0 2023-09-30T02:07:44.547 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.547+0000 7fb71dffb700 10 client.4513.objecter ms_handle_connect 0x7fb710068960 2023-09-30T02:07:44.547 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.547+0000 7fb731aba700 1 --2- 172.21.15.59:0/3496042629 >> v2:172.21.15.110:6800/115870 conn(0x7fb710068960 0x7fb71006ae10 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb728003aa0 tx=0x7fb728000bc0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:07:44.549 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.550+0000 7fb71dffb700 1 -- 172.21.15.59:0/3496042629 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fb7200573e0 con 0x7fb72c0a5f40 2023-09-30T02:07:44.550 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.550+0000 7fb71dffb700 10 client.4513.objecter ms_dispatch 0x7fb72c001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:07:44.550 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.550+0000 7fb71dffb700 3 client.4513.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:07:44.550 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.550+0000 7fb71dffb700 3 client.4513.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:07:44.550 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.550+0000 7fb71dffb700 20 client.4513.objecter dump_active .. 0 homeless 2023-09-30T02:07:44.559 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.559+0000 7fb71dffb700 1 -- 172.21.15.59:0/3496042629 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb720053020 con 0x7fb72c0a5f40 2023-09-30T02:07:44.559 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.559+0000 7fb71dffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:07:44.559 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.559+0000 7fb71dffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:07:44.821 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.822+0000 7fb733d1e700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:07:44.821 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.822+0000 7fb733d1e700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:07:44.821 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.822+0000 7fb733d1e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:07:44.821 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.822+0000 7fb733d1e700 1 -- 172.21.15.59:0/3496042629 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fb72c06a720 con 0x7fb72c0a5f40 2023-09-30T02:07:44.825 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.826+0000 7fb71dffb700 1 -- 172.21.15.59:0/3496042629 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fb720015100 con 0x7fb72c0a5f40 2023-09-30T02:07:44.825 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.826+0000 7fb71dffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:07:44.825 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.826+0000 7fb71dffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:07:44.825 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:07:44.825 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:07:44.828 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.828+0000 7fb733d1e700 10 librados: watch_flush enter 2023-09-30T02:07:44.828 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.828+0000 7fb733d1e700 10 librados: watch_flush exit 2023-09-30T02:07:44.828 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.828+0000 7fb733d1e700 20 client.4513.objecter shutdown clearing up homeless session... 2023-09-30T02:07:44.829 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.828+0000 7fb733d1e700 10 client.4513.objecter successfully canceled tick 2023-09-30T02:07:44.829 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.828+0000 7fb733d1e700 1 -- 172.21.15.59:0/3496042629 >> v2:172.21.15.110:6800/115870 conn(0x7fb710068960 msgr2=0x7fb71006ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:44.829 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.828+0000 7fb733d1e700 1 --2- 172.21.15.59:0/3496042629 >> v2:172.21.15.110:6800/115870 conn(0x7fb710068960 0x7fb71006ae10 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb728003aa0 tx=0x7fb728000bc0 comp rx=0 tx=0).stop 2023-09-30T02:07:44.829 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.828+0000 7fb733d1e700 10 monclient: shutdown 2023-09-30T02:07:44.829 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.828+0000 7fb733d1e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:07:44.829 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.828+0000 7fb733d1e700 1 -- 172.21.15.59:0/3496042629 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb72c0a5f40 msgr2=0x7fb72c140d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:44.829 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.828+0000 7fb733d1e700 1 --2- 172.21.15.59:0/3496042629 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb72c0a5f40 0x7fb72c140d20 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fb72000b3c0 tx=0x7fb720017000 comp rx=0 tx=0).stop 2023-09-30T02:07:44.829 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.828+0000 7fb733d1e700 1 -- 172.21.15.59:0/3496042629 shutdown_connections 2023-09-30T02:07:44.829 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.828+0000 7fb733d1e700 1 --2- 172.21.15.59:0/3496042629 >> v2:172.21.15.110:6800/115870 conn(0x7fb710068960 0x7fb71006ae10 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:44.829 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.828+0000 7fb733d1e700 1 --2- 172.21.15.59:0/3496042629 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb72c142290 0x7fb72c13e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:44.829 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.828+0000 7fb733d1e700 1 --2- 172.21.15.59:0/3496042629 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb72c0a5b70 0x7fb72c0ae3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:44.830 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.828+0000 7fb733d1e700 1 --2- 172.21.15.59:0/3496042629 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb72c0a5f40 0x7fb72c140d20 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:44.830 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.828+0000 7fb733d1e700 1 -- 172.21.15.59:0/3496042629 >> 172.21.15.59:0/3496042629 conn(0x7fb72c0631f0 msgr2=0x7fb72c0b0be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:07:44.830 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.828+0000 7fb733d1e700 1 -- 172.21.15.59:0/3496042629 shutdown_connections 2023-09-30T02:07:44.830 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.828+0000 7fb733d1e700 1 -- 172.21.15.59:0/3496042629 wait complete. 2023-09-30T02:07:44.830 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:44.828+0000 7fb733d1e700 1 librados: shutdown 2023-09-30T02:07:47.842 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:07:47.981 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.982+0000 7fa138183700 10 monclient: get_monmap_and_config 2023-09-30T02:07:47.982 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.982+0000 7fa138183700 10 monclient: build_initial_monmap 2023-09-30T02:07:47.982 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.982+0000 7fa138183700 10 monclient: monmap: 2023-09-30T02:07:47.982 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:07:47.982 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:47.982 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:07:47.982440+0000 2023-09-30T02:07:47.982 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:07:47.982440+0000 2023-09-30T02:07:47.982 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:07:47.982 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:47.982 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:07:47.983 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:07:47.983 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:07:47.983 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:07:47.983 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:07:47.983 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:47.983 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.982+0000 7fa138183700 1 Processor -- start 2023-09-30T02:07:47.985 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.982+0000 7fa138183700 1 -- start start 2023-09-30T02:07:47.985 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.982+0000 7fa138183700 10 monclient: init 2023-09-30T02:07:47.985 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.983+0000 7fa138183700 10 monclient: _reopen_session rank -1 2023-09-30T02:07:47.986 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.983+0000 7fa138183700 10 monclient: _add_conns ranks=[2,0,4,1,3] 2023-09-30T02:07:47.986 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.983+0000 7fa138183700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa1300a5b70 0x7fa1300b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:47.986 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.983+0000 7fa138183700 10 monclient(hunting): picked mon.noname-b con 0x7fa1300a5b70 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:07:47.986 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.983+0000 7fa138183700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa1300a5f40 addr v1:172.21.15.59:6789/0 2023-09-30T02:07:47.986 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.983+0000 7fa138183700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fa130142290 0x7fa13013af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:47.986 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.983+0000 7fa138183700 10 monclient(hunting): picked mon.noname-c con 0x7fa130142290 addr v2:172.21.15.110:3300/0 2023-09-30T02:07:47.986 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.983+0000 7fa138183700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1300a63b0 con 0x7fa1300a5f40 2023-09-30T02:07:47.986 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.983+0000 7fa138183700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:47.986 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.983+0000 7fa138183700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1301426d0 con 0x7fa130142290 2023-09-30T02:07:47.987 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.983+0000 7fa138183700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:47.987 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.983+0000 7fa138183700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1300a75a0 con 0x7fa1300a5b70 2023-09-30T02:07:47.987 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.983+0000 7fa138183700 10 monclient(hunting): _renew_subs 2023-09-30T02:07:47.987 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.983+0000 7fa138183700 10 monclient(hunting): authenticate will time out at 1913.285653s 2023-09-30T02:07:47.987 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.984+0000 7fa13571e700 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fa1300a5f40 0x7fa130141b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:43594/0 (socket says 172.21.15.59:43594) 2023-09-30T02:07:47.987 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.984+0000 7fa13571e700 1 -- 172.21.15.59:0/1367351048 learned_addr learned my addr 172.21.15.59:0/1367351048 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:07:47.987 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.984+0000 7fa135f1f700 1 --2- 172.21.15.59:0/1367351048 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa1300a5b70 0x7fa1300b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:47.987 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.984+0000 7fa136720700 1 --2- 172.21.15.59:0/1367351048 >> v2:172.21.15.110:3300/0 conn(0x7fa130142290 0x7fa13013af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:47.987 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.984+0000 7fa136720700 10 monclient(hunting): get_auth_request con 0x7fa130142290 auth_method 0 2023-09-30T02:07:47.987 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.984+0000 7fa136720700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:07:47.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.984+0000 7fa136720700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:07:47.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.984+0000 7fa136720700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:07:47.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.984+0000 7fa135f1f700 10 monclient(hunting): get_auth_request con 0x7fa1300a5b70 auth_method 0 2023-09-30T02:07:47.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.984+0000 7fa135f1f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:07:47.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.984+0000 7fa135f1f700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:07:47.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.984+0000 7fa135f1f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:07:47.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.985+0000 7fa136720700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:07:47.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.985+0000 7fa136720700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:07:47.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.985+0000 7fa136720700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:07:47.989 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.985+0000 7fa135f1f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:07:47.989 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.985+0000 7fa135f1f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:07:47.989 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.985+0000 7fa135f1f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:07:47.990 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.988+0000 7fa134f1d700 1 -- 172.21.15.59:0/1367351048 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 829534513 0 0) 0x7fa1300a63b0 con 0x7fa1300a5f40 2023-09-30T02:07:47.990 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.988+0000 7fa134f1d700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:07:47.990 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.988+0000 7fa134f1d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:07:47.990 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.988+0000 7fa134f1d700 10 monclient(hunting): my global_id is 4506 2023-09-30T02:07:47.991 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.988+0000 7fa134f1d700 1 -- 172.21.15.59:0/1367351048 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa11c003570 con 0x7fa1300a5f40 2023-09-30T02:07:47.991 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.988+0000 7fa135f1f700 10 monclient(hunting): handle_auth_done global_id 4448 payload 274 2023-09-30T02:07:47.991 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.988+0000 7fa135f1f700 1 -- 172.21.15.59:0/1367351048 >> v2:172.21.15.110:3300/0 conn(0x7fa130142290 msgr2=0x7fa13013af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:07:47.991 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.988+0000 7fa135f1f700 1 --2- 172.21.15.59:0/1367351048 >> v2:172.21.15.110:3300/0 conn(0x7fa130142290 0x7fa13013af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:47.991 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.988+0000 7fa135f1f700 1 -- 172.21.15.59:0/1367351048 >> v1:172.21.15.59:6789/0 conn(0x7fa1300a5f40 legacy=0x7fa130141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:47.991 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.988+0000 7fa135f1f700 10 monclient: _finish_hunting 0 2023-09-30T02:07:47.991 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.988+0000 7fa135f1f700 1 monclient: found mon.noname-b 2023-09-30T02:07:47.991 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.988+0000 7fa135f1f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:07:47.991 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.988+0000 7fa135f1f700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.59:3301/0 2023-09-30T02:07:47.992 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.988+0000 7fa135f1f700 1 -- 172.21.15.59:0/1367351048 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1300af3f0 con 0x7fa1300a5b70 2023-09-30T02:07:47.992 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.988+0000 7fa136720700 1 --2- 172.21.15.59:0/1367351048 >> v2:172.21.15.110:3300/0 conn(0x7fa130142290 0x7fa13013af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T02:07:47.992 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.990+0000 7fa135f1f700 1 --2- 172.21.15.59:0/1367351048 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa1300a5b70 0x7fa1300b1500 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa12c0098e0 tx=0x7fa12c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa84687f0bc1edcf server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:07:47.994 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.995+0000 7fa134f1d700 1 -- 172.21.15.59:0/1367351048 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa12c011630 con 0x7fa1300a5b70 2023-09-30T02:07:47.995 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.995+0000 7fa134f1d700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:07:47.995 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.995+0000 7fa134f1d700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:07:47.995 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.995+0000 7fa134f1d700 10 monclient: dump: 2023-09-30T02:07:47.995 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:07:47.995 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:47.995 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:47.995 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:47.995 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:07:47.996 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:47.996 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:07:47.996 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:07:47.996 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:07:47.996 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:47.996 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.995+0000 7fa134f1d700 10 monclient: _finish_auth 0 2023-09-30T02:07:47.996 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.995+0000 7fa134f1d700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:07:47.996 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.995+0000 7fa134f1d700 1 -- 172.21.15.59:0/1367351048 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa12c011790 con 0x7fa1300a5b70 2023-09-30T02:07:47.997 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.995+0000 7fa138183700 5 monclient: authenticate success, global_id 4448 2023-09-30T02:07:47.997 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.995+0000 7fa138183700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:07:47.997 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.995+0000 7fa134f1d700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:07:47.997 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.995+0000 7fa138183700 10 monclient: get_monmap_and_config success 2023-09-30T02:07:47.997 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.995+0000 7fa138183700 4 set_mon_vals no callback set 2023-09-30T02:07:47.997 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.995+0000 7fa134f1d700 1 -- 172.21.15.59:0/1367351048 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa12c011b30 con 0x7fa1300a5b70 2023-09-30T02:07:47.997 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.995+0000 7fa134f1d700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:07:47.997 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.995+0000 7fa134f1d700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:07:47.997 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.995+0000 7fa134f1d700 10 monclient: dump: 2023-09-30T02:07:47.997 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:07:47.998 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:47.998 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:47.998 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:47.998 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:07:47.998 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:47.998 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:07:47.998 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:07:47.998 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:07:47.998 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:48.002 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.996+0000 7fa138183700 10 monclient: shutdown 2023-09-30T02:07:48.002 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.996+0000 7fa138183700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:07:48.002 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.996+0000 7fa138183700 1 -- 172.21.15.59:0/1367351048 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa1300a5b70 msgr2=0x7fa1300b1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:48.002 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.996+0000 7fa138183700 1 --2- 172.21.15.59:0/1367351048 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa1300a5b70 0x7fa1300b1500 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa12c0098e0 tx=0x7fa12c0091a0 comp rx=0 tx=0).stop 2023-09-30T02:07:48.002 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.996+0000 7fa138183700 1 -- 172.21.15.59:0/1367351048 shutdown_connections 2023-09-30T02:07:48.002 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.996+0000 7fa138183700 1 --2- 172.21.15.59:0/1367351048 >> v2:172.21.15.110:3300/0 conn(0x7fa130142290 0x7fa13013af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:48.002 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.996+0000 7fa138183700 1 --2- 172.21.15.59:0/1367351048 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa1300a5b70 0x7fa1300b1500 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:48.002 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.996+0000 7fa138183700 1 -- 172.21.15.59:0/1367351048 >> 172.21.15.59:0/1367351048 conn(0x7fa1300631f0 msgr2=0x7fa1300635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:07:48.003 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.996+0000 7fa138183700 1 -- 172.21.15.59:0/1367351048 shutdown_connections 2023-09-30T02:07:48.003 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.996+0000 7fa138183700 1 -- 172.21.15.59:0/1367351048 wait complete. 2023-09-30T02:07:48.003 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.997+0000 7fa138183700 10 monclient: build_initial_monmap 2023-09-30T02:07:48.003 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.997+0000 7fa138183700 10 monclient: monmap: 2023-09-30T02:07:48.003 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:07:48.003 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:07:48.003 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:07:48.003 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:07:48.003 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:07:48.004 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:07:48.004 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:07:48.004 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:07:48.004 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:07:48.004 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:48.004 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.997+0000 7fa138183700 1 librados: starting msgr at 2023-09-30T02:07:48.004 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.997+0000 7fa138183700 1 librados: starting objecter 2023-09-30T02:07:48.004 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.997+0000 7fa138183700 1 Processor -- start 2023-09-30T02:07:48.004 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.997+0000 7fa138183700 1 -- start start 2023-09-30T02:07:48.004 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.997+0000 7fa138183700 1 librados: setting wanted keys 2023-09-30T02:07:48.005 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.997+0000 7fa138183700 1 librados: calling monclient init 2023-09-30T02:07:48.005 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.997+0000 7fa138183700 10 monclient: init 2023-09-30T02:07:48.005 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.998+0000 7fa138183700 10 monclient: _reopen_session rank -1 2023-09-30T02:07:48.005 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.998+0000 7fa138183700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T02:07:48.005 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.998+0000 7fa138183700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa1300a5b70 0x7fa1300a7c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:48.005 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.998+0000 7fa138183700 10 monclient(hunting): picked mon.noname-a con 0x7fa1300a5b70 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:07:48.005 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.998+0000 7fa138183700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa1300a5f40 0x7fa13013c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:48.005 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.998+0000 7fa138183700 10 monclient(hunting): picked mon.noname-c con 0x7fa1300a5f40 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:07:48.005 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.998+0000 7fa138183700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa130142290 0x7fa13013caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:48.005 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.998+0000 7fa138183700 10 monclient(hunting): picked mon.noname-b con 0x7fa130142290 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:07:48.005 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.998+0000 7fa138183700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:48.006 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.998+0000 7fa138183700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa130137a40 con 0x7fa1300a5b70 2023-09-30T02:07:48.006 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.998+0000 7fa138183700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:48.006 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.998+0000 7fa138183700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1300a8360 con 0x7fa130142290 2023-09-30T02:07:48.006 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.998+0000 7fa138183700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:48.006 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.998+0000 7fa138183700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1300a9ce0 con 0x7fa1300a5f40 2023-09-30T02:07:48.006 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.998+0000 7fa138183700 10 monclient(hunting): _renew_subs 2023-09-30T02:07:48.006 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.998+0000 7fa138183700 10 monclient(hunting): authenticate will time out at 1913.300502s 2023-09-30T02:07:48.006 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.999+0000 7fa13571e700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa1300a5f40 0x7fa13013c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:48.006 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.999+0000 7fa136720700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa130142290 0x7fa13013caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:48.007 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.999+0000 7fa136720700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa130142290 0x7fa13013caf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:48140/0 (socket says 172.21.15.59:48140) 2023-09-30T02:07:48.007 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.999+0000 7fa136720700 1 -- 172.21.15.59:0/3054046065 learned_addr learned my addr 172.21.15.59:0/3054046065 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:07:48.007 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.999+0000 7fa136720700 10 monclient(hunting): get_auth_request con 0x7fa130142290 auth_method 0 2023-09-30T02:07:48.007 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.999+0000 7fa136720700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:07:48.007 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.999+0000 7fa136720700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:07:48.007 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.999+0000 7fa136720700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:07:48.007 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.999+0000 7fa13571e700 10 monclient(hunting): get_auth_request con 0x7fa1300a5f40 auth_method 0 2023-09-30T02:07:48.007 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.999+0000 7fa13571e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:07:48.007 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.999+0000 7fa13571e700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:07:48.007 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:47.999+0000 7fa13571e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:07:48.007 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.000+0000 7fa136720700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:07:48.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.000+0000 7fa136720700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:07:48.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.000+0000 7fa136720700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:07:48.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.000+0000 7fa13571e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:07:48.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.000+0000 7fa13571e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:07:48.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.000+0000 7fa13571e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:07:48.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.000+0000 7fa135f1f700 1 -- 172.21.15.59:0/3054046065 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa1300a5b70 msgr2=0x7fa1300a7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2023-09-30T02:07:48.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.000+0000 7fa135f1f700 1 -- 172.21.15.59:0/3054046065 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa1300a5b70 msgr2=0x7fa1300a7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-09-30T02:07:48.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.000+0000 7fa135f1f700 1 --2- 172.21.15.59:0/3054046065 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa1300a5b70 0x7fa1300a7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner read peer banner failed r=-1 ((1) Operation not permitted) 2023-09-30T02:07:48.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.000+0000 7fa135f1f700 1 --2- 172.21.15.59:0/3054046065 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa1300a5b70 0x7fa1300a7c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-30T02:07:48.010 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.003+0000 7fa136720700 10 monclient(hunting): handle_auth_done global_id 4519 payload 824 2023-09-30T02:07:48.010 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.003+0000 7fa136720700 1 -- 172.21.15.59:0/3054046065 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa1300a5f40 msgr2=0x7fa13013c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:48.010 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.003+0000 7fa136720700 1 --2- 172.21.15.59:0/3054046065 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa1300a5f40 0x7fa13013c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:48.010 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.003+0000 7fa136720700 1 -- 172.21.15.59:0/3054046065 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa1300a5b70 msgr2=0x7fa1300a7c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-09-30T02:07:48.010 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.003+0000 7fa136720700 1 --2- 172.21.15.59:0/3054046065 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa1300a5b70 0x7fa1300a7c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:48.010 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.003+0000 7fa136720700 10 monclient: _finish_hunting 0 2023-09-30T02:07:48.010 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.003+0000 7fa136720700 1 monclient: found mon.noname-b 2023-09-30T02:07:48.011 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.003+0000 7fa136720700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:07:48.011 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.003+0000 7fa136720700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:07:48.011 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.003+0000 7fa136720700 1 -- 172.21.15.59:0/3054046065 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa128009a20 con 0x7fa130142290 2023-09-30T02:07:48.011 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.006+0000 7fa136720700 1 --2- 172.21.15.59:0/3054046065 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa130142290 0x7fa13013caf0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fa128009f80 tx=0x7fa12800b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:07:48.011 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.006+0000 7fa125ffb700 10 client.?.objecter ms_handle_connect 0x7fa130142290 2023-09-30T02:07:48.011 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.006+0000 7fa125ffb700 10 client.?.objecter resend_mon_ops 2023-09-30T02:07:48.011 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.010+0000 7fa125ffb700 1 -- 172.21.15.59:0/3054046065 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa12800cc80 con 0x7fa130142290 2023-09-30T02:07:48.011 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.010+0000 7fa125ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:07:48.011 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.010+0000 7fa125ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:07:48.012 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.010+0000 7fa125ffb700 10 monclient: dump: 2023-09-30T02:07:48.012 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:07:48.012 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:48.012 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:48.012 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:48.012 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:07:48.012 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:48.012 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:07:48.012 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:07:48.013 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:07:48.013 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:48.013 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.010+0000 7fa125ffb700 10 monclient: _finish_auth 0 2023-09-30T02:07:48.013 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.010+0000 7fa125ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:07:48.013 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.010+0000 7fa138183700 5 monclient: authenticate success, global_id 4519 2023-09-30T02:07:48.013 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.010+0000 7fa138183700 10 monclient: _renew_subs 2023-09-30T02:07:48.013 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.010+0000 7fa138183700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:07:48.013 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.010+0000 7fa138183700 1 -- 172.21.15.59:0/3054046065 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa12c009420 con 0x7fa130142290 2023-09-30T02:07:48.013 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.010+0000 7fa138183700 10 client.4519.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:07:48.014 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.010+0000 7fa138183700 10 monclient: _renew_subs 2023-09-30T02:07:48.014 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.010+0000 7fa138183700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:07:48.014 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.010+0000 7fa138183700 1 -- 172.21.15.59:0/3054046065 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa13013e2f0 con 0x7fa130142290 2023-09-30T02:07:48.014 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.010+0000 7fa138183700 1 librados: init done 2023-09-30T02:07:48.014 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.011+0000 7fa138183700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:07:48.015 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.011+0000 7fa138183700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:07:48.015 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.011+0000 7fa138183700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:07:48.015 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.011+0000 7fa138183700 1 -- 172.21.15.59:0/3054046065 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1300a8600 con 0x7fa130142290 2023-09-30T02:07:48.262 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.262+0000 7fa125ffb700 1 -- 172.21.15.59:0/3054046065 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa12800f040 con 0x7fa130142290 2023-09-30T02:07:48.262 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.262+0000 7fa125ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:07:48.262 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.262+0000 7fa125ffb700 1 -- 172.21.15.59:0/3054046065 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa128014480 con 0x7fa130142290 2023-09-30T02:07:48.262 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.262+0000 7fa125ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:07:48.262 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.262+0000 7fa125ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:07:48.262 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.262+0000 7fa125ffb700 10 monclient: dump: 2023-09-30T02:07:48.262 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:07:48.262 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:48.262 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:48.263 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:48.263 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:07:48.263 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:48.263 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:07:48.263 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:07:48.263 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:07:48.263 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:48.263 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.262+0000 7fa127fff700 4 set_mon_vals no callback set 2023-09-30T02:07:48.264 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.263+0000 7fa125ffb700 1 -- 172.21.15.59:0/3054046065 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fa128003020 con 0x7fa130142290 2023-09-30T02:07:48.264 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.263+0000 7fa125ffb700 1 --2- 172.21.15.59:0/3054046065 >> v2:172.21.15.110:6800/115870 conn(0x7fa114068890 0x7fa11406ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:48.264 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.263+0000 7fa135f1f700 1 --2- 172.21.15.59:0/3054046065 >> v2:172.21.15.110:6800/115870 conn(0x7fa114068890 0x7fa11406ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:48.264 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.263+0000 7fa135f1f700 10 monclient: get_auth_request con 0x7fa114068890 auth_method 0 2023-09-30T02:07:48.264 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.264+0000 7fa135f1f700 1 --2- 172.21.15.59:0/3054046065 >> v2:172.21.15.110:6800/115870 conn(0x7fa114068890 0x7fa11406ad40 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa12c0091e0 tx=0x7fa12c017000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:07:48.264 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.264+0000 7fa125ffb700 10 client.4519.objecter ms_handle_connect 0x7fa114068890 2023-09-30T02:07:48.267 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.268+0000 7fa125ffb700 1 -- 172.21.15.59:0/3054046065 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fa128056ce0 con 0x7fa130142290 2023-09-30T02:07:48.267 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.268+0000 7fa125ffb700 10 client.4519.objecter ms_dispatch 0x7fa130001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:07:48.267 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.268+0000 7fa125ffb700 3 client.4519.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:07:48.267 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.268+0000 7fa125ffb700 3 client.4519.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:07:48.268 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.268+0000 7fa125ffb700 20 client.4519.objecter dump_active .. 0 homeless 2023-09-30T02:07:48.278 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.279+0000 7fa125ffb700 1 -- 172.21.15.59:0/3054046065 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa128052920 con 0x7fa130142290 2023-09-30T02:07:48.279 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.279+0000 7fa125ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:07:48.279 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.279+0000 7fa125ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:07:48.533 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.534+0000 7fa138183700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:07:48.534 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.534+0000 7fa138183700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:07:48.534 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.534+0000 7fa138183700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:07:48.534 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.534+0000 7fa138183700 1 -- 172.21.15.59:0/3054046065 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fa130064a00 con 0x7fa130142290 2023-09-30T02:07:48.539 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.540+0000 7fa125ffb700 1 -- 172.21.15.59:0/3054046065 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fa128018080 con 0x7fa130142290 2023-09-30T02:07:48.540 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.540+0000 7fa125ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:07:48.540 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.540+0000 7fa125ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:07:48.540 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:07:48.540 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:07:48.542 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.542+0000 7fa138183700 10 librados: watch_flush enter 2023-09-30T02:07:48.543 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.542+0000 7fa138183700 10 librados: watch_flush exit 2023-09-30T02:07:48.543 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.542+0000 7fa138183700 20 client.4519.objecter shutdown clearing up homeless session... 2023-09-30T02:07:48.543 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.542+0000 7fa138183700 10 client.4519.objecter successfully canceled tick 2023-09-30T02:07:48.543 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.542+0000 7fa138183700 1 -- 172.21.15.59:0/3054046065 >> v2:172.21.15.110:6800/115870 conn(0x7fa114068890 msgr2=0x7fa11406ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:48.543 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.542+0000 7fa138183700 1 --2- 172.21.15.59:0/3054046065 >> v2:172.21.15.110:6800/115870 conn(0x7fa114068890 0x7fa11406ad40 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa12c0091e0 tx=0x7fa12c017000 comp rx=0 tx=0).stop 2023-09-30T02:07:48.543 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.542+0000 7fa138183700 10 monclient: shutdown 2023-09-30T02:07:48.543 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.542+0000 7fa138183700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:07:48.543 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.542+0000 7fa138183700 1 -- 172.21.15.59:0/3054046065 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa130142290 msgr2=0x7fa13013caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:48.543 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.542+0000 7fa138183700 1 --2- 172.21.15.59:0/3054046065 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa130142290 0x7fa13013caf0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fa128009f80 tx=0x7fa12800b040 comp rx=0 tx=0).stop 2023-09-30T02:07:48.543 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.543+0000 7fa138183700 1 -- 172.21.15.59:0/3054046065 shutdown_connections 2023-09-30T02:07:48.544 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.543+0000 7fa138183700 1 --2- 172.21.15.59:0/3054046065 >> v2:172.21.15.110:6800/115870 conn(0x7fa114068890 0x7fa11406ad40 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:48.544 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.543+0000 7fa138183700 1 --2- 172.21.15.59:0/3054046065 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa130142290 0x7fa13013caf0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:48.544 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.543+0000 7fa138183700 1 --2- 172.21.15.59:0/3054046065 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa1300a5b70 0x7fa1300a7c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:48.544 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.543+0000 7fa138183700 1 --2- 172.21.15.59:0/3054046065 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa1300a5f40 0x7fa13013c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:48.544 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.543+0000 7fa138183700 1 -- 172.21.15.59:0/3054046065 >> 172.21.15.59:0/3054046065 conn(0x7fa1300631f0 msgr2=0x7fa1300b3220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:07:48.544 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.543+0000 7fa138183700 1 -- 172.21.15.59:0/3054046065 shutdown_connections 2023-09-30T02:07:48.544 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.543+0000 7fa138183700 1 -- 172.21.15.59:0/3054046065 wait complete. 2023-09-30T02:07:48.544 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:48.543+0000 7fa138183700 1 librados: shutdown 2023-09-30T02:07:51.557 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:07:51.706 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.705+0000 7fb43d21d700 10 monclient: get_monmap_and_config 2023-09-30T02:07:51.706 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.705+0000 7fb43d21d700 10 monclient: build_initial_monmap 2023-09-30T02:07:51.707 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.705+0000 7fb43d21d700 10 monclient: monmap: 2023-09-30T02:07:51.707 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:07:51.707 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:51.707 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:07:51.707269+0000 2023-09-30T02:07:51.707 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:07:51.707269+0000 2023-09-30T02:07:51.707 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:07:51.707 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:51.708 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:07:51.708 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:07:51.708 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:07:51.708 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:07:51.708 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:07:51.708 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:51.708 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.706+0000 7fb43d21d700 1 Processor -- start 2023-09-30T02:07:51.708 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.706+0000 7fb43d21d700 1 -- start start 2023-09-30T02:07:51.709 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.706+0000 7fb43d21d700 10 monclient: init 2023-09-30T02:07:51.710 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.707+0000 7fb43d21d700 10 monclient: _reopen_session rank -1 2023-09-30T02:07:51.710 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.707+0000 7fb43d21d700 10 monclient: _add_conns ranks=[0,1,4,3,2] 2023-09-30T02:07:51.710 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.707+0000 7fb43d21d700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb4380a5b70 addr v1:172.21.15.59:6789/0 2023-09-30T02:07:51.710 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.707+0000 7fb43d21d700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb438142290 addr v1:172.21.15.110:6789/0 2023-09-30T02:07:51.711 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.707+0000 7fb43d21d700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fb4380b1500 0x7fb4380a5f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:51.711 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.707+0000 7fb43d21d700 10 monclient(hunting): picked mon.noname-c con 0x7fb4380b1500 addr v2:172.21.15.110:3300/0 2023-09-30T02:07:51.711 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.707+0000 7fb43d21d700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4380b1920 con 0x7fb4380a5b70 2023-09-30T02:07:51.711 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.707+0000 7fb43d21d700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4380a75a0 con 0x7fb438142290 2023-09-30T02:07:51.711 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.707+0000 7fb43d21d700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:51.711 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.707+0000 7fb43d21d700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4380af3f0 con 0x7fb4380b1500 2023-09-30T02:07:51.711 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.707+0000 7fb43d21d700 10 monclient(hunting): _renew_subs 2023-09-30T02:07:51.712 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.707+0000 7fb43d21d700 10 monclient(hunting): authenticate will time out at 1917.010270s 2023-09-30T02:07:51.712 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.707+0000 7fb43759e700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fb4380b1500 0x7fb4380a5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:51.712 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.707+0000 7fb436d9d700 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fb4380a5b70 0x7fb438141b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:43604/0 (socket says 172.21.15.59:43604) 2023-09-30T02:07:51.712 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.708+0000 7fb436d9d700 1 -- 172.21.15.59:0/433930318 learned_addr learned my addr 172.21.15.59:0/433930318 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:07:51.712 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.708+0000 7fb43759e700 10 monclient(hunting): get_auth_request con 0x7fb4380b1500 auth_method 0 2023-09-30T02:07:51.712 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.708+0000 7fb43759e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:07:51.712 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.708+0000 7fb43759e700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:07:51.712 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.708+0000 7fb43759e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:07:51.713 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.709+0000 7fb43759e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:07:51.713 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.709+0000 7fb43759e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:07:51.713 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.709+0000 7fb43759e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:07:51.714 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.711+0000 7fb435d9b700 1 -- 172.21.15.59:0/433930318 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1340281991 0 0) 0x7fb428001110 con 0x7fb4380a5b70 2023-09-30T02:07:51.714 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.711+0000 7fb435d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:07:51.714 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.711+0000 7fb435d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:07:51.714 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.711+0000 7fb435d9b700 10 monclient(hunting): my global_id is 4509 2023-09-30T02:07:51.715 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.711+0000 7fb435d9b700 1 -- 172.21.15.59:0/433930318 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb420003570 con 0x7fb4380a5b70 2023-09-30T02:07:51.715 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.712+0000 7fb43759e700 10 monclient(hunting): handle_auth_done global_id 4522 payload 274 2023-09-30T02:07:51.715 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.712+0000 7fb43759e700 1 -- 172.21.15.59:0/433930318 >> v1:172.21.15.110:6789/0 conn(0x7fb438142290 legacy=0x7fb43813af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:51.715 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.712+0000 7fb43759e700 1 -- 172.21.15.59:0/433930318 >> v1:172.21.15.59:6789/0 conn(0x7fb4380a5b70 legacy=0x7fb438141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:51.715 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.712+0000 7fb43759e700 10 monclient: _finish_hunting 0 2023-09-30T02:07:51.715 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.712+0000 7fb43759e700 1 monclient: found mon.noname-c 2023-09-30T02:07:51.715 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.712+0000 7fb43759e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:07:51.715 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.712+0000 7fb43759e700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3300/0 2023-09-30T02:07:51.716 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.712+0000 7fb43759e700 1 -- 172.21.15.59:0/433930318 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb43813b640 con 0x7fb4380b1500 2023-09-30T02:07:51.716 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.715+0000 7fb43759e700 1 --2- 172.21.15.59:0/433930318 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb4380b1500 0x7fb4380a5f40 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb430009e60 tx=0x7fb430009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9edbaaccd3e8b4c0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:07:51.723 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.722+0000 7fb435d9b700 1 -- 172.21.15.59:0/433930318 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb43000e070 con 0x7fb4380b1500 2023-09-30T02:07:51.723 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.722+0000 7fb435d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:07:51.723 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.722+0000 7fb435d9b700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:07:51.723 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.722+0000 7fb435d9b700 10 monclient: dump: 2023-09-30T02:07:51.723 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:07:51.723 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:51.724 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:51.724 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:51.724 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:07:51.724 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:51.725 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:07:51.725 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:07:51.725 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:07:51.725 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:51.725 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.722+0000 7fb435d9b700 10 monclient: _finish_auth 0 2023-09-30T02:07:51.725 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.722+0000 7fb435d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:07:51.725 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.722+0000 7fb435d9b700 1 -- 172.21.15.59:0/433930318 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb43000bb60 con 0x7fb4380b1500 2023-09-30T02:07:51.725 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.722+0000 7fb43d21d700 5 monclient: authenticate success, global_id 4522 2023-09-30T02:07:51.725 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.722+0000 7fb43d21d700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:07:51.725 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.722+0000 7fb435d9b700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:07:51.725 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.722+0000 7fb43d21d700 10 monclient: get_monmap_and_config success 2023-09-30T02:07:51.726 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.722+0000 7fb435d9b700 1 -- 172.21.15.59:0/433930318 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb430012430 con 0x7fb4380b1500 2023-09-30T02:07:51.726 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.722+0000 7fb435d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:07:51.726 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.722+0000 7fb43d21d700 4 set_mon_vals no callback set 2023-09-30T02:07:51.726 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.722+0000 7fb435d9b700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:07:51.726 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.722+0000 7fb435d9b700 10 monclient: dump: 2023-09-30T02:07:51.726 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:07:51.726 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:51.726 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:51.726 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:51.726 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:07:51.726 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:51.727 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:07:51.727 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:07:51.727 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:07:51.727 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:51.730 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.723+0000 7fb43d21d700 10 monclient: shutdown 2023-09-30T02:07:51.730 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.723+0000 7fb43d21d700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:07:51.730 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.723+0000 7fb43d21d700 1 -- 172.21.15.59:0/433930318 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb4380b1500 msgr2=0x7fb4380a5f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:51.730 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.723+0000 7fb43d21d700 1 --2- 172.21.15.59:0/433930318 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb4380b1500 0x7fb4380a5f40 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb430009e60 tx=0x7fb430009780 comp rx=0 tx=0).stop 2023-09-30T02:07:51.730 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.723+0000 7fb43d21d700 1 -- 172.21.15.59:0/433930318 shutdown_connections 2023-09-30T02:07:51.730 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.723+0000 7fb43d21d700 1 --2- 172.21.15.59:0/433930318 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb4380b1500 0x7fb4380a5f40 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:51.730 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.723+0000 7fb43d21d700 1 -- 172.21.15.59:0/433930318 >> 172.21.15.59:0/433930318 conn(0x7fb4380631f0 msgr2=0x7fb4380635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:07:51.730 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.723+0000 7fb43d21d700 1 -- 172.21.15.59:0/433930318 shutdown_connections 2023-09-30T02:07:51.731 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.723+0000 7fb43d21d700 1 -- 172.21.15.59:0/433930318 wait complete. 2023-09-30T02:07:51.731 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.724+0000 7fb43d21d700 10 monclient: build_initial_monmap 2023-09-30T02:07:51.731 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.724+0000 7fb43d21d700 10 monclient: monmap: 2023-09-30T02:07:51.731 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:07:51.731 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:07:51.731 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:07:51.731 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:07:51.731 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:07:51.731 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:07:51.731 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:07:51.731 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:07:51.731 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:07:51.732 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:51.732 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.724+0000 7fb43d21d700 1 librados: starting msgr at 2023-09-30T02:07:51.732 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.724+0000 7fb43d21d700 1 librados: starting objecter 2023-09-30T02:07:51.732 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.724+0000 7fb43d21d700 1 Processor -- start 2023-09-30T02:07:51.732 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.724+0000 7fb43d21d700 1 -- start start 2023-09-30T02:07:51.732 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.724+0000 7fb43d21d700 1 librados: setting wanted keys 2023-09-30T02:07:51.732 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.724+0000 7fb43d21d700 1 librados: calling monclient init 2023-09-30T02:07:51.732 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.724+0000 7fb43d21d700 10 monclient: init 2023-09-30T02:07:51.732 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.725+0000 7fb43d21d700 10 monclient: _reopen_session rank -1 2023-09-30T02:07:51.732 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.725+0000 7fb43d21d700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T02:07:51.732 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.725+0000 7fb43d21d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4380a5b70 0x7fb4381403a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:51.732 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.725+0000 7fb43d21d700 10 monclient(hunting): picked mon.noname-a con 0x7fb4380a5b70 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:07:51.732 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.725+0000 7fb43d21d700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb4380b1500 0x7fb43813e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:51.733 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.725+0000 7fb43d21d700 10 monclient(hunting): picked mon.noname-c con 0x7fb4380b1500 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:07:51.733 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.725+0000 7fb43d21d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb438142290 0x7fb438136c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:51.733 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.725+0000 7fb43d21d700 10 monclient(hunting): picked mon.noname-b con 0x7fb438142290 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:07:51.733 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.725+0000 7fb43d21d700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:51.733 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.725+0000 7fb43d21d700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb43813e7b0 con 0x7fb4380a5b70 2023-09-30T02:07:51.733 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.725+0000 7fb43d21d700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:51.733 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.725+0000 7fb43d21d700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4381410d0 con 0x7fb438142290 2023-09-30T02:07:51.733 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.725+0000 7fb43d21d700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:51.733 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.725+0000 7fb43d21d700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb43813d510 con 0x7fb4380b1500 2023-09-30T02:07:51.733 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.725+0000 7fb43d21d700 10 monclient(hunting): _renew_subs 2023-09-30T02:07:51.733 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.725+0000 7fb43d21d700 10 monclient(hunting): authenticate will time out at 1917.028499s 2023-09-30T02:07:51.734 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.726+0000 7fb43759e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb438142290 0x7fb438136c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:51.734 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.726+0000 7fb436d9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4380a5b70 0x7fb4381403a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:51.734 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.726+0000 7fb43659c700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb4380b1500 0x7fb43813e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:51.734 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.726+0000 7fb43759e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb438142290 0x7fb438136c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:48172/0 (socket says 172.21.15.59:48172) 2023-09-30T02:07:51.734 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.726+0000 7fb43759e700 1 -- 172.21.15.59:0/3024738014 learned_addr learned my addr 172.21.15.59:0/3024738014 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:07:51.734 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.726+0000 7fb43759e700 10 monclient(hunting): get_auth_request con 0x7fb438142290 auth_method 0 2023-09-30T02:07:51.734 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.726+0000 7fb43759e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:07:51.734 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.726+0000 7fb43759e700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:07:51.734 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.726+0000 7fb43759e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:07:51.734 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.726+0000 7fb43659c700 10 monclient(hunting): get_auth_request con 0x7fb4380b1500 auth_method 0 2023-09-30T02:07:51.735 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.726+0000 7fb43659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:07:51.735 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.726+0000 7fb43659c700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:07:51.735 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.726+0000 7fb43659c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:07:51.735 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.726+0000 7fb436d9d700 10 monclient(hunting): get_auth_request con 0x7fb4380a5b70 auth_method 0 2023-09-30T02:07:51.735 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.726+0000 7fb436d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:07:51.735 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.726+0000 7fb436d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:07:51.735 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.726+0000 7fb436d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:07:51.735 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.727+0000 7fb43759e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:07:51.735 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.727+0000 7fb43759e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:07:51.735 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.727+0000 7fb43759e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:07:51.735 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.727+0000 7fb43659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:07:51.735 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.727+0000 7fb43659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:07:51.736 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.727+0000 7fb43659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:07:51.736 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.727+0000 7fb436d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:07:51.736 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.727+0000 7fb436d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:07:51.736 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.727+0000 7fb436d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:07:51.736 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.731+0000 7fb43759e700 10 monclient(hunting): handle_auth_done global_id 4525 payload 824 2023-09-30T02:07:51.736 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.731+0000 7fb43759e700 1 -- 172.21.15.59:0/3024738014 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb4380b1500 msgr2=0x7fb43813e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:51.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.731+0000 7fb43759e700 1 --2- 172.21.15.59:0/3024738014 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb4380b1500 0x7fb43813e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:51.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.731+0000 7fb43759e700 1 -- 172.21.15.59:0/3024738014 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4380a5b70 msgr2=0x7fb4381403a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:51.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.731+0000 7fb43759e700 1 --2- 172.21.15.59:0/3024738014 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4380a5b70 0x7fb4381403a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:51.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.731+0000 7fb43759e700 10 monclient: _finish_hunting 0 2023-09-30T02:07:51.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.731+0000 7fb43759e700 1 monclient: found mon.noname-b 2023-09-30T02:07:51.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.731+0000 7fb43759e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:07:51.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.731+0000 7fb43759e700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:07:51.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.731+0000 7fb43759e700 1 -- 172.21.15.59:0/3024738014 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb430009a20 con 0x7fb438142290 2023-09-30T02:07:51.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.733+0000 7fb43759e700 1 --2- 172.21.15.59:0/3024738014 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb438142290 0x7fb438136c60 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb4300099f0 tx=0x7fb430019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:07:51.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.733+0000 7fb41effd700 10 client.?.objecter ms_handle_connect 0x7fb438142290 2023-09-30T02:07:51.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.733+0000 7fb41effd700 10 client.?.objecter resend_mon_ops 2023-09-30T02:07:51.739 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.737+0000 7fb41effd700 1 -- 172.21.15.59:0/3024738014 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb430009420 con 0x7fb438142290 2023-09-30T02:07:51.739 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.737+0000 7fb41effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:07:51.739 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.737+0000 7fb41effd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:07:51.739 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.737+0000 7fb41effd700 10 monclient: dump: 2023-09-30T02:07:51.739 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:07:51.739 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:51.739 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:51.739 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:51.739 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:07:51.740 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:51.740 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:07:51.740 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:07:51.740 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:07:51.740 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:51.740 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.737+0000 7fb41effd700 10 monclient: _finish_auth 0 2023-09-30T02:07:51.740 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.737+0000 7fb41effd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:07:51.740 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.737+0000 7fb41effd700 1 -- 172.21.15.59:0/3024738014 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb430027470 con 0x7fb438142290 2023-09-30T02:07:51.740 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.737+0000 7fb41effd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:07:51.740 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.737+0000 7fb41effd700 1 -- 172.21.15.59:0/3024738014 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb430027830 con 0x7fb438142290 2023-09-30T02:07:51.740 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.738+0000 7fb43d21d700 5 monclient: authenticate success, global_id 4525 2023-09-30T02:07:51.741 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.738+0000 7fb43d21d700 10 monclient: _renew_subs 2023-09-30T02:07:51.741 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.738+0000 7fb43d21d700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:07:51.741 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.738+0000 7fb43d21d700 1 -- 172.21.15.59:0/3024738014 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4381371a0 con 0x7fb438142290 2023-09-30T02:07:51.741 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.738+0000 7fb41effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:07:51.741 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.738+0000 7fb41effd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:07:51.741 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.738+0000 7fb41effd700 10 monclient: dump: 2023-09-30T02:07:51.741 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:07:51.741 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:51.741 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:51.741 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:51.741 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:07:51.741 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:51.741 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:07:51.742 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:07:51.742 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:07:51.742 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:51.742 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.738+0000 7fb43559a700 4 set_mon_vals no callback set 2023-09-30T02:07:51.742 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.738+0000 7fb43d21d700 10 client.4525.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:07:51.742 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.738+0000 7fb43d21d700 10 monclient: _renew_subs 2023-09-30T02:07:51.742 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.738+0000 7fb43d21d700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:07:51.742 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.738+0000 7fb43d21d700 1 -- 172.21.15.59:0/3024738014 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb438137690 con 0x7fb438142290 2023-09-30T02:07:51.742 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.738+0000 7fb43d21d700 1 librados: init done 2023-09-30T02:07:51.743 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.739+0000 7fb43d21d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:07:51.743 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.739+0000 7fb43d21d700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:07:51.743 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.739+0000 7fb43d21d700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:07:51.743 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.739+0000 7fb43d21d700 1 -- 172.21.15.59:0/3024738014 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb43813e590 con 0x7fb438142290 2023-09-30T02:07:51.989 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.989+0000 7fb41effd700 1 -- 172.21.15.59:0/3024738014 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fb430024070 con 0x7fb438142290 2023-09-30T02:07:51.989 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.989+0000 7fb41effd700 1 --2- 172.21.15.59:0/3024738014 >> v2:172.21.15.110:6800/115870 conn(0x7fb424068a10 0x7fb42406aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:51.990 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.989+0000 7fb41effd700 1 -- 172.21.15.59:0/3024738014 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fb430086c60 con 0x7fb438142290 2023-09-30T02:07:51.990 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.989+0000 7fb41effd700 10 client.4525.objecter ms_dispatch 0x7fb438001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:07:51.990 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.989+0000 7fb41effd700 3 client.4525.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:07:51.991 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.989+0000 7fb41effd700 3 client.4525.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:07:51.991 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.989+0000 7fb436d9d700 1 --2- 172.21.15.59:0/3024738014 >> v2:172.21.15.110:6800/115870 conn(0x7fb424068a10 0x7fb42406aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:51.991 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.989+0000 7fb41effd700 20 client.4525.objecter dump_active .. 0 homeless 2023-09-30T02:07:51.991 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.989+0000 7fb436d9d700 10 monclient: get_auth_request con 0x7fb424068a10 auth_method 0 2023-09-30T02:07:51.991 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.990+0000 7fb436d9d700 1 --2- 172.21.15.59:0/3024738014 >> v2:172.21.15.110:6800/115870 conn(0x7fb424068a10 0x7fb42406aec0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb428001270 tx=0x7fb428007bf0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:07:51.991 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.990+0000 7fb41effd700 10 client.4525.objecter ms_handle_connect 0x7fb424068a10 2023-09-30T02:07:51.997 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.997+0000 7fb41effd700 1 -- 172.21.15.59:0/3024738014 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb430054580 con 0x7fb438142290 2023-09-30T02:07:51.998 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.997+0000 7fb41effd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:07:51.998 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:51.997+0000 7fb41effd700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:07:52.262 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:52.262+0000 7fb43d21d700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:07:52.262 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:52.262+0000 7fb43d21d700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:07:52.263 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:52.262+0000 7fb43d21d700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:07:52.263 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:52.262+0000 7fb43d21d700 1 -- 172.21.15.59:0/3024738014 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fb43813e920 con 0x7fb438142290 2023-09-30T02:07:52.268 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:52.267+0000 7fb41effd700 1 -- 172.21.15.59:0/3024738014 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fb430057ba0 con 0x7fb438142290 2023-09-30T02:07:52.268 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:52.267+0000 7fb41effd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:07:52.268 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:52.267+0000 7fb41effd700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:07:52.268 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:07:52.268 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:07:52.271 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:52.270+0000 7fb43d21d700 10 librados: watch_flush enter 2023-09-30T02:07:52.271 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:52.270+0000 7fb43d21d700 10 librados: watch_flush exit 2023-09-30T02:07:52.271 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:52.270+0000 7fb43d21d700 20 client.4525.objecter shutdown clearing up homeless session... 2023-09-30T02:07:52.271 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:52.270+0000 7fb43d21d700 10 client.4525.objecter successfully canceled tick 2023-09-30T02:07:52.271 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:52.270+0000 7fb43d21d700 1 -- 172.21.15.59:0/3024738014 >> v2:172.21.15.110:6800/115870 conn(0x7fb424068a10 msgr2=0x7fb42406aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:52.271 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:52.270+0000 7fb43d21d700 1 --2- 172.21.15.59:0/3024738014 >> v2:172.21.15.110:6800/115870 conn(0x7fb424068a10 0x7fb42406aec0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb428001270 tx=0x7fb428007bf0 comp rx=0 tx=0).stop 2023-09-30T02:07:52.271 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:52.270+0000 7fb43d21d700 10 monclient: shutdown 2023-09-30T02:07:52.271 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:52.270+0000 7fb43d21d700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:07:52.272 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:52.270+0000 7fb43d21d700 1 -- 172.21.15.59:0/3024738014 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb438142290 msgr2=0x7fb438136c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:52.272 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:52.270+0000 7fb43d21d700 1 --2- 172.21.15.59:0/3024738014 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb438142290 0x7fb438136c60 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fb4300099f0 tx=0x7fb430019000 comp rx=0 tx=0).stop 2023-09-30T02:07:52.272 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:52.270+0000 7fb43d21d700 1 -- 172.21.15.59:0/3024738014 shutdown_connections 2023-09-30T02:07:52.272 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:52.270+0000 7fb43d21d700 1 --2- 172.21.15.59:0/3024738014 >> v2:172.21.15.110:6800/115870 conn(0x7fb424068a10 0x7fb42406aec0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:52.272 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:52.270+0000 7fb43d21d700 1 --2- 172.21.15.59:0/3024738014 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb438142290 0x7fb438136c60 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:52.272 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:52.270+0000 7fb43d21d700 1 --2- 172.21.15.59:0/3024738014 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4380a5b70 0x7fb4381403a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:52.272 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:52.270+0000 7fb43d21d700 1 --2- 172.21.15.59:0/3024738014 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb4380b1500 0x7fb43813e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:52.272 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:52.270+0000 7fb43d21d700 1 -- 172.21.15.59:0/3024738014 >> 172.21.15.59:0/3024738014 conn(0x7fb4380631f0 msgr2=0x7fb4380635c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:07:52.272 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:52.270+0000 7fb43d21d700 1 -- 172.21.15.59:0/3024738014 shutdown_connections 2023-09-30T02:07:52.272 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:52.270+0000 7fb43d21d700 1 -- 172.21.15.59:0/3024738014 wait complete. 2023-09-30T02:07:52.272 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:52.270+0000 7fb43d21d700 1 librados: shutdown 2023-09-30T02:07:53.676 INFO:tasks.ceph.mon.a.smithi059.stderr:==00:00:03:03.541 116256== Warning: unimplemented fcntl command: 1036 2023-09-30T02:07:53.682 INFO:tasks.ceph.mon.a.smithi059.stderr:==00:00:03:03.547 116256== Warning: unimplemented fcntl command: 1036 2023-09-30T02:07:53.709 INFO:tasks.ceph.mon.a.smithi059.stderr:==00:00:03:03.575 116256== Warning: unimplemented fcntl command: 1036 2023-09-30T02:07:53.732 INFO:tasks.ceph.mon.b.smithi110.stderr:==00:00:03:03.580 115869== Warning: unimplemented fcntl command: 1036 2023-09-30T02:07:53.735 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:03:03.601 116257== Warning: unimplemented fcntl command: 1036 2023-09-30T02:07:53.739 INFO:tasks.ceph.mon.b.smithi110.stderr:==00:00:03:03.587 115869== Warning: unimplemented fcntl command: 1036 2023-09-30T02:07:53.741 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:03:03.607 116257== Warning: unimplemented fcntl command: 1036 2023-09-30T02:07:53.761 INFO:tasks.ceph.mon.b.smithi110.stderr:==00:00:03:03.608 115869== Warning: unimplemented fcntl command: 1036 2023-09-30T02:07:53.765 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:03:03.631 116257== Warning: unimplemented fcntl command: 1036 2023-09-30T02:07:55.285 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:07:55.424 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.423+0000 7f1ecffe6700 10 monclient: get_monmap_and_config 2023-09-30T02:07:55.424 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.423+0000 7f1ecffe6700 10 monclient: build_initial_monmap 2023-09-30T02:07:55.424 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.424+0000 7f1ecffe6700 10 monclient: monmap: 2023-09-30T02:07:55.424 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:07:55.425 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:55.425 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:07:55.425373+0000 2023-09-30T02:07:55.425 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:07:55.425373+0000 2023-09-30T02:07:55.425 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:07:55.425 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:55.425 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:07:55.425 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:07:55.425 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:07:55.425 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:07:55.425 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:07:55.425 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:55.427 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.424+0000 7f1ecffe6700 1 Processor -- start 2023-09-30T02:07:55.427 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.424+0000 7f1ecffe6700 1 -- start start 2023-09-30T02:07:55.427 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.424+0000 7f1ecffe6700 10 monclient: init 2023-09-30T02:07:55.427 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.425+0000 7f1ecffe6700 10 monclient: _reopen_session rank -1 2023-09-30T02:07:55.427 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.425+0000 7f1ecffe6700 10 monclient: _add_conns ranks=[3,0,2,4,1] 2023-09-30T02:07:55.427 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.425+0000 7f1ecffe6700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f1ec80a5b70 0x7f1ec80b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:55.428 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.425+0000 7f1ecffe6700 10 monclient(hunting): picked mon.noname-a con 0x7f1ec80a5b70 addr v2:172.21.15.59:3300/0 2023-09-30T02:07:55.428 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.425+0000 7f1ecffe6700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1ec80a5f40 addr v1:172.21.15.59:6789/0 2023-09-30T02:07:55.428 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.425+0000 7f1ecffe6700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1ec8142290 0x7f1ec813af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:55.428 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.425+0000 7f1ecffe6700 10 monclient(hunting): picked mon.noname-b con 0x7f1ec8142290 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:07:55.428 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.425+0000 7f1ecffe6700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ec80a75a0 con 0x7f1ec80a5f40 2023-09-30T02:07:55.428 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.425+0000 7f1ecffe6700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:55.428 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.425+0000 7f1ecffe6700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ec81426d0 con 0x7f1ec80a5b70 2023-09-30T02:07:55.428 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.425+0000 7f1ecffe6700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:55.428 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.425+0000 7f1ecffe6700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec80a63e0 con 0x7f1ec8142290 2023-09-30T02:07:55.428 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.425+0000 7f1ecffe6700 10 monclient(hunting): _renew_subs 2023-09-30T02:07:55.428 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.425+0000 7f1ecffe6700 10 monclient(hunting): authenticate will time out at 1920.728573s 2023-09-30T02:07:55.429 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.426+0000 7f1ecdd82700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f1ec80a5b70 0x7f1ec80b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:55.429 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.426+0000 7f1ece583700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1ec8142290 0x7f1ec813af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:55.429 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.426+0000 7f1ece583700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1ec8142290 0x7f1ec813af30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:45818/0 (socket says 172.21.15.59:45818) 2023-09-30T02:07:55.429 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.426+0000 7f1ece583700 1 -- 172.21.15.59:0/636406130 learned_addr learned my addr 172.21.15.59:0/636406130 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:07:55.429 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.426+0000 7f1ece583700 10 monclient(hunting): get_auth_request con 0x7f1ec8142290 auth_method 0 2023-09-30T02:07:55.429 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.426+0000 7f1ece583700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:07:55.429 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.426+0000 7f1ece583700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:07:55.429 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.426+0000 7f1ece583700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:07:55.429 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.427+0000 7f1ecdd82700 10 monclient(hunting): get_auth_request con 0x7f1ec80a5b70 auth_method 0 2023-09-30T02:07:55.429 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.427+0000 7f1ecdd82700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:07:55.429 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.427+0000 7f1ecdd82700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:07:55.429 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.427+0000 7f1ecdd82700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:07:55.431 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.427+0000 7f1ece583700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:07:55.431 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.427+0000 7f1ece583700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:07:55.431 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.427+0000 7f1ece583700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:07:55.431 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.428+0000 7f1ecdd82700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:07:55.431 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.428+0000 7f1ecdd82700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:07:55.431 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.428+0000 7f1ecdd82700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:07:55.431 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.430+0000 7f1ece583700 10 monclient(hunting): handle_auth_done global_id 4457 payload 274 2023-09-30T02:07:55.431 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.430+0000 7f1ece583700 1 -- 172.21.15.59:0/636406130 >> v2:172.21.15.59:3300/0 conn(0x7f1ec80a5b70 msgr2=0x7f1ec80b1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:07:55.431 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.430+0000 7f1ece583700 1 --2- 172.21.15.59:0/636406130 >> v2:172.21.15.59:3300/0 conn(0x7f1ec80a5b70 0x7f1ec80b1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:55.431 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.430+0000 7f1ece583700 1 -- 172.21.15.59:0/636406130 >> v1:172.21.15.59:6789/0 conn(0x7f1ec80a5f40 legacy=0x7f1ec8141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:07:55.432 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.430+0000 7f1ece583700 10 monclient: _finish_hunting 0 2023-09-30T02:07:55.432 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.430+0000 7f1ece583700 1 monclient: found mon.noname-b 2023-09-30T02:07:55.432 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.430+0000 7f1ece583700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:07:55.432 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.430+0000 7f1ece583700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.59:3301/0 2023-09-30T02:07:55.432 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.430+0000 7f1ece583700 1 -- 172.21.15.59:0/636406130 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ec80af3f0 con 0x7f1ec8142290 2023-09-30T02:07:55.433 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.433+0000 7f1ece583700 1 --2- 172.21.15.59:0/636406130 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1ec8142290 0x7f1ec813af30 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f1ec0009f20 tx=0x7f1ec0009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f35a2833faa606c9 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:07:55.444 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.437+0000 7f1eccd80700 1 -- 172.21.15.59:0/636406130 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ec000e070 con 0x7f1ec8142290 2023-09-30T02:07:55.445 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.437+0000 7f1eccd80700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:07:55.445 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.437+0000 7f1eccd80700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:07:55.445 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.437+0000 7f1eccd80700 10 monclient: dump: 2023-09-30T02:07:55.445 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:07:55.445 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:55.445 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:55.446 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:55.446 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:07:55.446 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:55.446 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:07:55.446 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:07:55.446 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:07:55.446 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:55.446 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.437+0000 7f1eccd80700 10 monclient: _finish_auth 0 2023-09-30T02:07:55.447 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.437+0000 7f1eccd80700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:07:55.447 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.437+0000 7f1eccd80700 1 -- 172.21.15.59:0/636406130 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ec000b990 con 0x7f1ec8142290 2023-09-30T02:07:55.447 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.437+0000 7f1eccd80700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:07:55.447 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.437+0000 7f1eccd80700 1 -- 172.21.15.59:0/636406130 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ec00092e0 con 0x7f1ec8142290 2023-09-30T02:07:55.447 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.437+0000 7f1ecffe6700 5 monclient: authenticate success, global_id 4457 2023-09-30T02:07:55.447 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.437+0000 7f1ecffe6700 10 monclient: get_monmap_and_config success 2023-09-30T02:07:55.447 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.437+0000 7f1eccd80700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:07:55.447 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.437+0000 7f1ecffe6700 4 set_mon_vals no callback set 2023-09-30T02:07:55.447 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.437+0000 7f1eccd80700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:07:55.448 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.437+0000 7f1eccd80700 10 monclient: dump: 2023-09-30T02:07:55.448 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:07:55.448 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:55.448 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:55.448 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:55.448 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:07:55.448 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:55.448 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:07:55.448 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:07:55.448 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:07:55.448 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:55.449 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.438+0000 7f1ecffe6700 10 monclient: shutdown 2023-09-30T02:07:55.449 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.438+0000 7f1ecffe6700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:07:55.449 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.438+0000 7f1ecffe6700 1 -- 172.21.15.59:0/636406130 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1ec8142290 msgr2=0x7f1ec813af30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:55.449 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.438+0000 7f1ecffe6700 1 --2- 172.21.15.59:0/636406130 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1ec8142290 0x7f1ec813af30 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f1ec0009f20 tx=0x7f1ec0009840 comp rx=0 tx=0).stop 2023-09-30T02:07:55.449 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.438+0000 7f1ecffe6700 1 -- 172.21.15.59:0/636406130 shutdown_connections 2023-09-30T02:07:55.449 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.438+0000 7f1ecffe6700 1 --2- 172.21.15.59:0/636406130 >> v2:172.21.15.59:3300/0 conn(0x7f1ec80a5b70 0x7f1ec80b1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:55.449 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.438+0000 7f1ecffe6700 1 --2- 172.21.15.59:0/636406130 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1ec8142290 0x7f1ec813af30 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:55.449 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.438+0000 7f1ecffe6700 1 -- 172.21.15.59:0/636406130 >> 172.21.15.59:0/636406130 conn(0x7f1ec80631f0 msgr2=0x7f1ec80635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:07:55.449 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.438+0000 7f1ecffe6700 1 -- 172.21.15.59:0/636406130 shutdown_connections 2023-09-30T02:07:55.449 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.438+0000 7f1ecffe6700 1 -- 172.21.15.59:0/636406130 wait complete. 2023-09-30T02:07:55.450 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.439+0000 7f1ecffe6700 10 monclient: build_initial_monmap 2023-09-30T02:07:55.450 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.439+0000 7f1ecffe6700 10 monclient: monmap: 2023-09-30T02:07:55.450 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:07:55.450 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:07:55.450 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:07:55.450 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:07:55.450 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:07:55.450 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:07:55.451 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:07:55.451 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:07:55.451 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:07:55.451 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:55.451 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.440+0000 7f1ecffe6700 1 librados: starting msgr at 2023-09-30T02:07:55.451 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.440+0000 7f1ecffe6700 1 librados: starting objecter 2023-09-30T02:07:55.451 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.440+0000 7f1ecffe6700 1 Processor -- start 2023-09-30T02:07:55.451 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.440+0000 7f1ecffe6700 1 -- start start 2023-09-30T02:07:55.451 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.440+0000 7f1ecffe6700 1 librados: setting wanted keys 2023-09-30T02:07:55.452 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.440+0000 7f1ecffe6700 1 librados: calling monclient init 2023-09-30T02:07:55.452 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.440+0000 7f1ecffe6700 10 monclient: init 2023-09-30T02:07:55.452 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.440+0000 7f1ecffe6700 10 monclient: _reopen_session rank -1 2023-09-30T02:07:55.452 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.440+0000 7f1ecffe6700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T02:07:55.452 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.440+0000 7f1ecffe6700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1ec80a5b70 0x7f1ec8151430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:55.452 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.440+0000 7f1ecffe6700 10 monclient(hunting): picked mon.noname-a con 0x7f1ec80a5b70 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:07:55.452 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.440+0000 7f1ecffe6700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1ec80a5f40 0x7f1ec813c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:55.452 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.440+0000 7f1ecffe6700 10 monclient(hunting): picked mon.noname-b con 0x7f1ec80a5f40 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:07:55.453 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.440+0000 7f1ecffe6700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1ec8142290 0x7f1ec813c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:55.453 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.440+0000 7f1ecffe6700 10 monclient(hunting): picked mon.noname-c con 0x7f1ec8142290 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:07:55.453 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.440+0000 7f1ecffe6700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:55.453 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.440+0000 7f1ecffe6700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec8151ae0 con 0x7f1ec80a5b70 2023-09-30T02:07:55.453 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.440+0000 7f1ecffe6700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:55.454 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.440+0000 7f1ecffe6700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec8151970 con 0x7f1ec80a5f40 2023-09-30T02:07:55.454 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.440+0000 7f1ecffe6700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:55.454 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.440+0000 7f1ecffe6700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec80a9cd0 con 0x7f1ec8142290 2023-09-30T02:07:55.454 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.440+0000 7f1ecffe6700 10 monclient(hunting): _renew_subs 2023-09-30T02:07:55.454 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.441+0000 7f1ecffe6700 10 monclient(hunting): authenticate will time out at 1920.744007s 2023-09-30T02:07:55.454 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.441+0000 7f1ecdd82700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1ec80a5b70 0x7f1ec8151430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:55.455 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.441+0000 7f1ece583700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1ec8142290 0x7f1ec813c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:55.455 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.441+0000 7f1ecd581700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1ec80a5f40 0x7f1ec813c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:55.455 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.442+0000 7f1ece583700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1ec8142290 0x7f1ec813c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:45832/0 (socket says 172.21.15.59:45832) 2023-09-30T02:07:55.455 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.442+0000 7f1ece583700 1 -- 172.21.15.59:0/330644021 learned_addr learned my addr 172.21.15.59:0/330644021 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:07:55.455 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.442+0000 7f1ece583700 10 monclient(hunting): get_auth_request con 0x7f1ec8142290 auth_method 0 2023-09-30T02:07:55.456 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.442+0000 7f1ecdd82700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1ec80a5b70 0x7f1ec8151430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:57364/0 (socket says 172.21.15.59:57364) 2023-09-30T02:07:55.456 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.442+0000 7f1ece583700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:07:55.456 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.442+0000 7f1ece583700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:07:55.456 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.442+0000 7f1ece583700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:07:55.456 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.442+0000 7f1ecdd82700 10 monclient(hunting): get_auth_request con 0x7f1ec80a5b70 auth_method 0 2023-09-30T02:07:55.456 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.442+0000 7f1ecdd82700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:07:55.457 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.442+0000 7f1ecdd82700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:07:55.457 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.442+0000 7f1ecdd82700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:07:55.457 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.442+0000 7f1ecd581700 10 monclient(hunting): get_auth_request con 0x7f1ec80a5f40 auth_method 0 2023-09-30T02:07:55.457 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.442+0000 7f1ecd581700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:07:55.457 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.442+0000 7f1ecd581700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:07:55.457 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.442+0000 7f1ecd581700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:07:55.458 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.442+0000 7f1ecdd82700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:07:55.458 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.442+0000 7f1ecdd82700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:07:55.458 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.442+0000 7f1ecdd82700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:07:55.458 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.442+0000 7f1ece583700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:07:55.458 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.442+0000 7f1ece583700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:07:55.458 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.442+0000 7f1ece583700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:07:55.459 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.443+0000 7f1ecd581700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:07:55.459 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.443+0000 7f1ecd581700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:07:55.459 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.443+0000 7f1ecd581700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:07:55.462 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.446+0000 7f1ecdd82700 10 monclient(hunting): handle_auth_done global_id 4518 payload 824 2023-09-30T02:07:55.462 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.446+0000 7f1ecdd82700 1 -- 172.21.15.59:0/330644021 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1ec8142290 msgr2=0x7f1ec813c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:55.462 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.446+0000 7f1ecdd82700 1 --2- 172.21.15.59:0/330644021 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1ec8142290 0x7f1ec813c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:55.462 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.446+0000 7f1ecdd82700 1 -- 172.21.15.59:0/330644021 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1ec80a5f40 msgr2=0x7f1ec813c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:55.462 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.446+0000 7f1ecdd82700 1 --2- 172.21.15.59:0/330644021 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1ec80a5f40 0x7f1ec813c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:55.462 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.446+0000 7f1ecdd82700 10 monclient: _finish_hunting 0 2023-09-30T02:07:55.462 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.446+0000 7f1ecdd82700 1 monclient: found mon.noname-a 2023-09-30T02:07:55.462 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.446+0000 7f1ecdd82700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:07:55.462 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.446+0000 7f1ecdd82700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.59:3300/0 2023-09-30T02:07:55.463 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.446+0000 7f1ecdd82700 1 -- 172.21.15.59:0/330644021 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ec0009ae0 con 0x7f1ec80a5b70 2023-09-30T02:07:55.463 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.446+0000 7f1ece583700 1 --2- 172.21.15.59:0/330644021 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1ec8142290 0x7f1ec813c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T02:07:55.463 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.449+0000 7f1ecdd82700 1 --2- 172.21.15.59:0/330644021 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1ec80a5b70 0x7f1ec8151430 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f1ec400e9d0 tx=0x7f1ec40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:07:55.463 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.449+0000 7f1ebdffb700 10 client.?.objecter ms_handle_connect 0x7f1ec80a5b70 2023-09-30T02:07:55.463 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.449+0000 7f1ebdffb700 10 client.?.objecter resend_mon_ops 2023-09-30T02:07:55.463 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.454+0000 7f1ebdffb700 1 -- 172.21.15.59:0/330644021 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ec4013680 con 0x7f1ec80a5b70 2023-09-30T02:07:55.463 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.454+0000 7f1ebdffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:07:55.463 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.454+0000 7f1ebdffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T02:07:55.464 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.454+0000 7f1ebdffb700 10 monclient: dump: 2023-09-30T02:07:55.464 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:07:55.464 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:55.464 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:55.464 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:55.464 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:07:55.464 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:55.465 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:07:55.465 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:07:55.465 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:07:55.465 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:55.465 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.454+0000 7f1ebdffb700 10 monclient: _finish_auth 0 2023-09-30T02:07:55.465 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.454+0000 7f1ebdffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:07:55.465 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.454+0000 7f1ebdffb700 1 -- 172.21.15.59:0/330644021 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ec4013800 con 0x7f1ec80a5b70 2023-09-30T02:07:55.466 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.454+0000 7f1ecffe6700 5 monclient: authenticate success, global_id 4518 2023-09-30T02:07:55.466 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.454+0000 7f1ebdffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:07:55.466 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.454+0000 7f1ebdffb700 1 -- 172.21.15.59:0/330644021 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ec4013bc0 con 0x7f1ec80a5b70 2023-09-30T02:07:55.466 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.454+0000 7f1ecffe6700 10 monclient: _renew_subs 2023-09-30T02:07:55.466 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.454+0000 7f1ecffe6700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:07:55.466 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.454+0000 7f1ecffe6700 1 -- 172.21.15.59:0/330644021 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ec813e320 con 0x7f1ec80a5b70 2023-09-30T02:07:55.467 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.454+0000 7f1ebffff700 4 set_mon_vals no callback set 2023-09-30T02:07:55.467 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.454+0000 7f1ebdffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:07:55.467 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.454+0000 7f1ebdffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T02:07:55.467 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.454+0000 7f1ebdffb700 10 monclient: dump: 2023-09-30T02:07:55.467 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:07:55.467 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:55.468 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:55.468 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:55.468 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:07:55.468 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:55.468 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:07:55.469 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:07:55.469 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:07:55.469 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:55.469 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.454+0000 7f1ecffe6700 10 client.4518.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:07:55.469 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.454+0000 7f1ecffe6700 10 monclient: _renew_subs 2023-09-30T02:07:55.469 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.454+0000 7f1ecffe6700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:07:55.470 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.454+0000 7f1ecffe6700 1 -- 172.21.15.59:0/330644021 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ec8151f30 con 0x7f1ec80a5b70 2023-09-30T02:07:55.470 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.454+0000 7f1ecffe6700 1 librados: init done 2023-09-30T02:07:55.470 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.455+0000 7f1ecffe6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:07:55.470 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.455+0000 7f1ecffe6700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:07:55.470 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.455+0000 7f1ecffe6700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:07:55.471 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.455+0000 7f1ecffe6700 1 -- 172.21.15.59:0/330644021 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ec8151d80 con 0x7f1ec80a5b70 2023-09-30T02:07:55.759 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.758+0000 7f1ebdffb700 1 -- 172.21.15.59:0/330644021 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f1ec400c490 con 0x7f1ec80a5b70 2023-09-30T02:07:55.759 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.758+0000 7f1ebdffb700 1 --2- 172.21.15.59:0/330644021 >> v2:172.21.15.110:6800/115870 conn(0x7f1eac068910 0x7f1eac06adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:55.759 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.758+0000 7f1ebdffb700 1 -- 172.21.15.59:0/330644021 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f1ec40845d0 con 0x7f1ec80a5b70 2023-09-30T02:07:55.759 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.758+0000 7f1ebdffb700 10 client.4518.objecter ms_dispatch 0x7f1ec8001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:07:55.759 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.758+0000 7f1ebdffb700 3 client.4518.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:07:55.760 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.758+0000 7f1ebdffb700 3 client.4518.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:07:55.760 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.759+0000 7f1ebdffb700 20 client.4518.objecter dump_active .. 0 homeless 2023-09-30T02:07:55.760 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.759+0000 7f1ebdffb700 1 -- 172.21.15.59:0/330644021 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1ec4084a60 con 0x7f1ec80a5b70 2023-09-30T02:07:55.760 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.759+0000 7f1ebdffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:07:55.760 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.759+0000 7f1ebdffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:07:55.760 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.759+0000 7f1ecd581700 1 --2- 172.21.15.59:0/330644021 >> v2:172.21.15.110:6800/115870 conn(0x7f1eac068910 0x7f1eac06adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:55.760 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.759+0000 7f1ecd581700 10 monclient: get_auth_request con 0x7f1eac068910 auth_method 0 2023-09-30T02:07:55.761 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.759+0000 7f1ecd581700 1 --2- 172.21.15.59:0/330644021 >> v2:172.21.15.110:6800/115870 conn(0x7f1eac068910 0x7f1eac06adc0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f1ec813de00 tx=0x7f1eb8007ea0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:07:55.761 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:55.759+0000 7f1ebdffb700 10 client.4518.objecter ms_handle_connect 0x7f1eac068910 2023-09-30T02:07:56.055 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:56.054+0000 7f1ecffe6700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:07:56.055 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:56.054+0000 7f1ecffe6700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:07:56.055 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:56.054+0000 7f1ecffe6700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:07:56.055 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:56.054+0000 7f1ecffe6700 1 -- 172.21.15.59:0/330644021 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f1ec8064a00 con 0x7f1ec80a5b70 2023-09-30T02:07:56.060 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:56.060+0000 7f1ebdffb700 1 -- 172.21.15.59:0/330644021 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f1ec4017100 con 0x7f1ec80a5b70 2023-09-30T02:07:56.061 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:56.060+0000 7f1ebdffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:07:56.061 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:56.060+0000 7f1ebdffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:07:56.061 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:07:56.061 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:07:56.063 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:56.062+0000 7f1ecffe6700 10 librados: watch_flush enter 2023-09-30T02:07:56.064 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:56.062+0000 7f1ecffe6700 10 librados: watch_flush exit 2023-09-30T02:07:56.064 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:56.062+0000 7f1ecffe6700 20 client.4518.objecter shutdown clearing up homeless session... 2023-09-30T02:07:56.064 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:56.062+0000 7f1ecffe6700 10 client.4518.objecter successfully canceled tick 2023-09-30T02:07:56.064 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:56.062+0000 7f1ecffe6700 1 -- 172.21.15.59:0/330644021 >> v2:172.21.15.110:6800/115870 conn(0x7f1eac068910 msgr2=0x7f1eac06adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:56.064 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:56.063+0000 7f1ecffe6700 1 --2- 172.21.15.59:0/330644021 >> v2:172.21.15.110:6800/115870 conn(0x7f1eac068910 0x7f1eac06adc0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f1ec813de00 tx=0x7f1eb8007ea0 comp rx=0 tx=0).stop 2023-09-30T02:07:56.064 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:56.063+0000 7f1ecffe6700 10 monclient: shutdown 2023-09-30T02:07:56.064 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:56.063+0000 7f1ecffe6700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:07:56.064 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:56.063+0000 7f1ecffe6700 1 -- 172.21.15.59:0/330644021 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1ec80a5b70 msgr2=0x7f1ec8151430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:56.064 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:56.063+0000 7f1ecffe6700 1 --2- 172.21.15.59:0/330644021 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1ec80a5b70 0x7f1ec8151430 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f1ec400e9d0 tx=0x7f1ec40091a0 comp rx=0 tx=0).stop 2023-09-30T02:07:56.064 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:56.063+0000 7f1ecffe6700 1 -- 172.21.15.59:0/330644021 shutdown_connections 2023-09-30T02:07:56.064 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:56.063+0000 7f1ecffe6700 1 --2- 172.21.15.59:0/330644021 >> v2:172.21.15.110:6800/115870 conn(0x7f1eac068910 0x7f1eac06adc0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:56.064 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:56.063+0000 7f1ecffe6700 1 --2- 172.21.15.59:0/330644021 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1ec8142290 0x7f1ec813c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:56.065 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:56.063+0000 7f1ecffe6700 1 --2- 172.21.15.59:0/330644021 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1ec80a5b70 0x7f1ec8151430 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:56.065 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:56.063+0000 7f1ecffe6700 1 --2- 172.21.15.59:0/330644021 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1ec80a5f40 0x7f1ec813c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:56.065 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:56.063+0000 7f1ecffe6700 1 -- 172.21.15.59:0/330644021 >> 172.21.15.59:0/330644021 conn(0x7f1ec80631f0 msgr2=0x7f1ec80b3220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:07:56.065 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:56.063+0000 7f1ecffe6700 1 -- 172.21.15.59:0/330644021 shutdown_connections 2023-09-30T02:07:56.065 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:56.063+0000 7f1ecffe6700 1 -- 172.21.15.59:0/330644021 wait complete. 2023-09-30T02:07:56.065 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:56.063+0000 7f1ecffe6700 1 librados: shutdown 2023-09-30T02:07:59.078 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:07:59.217 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.216+0000 7f0b36507700 10 monclient: get_monmap_and_config 2023-09-30T02:07:59.217 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.216+0000 7f0b36507700 10 monclient: build_initial_monmap 2023-09-30T02:07:59.217 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.216+0000 7f0b36507700 10 monclient: monmap: 2023-09-30T02:07:59.217 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:07:59.217 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:59.217 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:07:59.218334+0000 2023-09-30T02:07:59.218 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:07:59.218334+0000 2023-09-30T02:07:59.218 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:07:59.218 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:59.218 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:07:59.218 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:07:59.218 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:07:59.218 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:07:59.218 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:07:59.218 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:59.220 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.217+0000 7f0b36507700 1 Processor -- start 2023-09-30T02:07:59.220 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.217+0000 7f0b36507700 1 -- start start 2023-09-30T02:07:59.220 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.217+0000 7f0b36507700 10 monclient: init 2023-09-30T02:07:59.220 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.218+0000 7f0b36507700 10 monclient: _reopen_session rank -1 2023-09-30T02:07:59.220 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.218+0000 7f0b36507700 10 monclient: _add_conns ranks=[0,1,2,3,4] 2023-09-30T02:07:59.221 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.218+0000 7f0b36507700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0b300a5b70 addr v1:172.21.15.59:6789/0 2023-09-30T02:07:59.221 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.218+0000 7f0b36507700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0b30142290 addr v1:172.21.15.110:6789/0 2023-09-30T02:07:59.221 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.218+0000 7f0b36507700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0b300b1500 0x7f0b300a5f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:59.221 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.218+0000 7f0b36507700 10 monclient(hunting): picked mon.noname-b con 0x7f0b300b1500 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:07:59.221 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.218+0000 7f0b36507700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b300a75a0 con 0x7f0b300a5b70 2023-09-30T02:07:59.221 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.218+0000 7f0b36507700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b300af3f0 con 0x7f0b30142290 2023-09-30T02:07:59.221 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.218+0000 7f0b36507700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:59.221 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.218+0000 7f0b36507700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b300b1940 con 0x7f0b300b1500 2023-09-30T02:07:59.221 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.218+0000 7f0b36507700 10 monclient(hunting): _renew_subs 2023-09-30T02:07:59.221 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.218+0000 7f0b36507700 10 monclient(hunting): authenticate will time out at 1924.521603s 2023-09-30T02:07:59.221 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.219+0000 7f0b34aa4700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0b300b1500 0x7f0b300a5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:59.221 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.219+0000 7f0b2ffff700 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f0b300a5b70 0x7f0b30141b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:38130/0 (socket says 172.21.15.59:38130) 2023-09-30T02:07:59.222 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.219+0000 7f0b2ffff700 1 -- 172.21.15.59:0/471618754 learned_addr learned my addr 172.21.15.59:0/471618754 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:07:59.222 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.219+0000 7f0b34aa4700 10 monclient(hunting): get_auth_request con 0x7f0b300b1500 auth_method 0 2023-09-30T02:07:59.222 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.219+0000 7f0b34aa4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:07:59.222 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.219+0000 7f0b34aa4700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:07:59.222 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.219+0000 7f0b34aa4700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:07:59.222 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.220+0000 7f0b34aa4700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:07:59.222 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.220+0000 7f0b34aa4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:07:59.222 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.220+0000 7f0b34aa4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:07:59.223 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.222+0000 7f0b2f7fe700 1 -- 172.21.15.59:0/471618754 >> v1:172.21.15.110:6789/0 conn(0x7f0b30142290 legacy=0x7f0b3013af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2023-09-30T02:07:59.223 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.222+0000 7f0b2f7fe700 1 -- 172.21.15.59:0/471618754 >> v1:172.21.15.110:6789/0 conn(0x7f0b30142290 legacy=0x7f0b3013af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-09-30T02:07:59.223 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.222+0000 7f0b2f7fe700 1 --1- 172.21.15.59:0/471618754 >> v1:172.21.15.110:6789/0 conn(0x7f0b30142290 0x7f0b3013af30 :-1 s=OPENED pgs=143 cs=1 l=1).handle_message read tag failed 2023-09-30T02:07:59.224 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.222+0000 7f0b2f7fe700 1 --1- 172.21.15.59:0/471618754 >> v1:172.21.15.110:6789/0 conn(0x7f0b30142290 0x7f0b3013af30 :-1 s=OPENED pgs=143 cs=1 l=1).fault on lossy channel, failing 2023-09-30T02:07:59.224 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.222+0000 7f0b2effd700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.110:6789/0 2023-09-30T02:07:59.224 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.222+0000 7f0b34aa4700 10 monclient(hunting): handle_auth_done global_id 4463 payload 274 2023-09-30T02:07:59.224 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.222+0000 7f0b34aa4700 1 -- 172.21.15.59:0/471618754 >> v1:172.21.15.110:6789/0 conn(0x7f0b30142290 legacy=0x7f0b3013af30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-09-30T02:07:59.224 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.222+0000 7f0b34aa4700 1 -- 172.21.15.59:0/471618754 >> v1:172.21.15.59:6789/0 conn(0x7f0b300a5b70 legacy=0x7f0b30141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:59.224 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.222+0000 7f0b34aa4700 10 monclient: _finish_hunting 0 2023-09-30T02:07:59.224 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.222+0000 7f0b34aa4700 1 monclient: found mon.noname-b 2023-09-30T02:07:59.224 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.222+0000 7f0b34aa4700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:07:59.224 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.222+0000 7f0b34aa4700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.59:3301/0 2023-09-30T02:07:59.224 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.222+0000 7f0b34aa4700 1 -- 172.21.15.59:0/471618754 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b3013b640 con 0x7f0b300b1500 2023-09-30T02:07:59.225 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.224+0000 7f0b34aa4700 1 --2- 172.21.15.59:0/471618754 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0b300b1500 0x7f0b300a5f40 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0b28009de0 tx=0x7f0b28009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=108200154b9f594d server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:07:59.230 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.229+0000 7f0b2effd700 1 -- 172.21.15.59:0/471618754 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b2800e070 con 0x7f0b300b1500 2023-09-30T02:07:59.230 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.229+0000 7f0b2effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:07:59.230 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.229+0000 7f0b2effd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:07:59.230 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.229+0000 7f0b2effd700 10 monclient: dump: 2023-09-30T02:07:59.230 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:07:59.230 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:59.230 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:59.230 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:59.231 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:07:59.231 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:59.231 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:07:59.231 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:07:59.231 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:07:59.231 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:59.231 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.229+0000 7f0b2effd700 10 monclient: _finish_auth 0 2023-09-30T02:07:59.231 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.229+0000 7f0b2effd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:07:59.231 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.229+0000 7f0b2effd700 1 -- 172.21.15.59:0/471618754 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b2800bc30 con 0x7f0b300b1500 2023-09-30T02:07:59.231 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.229+0000 7f0b36507700 5 monclient: authenticate success, global_id 4463 2023-09-30T02:07:59.231 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.229+0000 7f0b36507700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:07:59.231 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.229+0000 7f0b2effd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:07:59.232 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.229+0000 7f0b36507700 10 monclient: get_monmap_and_config success 2023-09-30T02:07:59.232 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.229+0000 7f0b2effd700 1 -- 172.21.15.59:0/471618754 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b28012430 con 0x7f0b300b1500 2023-09-30T02:07:59.232 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.229+0000 7f0b36507700 4 set_mon_vals no callback set 2023-09-30T02:07:59.232 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.229+0000 7f0b2effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:07:59.232 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.229+0000 7f0b2effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:07:59.232 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.229+0000 7f0b2effd700 10 monclient: dump: 2023-09-30T02:07:59.232 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:07:59.232 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:59.232 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:59.232 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:59.232 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:07:59.232 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:59.233 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:07:59.233 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:07:59.233 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:07:59.233 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:59.233 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.229+0000 7f0b36507700 10 monclient: shutdown 2023-09-30T02:07:59.233 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.229+0000 7f0b36507700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:07:59.233 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.229+0000 7f0b36507700 1 -- 172.21.15.59:0/471618754 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0b300b1500 msgr2=0x7f0b300a5f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:59.233 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.229+0000 7f0b36507700 1 --2- 172.21.15.59:0/471618754 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0b300b1500 0x7f0b300a5f40 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0b28009de0 tx=0x7f0b28009780 comp rx=0 tx=0).stop 2023-09-30T02:07:59.233 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.229+0000 7f0b36507700 1 -- 172.21.15.59:0/471618754 shutdown_connections 2023-09-30T02:07:59.233 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.229+0000 7f0b36507700 1 --2- 172.21.15.59:0/471618754 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0b300b1500 0x7f0b300a5f40 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:59.233 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.229+0000 7f0b36507700 1 -- 172.21.15.59:0/471618754 >> 172.21.15.59:0/471618754 conn(0x7f0b300631f0 msgr2=0x7f0b300635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:07:59.233 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.229+0000 7f0b36507700 1 -- 172.21.15.59:0/471618754 shutdown_connections 2023-09-30T02:07:59.234 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.229+0000 7f0b36507700 1 -- 172.21.15.59:0/471618754 wait complete. 2023-09-30T02:07:59.236 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.230+0000 7f0b36507700 10 monclient: build_initial_monmap 2023-09-30T02:07:59.236 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.230+0000 7f0b36507700 10 monclient: monmap: 2023-09-30T02:07:59.236 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:07:59.236 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:07:59.237 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:07:59.237 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:07:59.237 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:07:59.237 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:07:59.237 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:07:59.237 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:07:59.237 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:07:59.237 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:59.237 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.231+0000 7f0b36507700 1 librados: starting msgr at 2023-09-30T02:07:59.237 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.231+0000 7f0b36507700 1 librados: starting objecter 2023-09-30T02:07:59.237 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.231+0000 7f0b36507700 1 Processor -- start 2023-09-30T02:07:59.238 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.231+0000 7f0b36507700 1 -- start start 2023-09-30T02:07:59.238 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.231+0000 7f0b36507700 1 librados: setting wanted keys 2023-09-30T02:07:59.238 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.231+0000 7f0b36507700 1 librados: calling monclient init 2023-09-30T02:07:59.238 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.231+0000 7f0b36507700 10 monclient: init 2023-09-30T02:07:59.238 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.232+0000 7f0b36507700 10 monclient: _reopen_session rank -1 2023-09-30T02:07:59.238 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.232+0000 7f0b36507700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T02:07:59.238 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.232+0000 7f0b36507700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0b300a5b70 0x7f0b30144dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:59.238 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.232+0000 7f0b36507700 10 monclient(hunting): picked mon.noname-c con 0x7f0b300a5b70 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:07:59.238 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.232+0000 7f0b36507700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0b300b1500 0x7f0b3013ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:59.238 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.232+0000 7f0b36507700 10 monclient(hunting): picked mon.noname-b con 0x7f0b300b1500 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:07:59.238 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.232+0000 7f0b36507700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b30142290 0x7f0b30136bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:59.239 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.232+0000 7f0b36507700 10 monclient(hunting): picked mon.noname-a con 0x7f0b30142290 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:07:59.239 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.232+0000 7f0b36507700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:59.239 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.232+0000 7f0b36507700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b30145be0 con 0x7f0b30142290 2023-09-30T02:07:59.239 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.232+0000 7f0b36507700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:59.239 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.232+0000 7f0b36507700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b3013e020 con 0x7f0b300b1500 2023-09-30T02:07:59.239 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.232+0000 7f0b36507700 10 monclient(hunting): start opening mon connection 2023-09-30T02:07:59.239 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.232+0000 7f0b36507700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b3013da00 con 0x7f0b300a5b70 2023-09-30T02:07:59.239 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.232+0000 7f0b36507700 10 monclient(hunting): _renew_subs 2023-09-30T02:07:59.239 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.232+0000 7f0b36507700 10 monclient(hunting): authenticate will time out at 1924.535641s 2023-09-30T02:07:59.239 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.233+0000 7f0b2ffff700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0b300a5b70 0x7f0b30144dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:59.239 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.233+0000 7f0b34aa4700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b30142290 0x7f0b30136bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:59.239 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.233+0000 7f0b2f7fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0b300b1500 0x7f0b3013ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:59.239 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.233+0000 7f0b2ffff700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0b300a5b70 0x7f0b30144dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:36984/0 (socket says 172.21.15.59:36984) 2023-09-30T02:07:59.240 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.233+0000 7f0b2ffff700 1 -- 172.21.15.59:0/2176124423 learned_addr learned my addr 172.21.15.59:0/2176124423 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:07:59.240 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.233+0000 7f0b2ffff700 10 monclient(hunting): get_auth_request con 0x7f0b300a5b70 auth_method 0 2023-09-30T02:07:59.240 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.233+0000 7f0b2ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:07:59.240 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.233+0000 7f0b2ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:07:59.240 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.233+0000 7f0b2ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:07:59.240 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.233+0000 7f0b34aa4700 10 monclient(hunting): get_auth_request con 0x7f0b30142290 auth_method 0 2023-09-30T02:07:59.240 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.233+0000 7f0b34aa4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:07:59.240 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.233+0000 7f0b34aa4700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:07:59.240 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.233+0000 7f0b34aa4700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:07:59.240 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.233+0000 7f0b2f7fe700 10 monclient(hunting): get_auth_request con 0x7f0b300b1500 auth_method 0 2023-09-30T02:07:59.240 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.233+0000 7f0b2f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:07:59.240 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.233+0000 7f0b2f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:07:59.241 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.233+0000 7f0b2f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:07:59.241 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.234+0000 7f0b2ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:07:59.241 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.234+0000 7f0b2ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:07:59.241 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.234+0000 7f0b2ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:07:59.241 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.234+0000 7f0b2f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:07:59.241 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.234+0000 7f0b2f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:07:59.241 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.234+0000 7f0b2f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:07:59.241 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.234+0000 7f0b34aa4700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:07:59.241 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.234+0000 7f0b34aa4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:07:59.241 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.234+0000 7f0b34aa4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:07:59.242 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.237+0000 7f0b2ffff700 10 monclient(hunting): handle_auth_done global_id 4466 payload 824 2023-09-30T02:07:59.242 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.238+0000 7f0b2ffff700 1 -- 172.21.15.59:0/2176124423 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0b300b1500 msgr2=0x7f0b3013ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:59.242 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.238+0000 7f0b2ffff700 1 --2- 172.21.15.59:0/2176124423 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0b300b1500 0x7f0b3013ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:59.242 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.238+0000 7f0b2ffff700 1 -- 172.21.15.59:0/2176124423 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b30142290 msgr2=0x7f0b30136bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:59.243 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.238+0000 7f0b2ffff700 1 --2- 172.21.15.59:0/2176124423 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b30142290 0x7f0b30136bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:59.243 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.238+0000 7f0b2ffff700 10 monclient: _finish_hunting 0 2023-09-30T02:07:59.243 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.238+0000 7f0b2ffff700 1 monclient: found mon.noname-c 2023-09-30T02:07:59.243 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.238+0000 7f0b2ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:07:59.243 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.238+0000 7f0b2ffff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.59:3301/0 2023-09-30T02:07:59.243 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.238+0000 7f0b2ffff700 1 -- 172.21.15.59:0/2176124423 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b28009a20 con 0x7f0b300a5b70 2023-09-30T02:07:59.243 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.240+0000 7f0b2ffff700 1 --2- 172.21.15.59:0/2176124423 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0b300a5b70 0x7f0b30144dc0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f0b2000c7c0 tx=0x7f0b20007f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:07:59.243 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.240+0000 7f0b17fff700 10 client.?.objecter ms_handle_connect 0x7f0b300a5b70 2023-09-30T02:07:59.243 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.240+0000 7f0b17fff700 10 client.?.objecter resend_mon_ops 2023-09-30T02:07:59.245 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.244+0000 7f0b17fff700 1 -- 172.21.15.59:0/2176124423 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b20005b30 con 0x7f0b300a5b70 2023-09-30T02:07:59.245 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.244+0000 7f0b17fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:07:59.245 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.244+0000 7f0b17fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:07:59.245 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.244+0000 7f0b17fff700 10 monclient: dump: 2023-09-30T02:07:59.245 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:07:59.245 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:59.245 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:59.245 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:59.246 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:07:59.246 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:59.246 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:07:59.246 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:07:59.246 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:07:59.246 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:59.246 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.244+0000 7f0b17fff700 10 monclient: _finish_auth 0 2023-09-30T02:07:59.246 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.244+0000 7f0b17fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:07:59.246 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.244+0000 7f0b17fff700 1 -- 172.21.15.59:0/2176124423 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b20005c90 con 0x7f0b300a5b70 2023-09-30T02:07:59.246 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.244+0000 7f0b17fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:07:59.246 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.244+0000 7f0b17fff700 1 -- 172.21.15.59:0/2176124423 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b20004a80 con 0x7f0b300a5b70 2023-09-30T02:07:59.247 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.244+0000 7f0b17fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:07:59.247 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.244+0000 7f0b17fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:07:59.247 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.244+0000 7f0b17fff700 10 monclient: dump: 2023-09-30T02:07:59.247 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:07:59.247 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:07:59.247 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:59.247 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:07:59.247 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:07:59.247 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:07:59.247 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:07:59.247 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:07:59.247 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:07:59.247 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:07:59.248 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.244+0000 7f0b36507700 5 monclient: authenticate success, global_id 4466 2023-09-30T02:07:59.248 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.244+0000 7f0b36507700 10 monclient: _renew_subs 2023-09-30T02:07:59.248 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.244+0000 7f0b36507700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:07:59.248 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.244+0000 7f0b36507700 1 -- 172.21.15.59:0/2176124423 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b3013dc20 con 0x7f0b300a5b70 2023-09-30T02:07:59.248 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.244+0000 7f0b2e7fc700 4 set_mon_vals no callback set 2023-09-30T02:07:59.248 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.244+0000 7f0b36507700 10 client.4466.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:07:59.248 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.244+0000 7f0b36507700 10 monclient: _renew_subs 2023-09-30T02:07:59.248 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.244+0000 7f0b36507700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:07:59.248 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.244+0000 7f0b36507700 1 -- 172.21.15.59:0/2176124423 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b30137350 con 0x7f0b300a5b70 2023-09-30T02:07:59.248 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.244+0000 7f0b36507700 1 librados: init done 2023-09-30T02:07:59.249 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.245+0000 7f0b36507700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:07:59.249 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.245+0000 7f0b36507700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:07:59.249 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.245+0000 7f0b36507700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:07:59.249 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.245+0000 7f0b36507700 1 -- 172.21.15.59:0/2176124423 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b3013e300 con 0x7f0b300a5b70 2023-09-30T02:07:59.545 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.544+0000 7f0b17fff700 1 -- 172.21.15.59:0/2176124423 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f0b200070d0 con 0x7f0b300a5b70 2023-09-30T02:07:59.546 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.545+0000 7f0b17fff700 1 --2- 172.21.15.59:0/2176124423 >> v2:172.21.15.110:6800/115870 conn(0x7f0b1c068910 0x7f0b1c06adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:07:59.546 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.545+0000 7f0b17fff700 1 -- 172.21.15.59:0/2176124423 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f0b20083e60 con 0x7f0b300a5b70 2023-09-30T02:07:59.546 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.545+0000 7f0b17fff700 10 client.4466.objecter ms_dispatch 0x7f0b30001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:07:59.546 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.545+0000 7f0b17fff700 3 client.4466.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:07:59.546 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.545+0000 7f0b17fff700 3 client.4466.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:07:59.547 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.545+0000 7f0b17fff700 20 client.4466.objecter dump_active .. 0 homeless 2023-09-30T02:07:59.547 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.545+0000 7f0b17fff700 1 -- 172.21.15.59:0/2176124423 <== mon.2 v2:172.21.15.59:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0b200842f0 con 0x7f0b300a5b70 2023-09-30T02:07:59.547 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.545+0000 7f0b17fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:07:59.547 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.545+0000 7f0b17fff700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:07:59.547 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.545+0000 7f0b2f7fe700 1 --2- 172.21.15.59:0/2176124423 >> v2:172.21.15.110:6800/115870 conn(0x7f0b1c068910 0x7f0b1c06adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:07:59.547 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.545+0000 7f0b2f7fe700 10 monclient: get_auth_request con 0x7f0b1c068910 auth_method 0 2023-09-30T02:07:59.547 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.546+0000 7f0b17fff700 10 client.4466.objecter ms_handle_connect 0x7f0b1c068910 2023-09-30T02:07:59.547 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.546+0000 7f0b2f7fe700 1 --2- 172.21.15.59:0/2176124423 >> v2:172.21.15.110:6800/115870 conn(0x7f0b1c068910 0x7f0b1c06adc0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0b300a8280 tx=0x7f0b24009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:07:59.796 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.796+0000 7f0b36507700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:07:59.796 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.796+0000 7f0b36507700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:07:59.796 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.796+0000 7f0b36507700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:07:59.797 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.796+0000 7f0b36507700 1 -- 172.21.15.59:0/2176124423 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f0b30064a00 con 0x7f0b300a5b70 2023-09-30T02:07:59.802 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.802+0000 7f0b17fff700 1 -- 172.21.15.59:0/2176124423 <== mon.2 v2:172.21.15.59:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f0b20021020 con 0x7f0b300a5b70 2023-09-30T02:07:59.802 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.802+0000 7f0b17fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:07:59.802 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.802+0000 7f0b17fff700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:07:59.803 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:07:59.803 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:07:59.806 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.804+0000 7f0b36507700 10 librados: watch_flush enter 2023-09-30T02:07:59.806 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.804+0000 7f0b36507700 10 librados: watch_flush exit 2023-09-30T02:07:59.806 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.804+0000 7f0b36507700 20 client.4466.objecter shutdown clearing up homeless session... 2023-09-30T02:07:59.806 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.804+0000 7f0b36507700 10 client.4466.objecter successfully canceled tick 2023-09-30T02:07:59.806 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.805+0000 7f0b36507700 1 -- 172.21.15.59:0/2176124423 >> v2:172.21.15.110:6800/115870 conn(0x7f0b1c068910 msgr2=0x7f0b1c06adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:59.806 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.805+0000 7f0b36507700 1 --2- 172.21.15.59:0/2176124423 >> v2:172.21.15.110:6800/115870 conn(0x7f0b1c068910 0x7f0b1c06adc0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0b300a8280 tx=0x7f0b24009040 comp rx=0 tx=0).stop 2023-09-30T02:07:59.806 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.805+0000 7f0b36507700 10 monclient: shutdown 2023-09-30T02:07:59.806 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.805+0000 7f0b36507700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:07:59.806 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.805+0000 7f0b36507700 1 -- 172.21.15.59:0/2176124423 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0b300a5b70 msgr2=0x7f0b30144dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:07:59.806 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.805+0000 7f0b36507700 1 --2- 172.21.15.59:0/2176124423 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0b300a5b70 0x7f0b30144dc0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f0b2000c7c0 tx=0x7f0b20007f70 comp rx=0 tx=0).stop 2023-09-30T02:07:59.807 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.805+0000 7f0b36507700 1 -- 172.21.15.59:0/2176124423 shutdown_connections 2023-09-30T02:07:59.807 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.805+0000 7f0b36507700 1 --2- 172.21.15.59:0/2176124423 >> v2:172.21.15.110:6800/115870 conn(0x7f0b1c068910 0x7f0b1c06adc0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:59.807 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.805+0000 7f0b36507700 1 --2- 172.21.15.59:0/2176124423 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b30142290 0x7f0b30136bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:59.807 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.805+0000 7f0b36507700 1 --2- 172.21.15.59:0/2176124423 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0b300a5b70 0x7f0b30144dc0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:59.807 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.805+0000 7f0b36507700 1 --2- 172.21.15.59:0/2176124423 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0b300b1500 0x7f0b3013ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:07:59.807 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.805+0000 7f0b36507700 1 -- 172.21.15.59:0/2176124423 >> 172.21.15.59:0/2176124423 conn(0x7f0b300631f0 msgr2=0x7f0b300635c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:07:59.807 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.805+0000 7f0b36507700 1 -- 172.21.15.59:0/2176124423 shutdown_connections 2023-09-30T02:07:59.807 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.805+0000 7f0b36507700 1 -- 172.21.15.59:0/2176124423 wait complete. 2023-09-30T02:07:59.807 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:07:59.805+0000 7f0b36507700 1 librados: shutdown 2023-09-30T02:08:02.821 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:08:02.960 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:02.959+0000 7f919b687700 10 monclient: get_monmap_and_config 2023-09-30T02:08:02.960 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:02.959+0000 7f919b687700 10 monclient: build_initial_monmap 2023-09-30T02:08:02.960 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:02.959+0000 7f919b687700 10 monclient: monmap: 2023-09-30T02:08:02.960 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:08:02.960 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:02.960 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:08:02.961168+0000 2023-09-30T02:08:02.960 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:08:02.961168+0000 2023-09-30T02:08:02.961 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:08:02.961 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:02.961 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:08:02.961 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:08:02.961 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:08:02.961 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:08:02.961 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:08:02.961 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:02.963 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:02.960+0000 7f919b687700 1 Processor -- start 2023-09-30T02:08:02.963 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:02.960+0000 7f919b687700 1 -- start start 2023-09-30T02:08:02.963 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:02.960+0000 7f919b687700 10 monclient: init 2023-09-30T02:08:02.963 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:02.961+0000 7f919b687700 10 monclient: _reopen_session rank -1 2023-09-30T02:08:02.963 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:02.961+0000 7f919b687700 10 monclient: _add_conns ranks=[1,3,0,2,4] 2023-09-30T02:08:02.963 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:02.961+0000 7f919b687700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f91940a5b70 addr v1:172.21.15.110:6789/0 2023-09-30T02:08:02.963 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:02.961+0000 7f919b687700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f9194142290 0x7f91940b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:02.963 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:02.961+0000 7f919b687700 10 monclient(hunting): picked mon.noname-a con 0x7f9194142290 addr v2:172.21.15.59:3300/0 2023-09-30T02:08:02.964 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:02.961+0000 7f919b687700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f91940a5f40 addr v1:172.21.15.59:6789/0 2023-09-30T02:08:02.964 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:02.961+0000 7f919b687700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91940a75a0 con 0x7f91940a5f40 2023-09-30T02:08:02.964 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:02.961+0000 7f919b687700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91940a6360 con 0x7f91940a5b70 2023-09-30T02:08:02.964 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:02.961+0000 7f919b687700 10 monclient(hunting): start opening mon connection 2023-09-30T02:08:02.964 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:02.961+0000 7f919b687700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91940af3f0 con 0x7f9194142290 2023-09-30T02:08:02.964 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:02.961+0000 7f919b687700 10 monclient(hunting): _renew_subs 2023-09-30T02:08:02.964 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:02.961+0000 7f919b687700 10 monclient(hunting): authenticate will time out at 1928.264454s 2023-09-30T02:08:02.964 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:02.962+0000 7f9198c22700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f9194142290 0x7f91940b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:02.964 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:02.962+0000 7f9199423700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f91940a5b70 0x7f9194141b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.59:34878/0 (socket says 172.21.15.59:34878) 2023-09-30T02:08:02.965 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:02.962+0000 7f9199423700 1 -- 172.21.15.59:0/1154762700 learned_addr learned my addr 172.21.15.59:0/1154762700 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:08:02.965 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:02.962+0000 7f9198c22700 10 monclient(hunting): get_auth_request con 0x7f9194142290 auth_method 0 2023-09-30T02:08:02.965 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:02.962+0000 7f9198c22700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:08:02.965 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:02.962+0000 7f9198c22700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:02.965 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:02.962+0000 7f9198c22700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:02.966 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:02.963+0000 7f9199423700 1 -- 172.21.15.59:0/1154762700 >> v1:172.21.15.110:6789/0 conn(0x7f91940a5b70 legacy=0x7f9194141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2023-09-30T02:08:02.966 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:02.963+0000 7f9199423700 1 -- 172.21.15.59:0/1154762700 >> v1:172.21.15.110:6789/0 conn(0x7f91940a5b70 legacy=0x7f9194141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-09-30T02:08:02.966 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:02.963+0000 7f9199423700 1 --1- 172.21.15.59:0/1154762700 >> v1:172.21.15.110:6789/0 conn(0x7f91940a5b70 0x7f9194141b80 :-1 s=OPENED pgs=144 cs=1 l=1).handle_message read tag failed 2023-09-30T02:08:02.966 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:02.963+0000 7f9199423700 1 --1- 172.21.15.59:0/1154762700 >> v1:172.21.15.110:6789/0 conn(0x7f91940a5b70 0x7f9194141b80 :-1 s=OPENED pgs=144 cs=1 l=1).fault on lossy channel, failing 2023-09-30T02:08:02.966 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:02.963+0000 7f918bfff700 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.110:6789/0 2023-09-30T02:08:02.966 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:02.964+0000 7f9198c22700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:08:02.966 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:02.964+0000 7f9198c22700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:08:02.966 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:02.964+0000 7f9198c22700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:08:02.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:02.967+0000 7f9198c22700 10 monclient(hunting): handle_auth_done global_id 4527 payload 274 2023-09-30T02:08:02.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:02.967+0000 7f9198c22700 1 -- 172.21.15.59:0/1154762700 >> v1:172.21.15.110:6789/0 conn(0x7f91940a5b70 legacy=0x7f9194141b80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-09-30T02:08:02.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:02.967+0000 7f9198c22700 1 -- 172.21.15.59:0/1154762700 >> v1:172.21.15.59:6789/0 conn(0x7f91940a5f40 legacy=0x7f919413af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:02.970 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:02.967+0000 7f9198c22700 10 monclient: _finish_hunting 0 2023-09-30T02:08:02.970 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:02.967+0000 7f9198c22700 1 monclient: found mon.noname-a 2023-09-30T02:08:02.970 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:02.967+0000 7f9198c22700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:08:02.970 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:02.967+0000 7f9198c22700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.59:3300/0 2023-09-30T02:08:02.970 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:02.967+0000 7f9198c22700 1 -- 172.21.15.59:0/1154762700 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f919413b640 con 0x7f9194142290 2023-09-30T02:08:02.974 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:02.973+0000 7f9198c22700 1 --2- 172.21.15.59:0/1154762700 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9194142290 0x7f91940b1500 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f91940b1fc0 tx=0x7f91840091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de45edae08da746e server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:08:02.998 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:02.997+0000 7f918bfff700 1 -- 172.21.15.59:0/1154762700 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f918400b640 con 0x7f9194142290 2023-09-30T02:08:02.998 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:02.997+0000 7f918bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:08:02.998 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:02.997+0000 7f918bfff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T02:08:02.998 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:02.997+0000 7f918bfff700 10 monclient: dump: 2023-09-30T02:08:02.998 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:08:02.998 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:02.998 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:02.998 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:02.999 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:08:02.999 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:02.999 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:08:02.999 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:08:02.999 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:08:02.999 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:02.999 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:02.997+0000 7f918bfff700 10 monclient: _finish_auth 0 2023-09-30T02:08:02.999 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:02.997+0000 7f918bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:08:02.999 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:02.997+0000 7f918bfff700 1 -- 172.21.15.59:0/1154762700 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f918400b7a0 con 0x7f9194142290 2023-09-30T02:08:02.999 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:02.997+0000 7f919b687700 5 monclient: authenticate success, global_id 4527 2023-09-30T02:08:02.999 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:02.997+0000 7f919b687700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:08:02.999 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:02.997+0000 7f918bfff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:08:03.000 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:02.997+0000 7f919b687700 10 monclient: get_monmap_and_config success 2023-09-30T02:08:03.000 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:02.997+0000 7f919b687700 4 set_mon_vals no callback set 2023-09-30T02:08:03.000 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:02.997+0000 7f918bfff700 1 -- 172.21.15.59:0/1154762700 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f918400bb40 con 0x7f9194142290 2023-09-30T02:08:03.000 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:02.997+0000 7f918bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:08:03.000 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:02.997+0000 7f918bfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T02:08:03.000 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:02.997+0000 7f918bfff700 10 monclient: dump: 2023-09-30T02:08:03.000 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:08:03.000 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:03.000 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:03.000 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:03.000 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:08:03.000 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:03.001 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:08:03.001 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:08:03.001 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:08:03.001 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:03.004 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:02.998+0000 7f919b687700 10 monclient: shutdown 2023-09-30T02:08:03.004 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:02.998+0000 7f919b687700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:08:03.004 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:02.998+0000 7f919b687700 1 -- 172.21.15.59:0/1154762700 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9194142290 msgr2=0x7f91940b1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:03.005 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:02.998+0000 7f919b687700 1 --2- 172.21.15.59:0/1154762700 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9194142290 0x7f91940b1500 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f91940b1fc0 tx=0x7f91840091a0 comp rx=0 tx=0).stop 2023-09-30T02:08:03.005 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:02.998+0000 7f919b687700 1 -- 172.21.15.59:0/1154762700 shutdown_connections 2023-09-30T02:08:03.005 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:02.998+0000 7f919b687700 1 --2- 172.21.15.59:0/1154762700 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9194142290 0x7f91940b1500 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:03.005 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:02.998+0000 7f919b687700 1 -- 172.21.15.59:0/1154762700 >> 172.21.15.59:0/1154762700 conn(0x7f91940631f0 msgr2=0x7f91940635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:08:03.005 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:02.998+0000 7f919b687700 1 -- 172.21.15.59:0/1154762700 shutdown_connections 2023-09-30T02:08:03.005 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:02.998+0000 7f919b687700 1 -- 172.21.15.59:0/1154762700 wait complete. 2023-09-30T02:08:03.005 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:02.999+0000 7f919b687700 10 monclient: build_initial_monmap 2023-09-30T02:08:03.005 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:02.999+0000 7f919b687700 10 monclient: monmap: 2023-09-30T02:08:03.005 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:08:03.005 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:08:03.005 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:08:03.006 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:08:03.006 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:08:03.006 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:08:03.006 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:08:03.006 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:08:03.006 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:08:03.006 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:03.006 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:02.999+0000 7f919b687700 1 librados: starting msgr at 2023-09-30T02:08:03.006 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:02.999+0000 7f919b687700 1 librados: starting objecter 2023-09-30T02:08:03.006 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:02.999+0000 7f919b687700 1 Processor -- start 2023-09-30T02:08:03.006 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:02.999+0000 7f919b687700 1 -- start start 2023-09-30T02:08:03.007 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:02.999+0000 7f919b687700 1 librados: setting wanted keys 2023-09-30T02:08:03.007 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:02.999+0000 7f919b687700 1 librados: calling monclient init 2023-09-30T02:08:03.007 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:02.999+0000 7f919b687700 10 monclient: init 2023-09-30T02:08:03.007 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.000+0000 7f919b687700 10 monclient: _reopen_session rank -1 2023-09-30T02:08:03.007 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.000+0000 7f919b687700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T02:08:03.007 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.000+0000 7f919b687700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91940a5b70 0x7f9194134de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:03.007 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.000+0000 7f919b687700 10 monclient(hunting): picked mon.noname-a con 0x7f91940a5b70 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:08:03.007 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.000+0000 7f919b687700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f91940a5f40 0x7f919413c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:03.007 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.000+0000 7f919b687700 10 monclient(hunting): picked mon.noname-b con 0x7f91940a5f40 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:08:03.007 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.000+0000 7f919b687700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9194142290 0x7f919413ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:03.007 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.000+0000 7f919b687700 10 monclient(hunting): picked mon.noname-c con 0x7f9194142290 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:08:03.007 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.000+0000 7f919b687700 10 monclient(hunting): start opening mon connection 2023-09-30T02:08:03.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.000+0000 7f919b687700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f919413def0 con 0x7f91940a5b70 2023-09-30T02:08:03.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.000+0000 7f919b687700 10 monclient(hunting): start opening mon connection 2023-09-30T02:08:03.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.000+0000 7f919b687700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f919413e580 con 0x7f91940a5f40 2023-09-30T02:08:03.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.000+0000 7f919b687700 10 monclient(hunting): start opening mon connection 2023-09-30T02:08:03.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.000+0000 7f919b687700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f919413e160 con 0x7f9194142290 2023-09-30T02:08:03.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.000+0000 7f919b687700 10 monclient(hunting): _renew_subs 2023-09-30T02:08:03.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.000+0000 7f919b687700 10 monclient(hunting): authenticate will time out at 1928.303711s 2023-09-30T02:08:03.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.001+0000 7f9199423700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91940a5b70 0x7f9194134de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:03.009 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.001+0000 7f9199c24700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9194142290 0x7f919413ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:03.010 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.001+0000 7f9198c22700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f91940a5f40 0x7f919413c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:03.010 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.001+0000 7f9199423700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91940a5b70 0x7f9194134de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60572/0 (socket says 172.21.15.59:60572) 2023-09-30T02:08:03.010 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.001+0000 7f9199423700 1 -- 172.21.15.59:0/3084330434 learned_addr learned my addr 172.21.15.59:0/3084330434 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:08:03.010 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.001+0000 7f9199423700 10 monclient(hunting): get_auth_request con 0x7f91940a5b70 auth_method 0 2023-09-30T02:08:03.010 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.001+0000 7f9199423700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:08:03.011 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.001+0000 7f9199423700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:03.011 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.001+0000 7f9199423700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:03.011 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.001+0000 7f9199c24700 10 monclient(hunting): get_auth_request con 0x7f9194142290 auth_method 0 2023-09-30T02:08:03.011 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.001+0000 7f9199c24700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:08:03.011 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.001+0000 7f9199c24700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:03.011 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.001+0000 7f9199c24700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:03.011 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.001+0000 7f9198c22700 10 monclient(hunting): get_auth_request con 0x7f91940a5f40 auth_method 0 2023-09-30T02:08:03.011 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.001+0000 7f9198c22700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:08:03.011 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.001+0000 7f9198c22700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:03.011 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.001+0000 7f9198c22700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:03.012 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.002+0000 7f9199423700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:08:03.012 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.002+0000 7f9199423700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:08:03.012 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.002+0000 7f9199423700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:08:03.012 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.002+0000 7f9199c24700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:08:03.012 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.002+0000 7f9199c24700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:08:03.012 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.002+0000 7f9199c24700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:08:03.012 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.002+0000 7f9198c22700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:08:03.012 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.002+0000 7f9198c22700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:08:03.012 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.002+0000 7f9198c22700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:08:03.014 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.006+0000 7f9199423700 10 monclient(hunting): handle_auth_done global_id 4533 payload 824 2023-09-30T02:08:03.014 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.006+0000 7f9199423700 1 -- 172.21.15.59:0/3084330434 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9194142290 msgr2=0x7f919413ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:03.014 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.006+0000 7f9199423700 1 --2- 172.21.15.59:0/3084330434 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9194142290 0x7f919413ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:03.014 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.006+0000 7f9199423700 1 -- 172.21.15.59:0/3084330434 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f91940a5f40 msgr2=0x7f919413c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:03.014 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.006+0000 7f9199423700 1 --2- 172.21.15.59:0/3084330434 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f91940a5f40 0x7f919413c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:03.014 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.006+0000 7f9199423700 10 monclient: _finish_hunting 0 2023-09-30T02:08:03.014 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.006+0000 7f9199423700 1 monclient: found mon.noname-a 2023-09-30T02:08:03.015 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.006+0000 7f9199423700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:08:03.015 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.006+0000 7f9199423700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.59:3300/0 2023-09-30T02:08:03.015 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.006+0000 7f9199423700 1 -- 172.21.15.59:0/3084330434 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9184009420 con 0x7f91940a5b70 2023-09-30T02:08:03.015 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.009+0000 7f9199423700 1 --2- 172.21.15.59:0/3084330434 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91940a5b70 0x7f9194134de0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f919000c7e0 tx=0x7f9190001180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:08:03.015 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.009+0000 7f91897fa700 10 client.?.objecter ms_handle_connect 0x7f91940a5b70 2023-09-30T02:08:03.015 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.009+0000 7f91897fa700 10 client.?.objecter resend_mon_ops 2023-09-30T02:08:03.015 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.013+0000 7f91897fa700 1 -- 172.21.15.59:0/3084330434 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9190005b30 con 0x7f91940a5b70 2023-09-30T02:08:03.015 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.013+0000 7f91897fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:08:03.015 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.013+0000 7f91897fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T02:08:03.015 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.013+0000 7f91897fa700 10 monclient: dump: 2023-09-30T02:08:03.015 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:08:03.015 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:03.016 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:03.016 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:03.016 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:08:03.016 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:03.016 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:08:03.016 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:08:03.016 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:08:03.016 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:03.016 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.013+0000 7f91897fa700 10 monclient: _finish_auth 0 2023-09-30T02:08:03.016 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.013+0000 7f91897fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:08:03.016 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.013+0000 7f91897fa700 1 -- 172.21.15.59:0/3084330434 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9190005c90 con 0x7f91940a5b70 2023-09-30T02:08:03.017 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.013+0000 7f919b687700 5 monclient: authenticate success, global_id 4533 2023-09-30T02:08:03.017 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.013+0000 7f919b687700 10 monclient: _renew_subs 2023-09-30T02:08:03.017 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.013+0000 7f919b687700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:08:03.017 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.013+0000 7f919b687700 1 -- 172.21.15.59:0/3084330434 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f919413cef0 con 0x7f91940a5b70 2023-09-30T02:08:03.017 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.013+0000 7f91897fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:08:03.017 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.013+0000 7f91897fa700 1 -- 172.21.15.59:0/3084330434 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9190007320 con 0x7f91940a5b70 2023-09-30T02:08:03.017 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.013+0000 7f91897fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:08:03.017 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.013+0000 7f91897fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T02:08:03.017 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.013+0000 7f91897fa700 10 monclient: dump: 2023-09-30T02:08:03.017 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:08:03.018 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:03.018 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:03.018 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:03.018 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:08:03.018 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:03.018 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:08:03.018 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:08:03.018 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:08:03.018 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:03.018 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.013+0000 7f919b687700 10 client.4533.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:08:03.018 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.013+0000 7f919b687700 10 monclient: _renew_subs 2023-09-30T02:08:03.018 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.013+0000 7f919b687700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:08:03.019 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.013+0000 7f919b687700 1 -- 172.21.15.59:0/3084330434 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91941410b0 con 0x7f91940a5b70 2023-09-30T02:08:03.019 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.013+0000 7f919b687700 1 librados: init done 2023-09-30T02:08:03.019 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.013+0000 7f918b7fe700 4 set_mon_vals no callback set 2023-09-30T02:08:03.019 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.014+0000 7f919b687700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:08:03.019 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.014+0000 7f919b687700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:08:03.020 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.014+0000 7f919b687700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:08:03.024 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.014+0000 7f919b687700 1 -- 172.21.15.59:0/3084330434 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f919413e890 con 0x7f91940a5b70 2023-09-30T02:08:03.306 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.306+0000 7f91897fa700 1 -- 172.21.15.59:0/3084330434 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f919001b3b0 con 0x7f91940a5b70 2023-09-30T02:08:03.307 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.307+0000 7f91897fa700 1 --2- 172.21.15.59:0/3084330434 >> v2:172.21.15.110:6800/115870 conn(0x7f9178068890 0x7f917806ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:03.307 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.307+0000 7f9198c22700 1 --2- 172.21.15.59:0/3084330434 >> v2:172.21.15.110:6800/115870 conn(0x7f9178068890 0x7f917806ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:03.307 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.307+0000 7f9198c22700 10 monclient: get_auth_request con 0x7f9178068890 auth_method 0 2023-09-30T02:08:03.308 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.307+0000 7f9198c22700 1 --2- 172.21.15.59:0/3084330434 >> v2:172.21.15.110:6800/115870 conn(0x7f9178068890 0x7f917806ad40 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f918400c010 tx=0x7f9184016000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:08:03.308 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.307+0000 7f91897fa700 10 client.4533.objecter ms_handle_connect 0x7f9178068890 2023-09-30T02:08:03.326 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.325+0000 7f91897fa700 1 -- 172.21.15.59:0/3084330434 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f91900554f0 con 0x7f91940a5b70 2023-09-30T02:08:03.326 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.325+0000 7f91897fa700 10 client.4533.objecter ms_dispatch 0x7f9194001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:08:03.326 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.325+0000 7f91897fa700 3 client.4533.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:08:03.326 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.325+0000 7f91897fa700 3 client.4533.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:08:03.326 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.325+0000 7f91897fa700 20 client.4533.objecter dump_active .. 0 homeless 2023-09-30T02:08:03.326 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.325+0000 7f91897fa700 1 -- 172.21.15.59:0/3084330434 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9190051130 con 0x7f91940a5b70 2023-09-30T02:08:03.326 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.325+0000 7f91897fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:08:03.327 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.325+0000 7f91897fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:08:03.590 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.589+0000 7f919b687700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:08:03.590 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.589+0000 7f919b687700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:08:03.590 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.589+0000 7f919b687700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:08:03.590 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.589+0000 7f919b687700 1 -- 172.21.15.59:0/3084330434 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f91941416e0 con 0x7f91940a5b70 2023-09-30T02:08:03.596 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.595+0000 7f91897fa700 1 -- 172.21.15.59:0/3084330434 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f9190017080 con 0x7f91940a5b70 2023-09-30T02:08:03.596 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.595+0000 7f91897fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:08:03.596 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.595+0000 7f91897fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:08:03.596 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:08:03.596 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:08:03.598 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.597+0000 7f919b687700 10 librados: watch_flush enter 2023-09-30T02:08:03.599 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.597+0000 7f919b687700 10 librados: watch_flush exit 2023-09-30T02:08:03.599 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.597+0000 7f919b687700 20 client.4533.objecter shutdown clearing up homeless session... 2023-09-30T02:08:03.599 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.597+0000 7f919b687700 10 client.4533.objecter successfully canceled tick 2023-09-30T02:08:03.599 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.598+0000 7f919b687700 1 -- 172.21.15.59:0/3084330434 >> v2:172.21.15.110:6800/115870 conn(0x7f9178068890 msgr2=0x7f917806ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:03.599 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.598+0000 7f919b687700 1 --2- 172.21.15.59:0/3084330434 >> v2:172.21.15.110:6800/115870 conn(0x7f9178068890 0x7f917806ad40 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f918400c010 tx=0x7f9184016000 comp rx=0 tx=0).stop 2023-09-30T02:08:03.599 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.598+0000 7f919b687700 10 monclient: shutdown 2023-09-30T02:08:03.599 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.598+0000 7f919b687700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:08:03.599 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.598+0000 7f919b687700 1 -- 172.21.15.59:0/3084330434 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91940a5b70 msgr2=0x7f9194134de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:03.599 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.598+0000 7f919b687700 1 --2- 172.21.15.59:0/3084330434 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91940a5b70 0x7f9194134de0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f919000c7e0 tx=0x7f9190001180 comp rx=0 tx=0).stop 2023-09-30T02:08:03.599 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.598+0000 7f919b687700 1 -- 172.21.15.59:0/3084330434 shutdown_connections 2023-09-30T02:08:03.599 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.598+0000 7f919b687700 1 --2- 172.21.15.59:0/3084330434 >> v2:172.21.15.110:6800/115870 conn(0x7f9178068890 0x7f917806ad40 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:03.600 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.598+0000 7f919b687700 1 --2- 172.21.15.59:0/3084330434 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9194142290 0x7f919413ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:03.600 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.598+0000 7f919b687700 1 --2- 172.21.15.59:0/3084330434 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91940a5b70 0x7f9194134de0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:03.600 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.598+0000 7f919b687700 1 --2- 172.21.15.59:0/3084330434 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f91940a5f40 0x7f919413c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:03.600 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.598+0000 7f919b687700 1 -- 172.21.15.59:0/3084330434 >> 172.21.15.59:0/3084330434 conn(0x7f91940631f0 msgr2=0x7f91940afda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:08:03.600 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.598+0000 7f919b687700 1 -- 172.21.15.59:0/3084330434 shutdown_connections 2023-09-30T02:08:03.600 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.598+0000 7f919b687700 1 -- 172.21.15.59:0/3084330434 wait complete. 2023-09-30T02:08:03.600 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:03.598+0000 7f919b687700 1 librados: shutdown 2023-09-30T02:08:06.613 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:08:06.778 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.778+0000 7fb632e99700 10 monclient: get_monmap_and_config 2023-09-30T02:08:06.779 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.778+0000 7fb632e99700 10 monclient: build_initial_monmap 2023-09-30T02:08:06.779 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.778+0000 7fb632e99700 10 monclient: monmap: 2023-09-30T02:08:06.779 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:08:06.779 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:06.779 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:08:06.779675+0000 2023-09-30T02:08:06.779 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:08:06.779675+0000 2023-09-30T02:08:06.779 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:08:06.779 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:06.779 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:08:06.779 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:08:06.779 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:08:06.779 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:08:06.780 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:08:06.780 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:06.782 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.779+0000 7fb632e99700 1 Processor -- start 2023-09-30T02:08:06.782 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.779+0000 7fb632e99700 1 -- start start 2023-09-30T02:08:06.782 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.779+0000 7fb632e99700 10 monclient: init 2023-09-30T02:08:06.782 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.779+0000 7fb632e99700 10 monclient: _reopen_session rank -1 2023-09-30T02:08:06.782 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.779+0000 7fb632e99700 10 monclient: _add_conns ranks=[0,2,1,4,3] 2023-09-30T02:08:06.782 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.779+0000 7fb632e99700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb62c0a5b70 addr v1:172.21.15.59:6789/0 2023-09-30T02:08:06.782 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.779+0000 7fb632e99700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb62c142290 0x7fb62c0b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:06.782 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.779+0000 7fb632e99700 10 monclient(hunting): picked mon.noname-b con 0x7fb62c142290 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:08:06.782 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.779+0000 7fb632e99700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb62c0a5f40 addr v1:172.21.15.110:6789/0 2023-09-30T02:08:06.782 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.779+0000 7fb632e99700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb62c0a6380 con 0x7fb62c0a5b70 2023-09-30T02:08:06.783 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.779+0000 7fb632e99700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb62c0a75a0 con 0x7fb62c0a5f40 2023-09-30T02:08:06.783 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.779+0000 7fb632e99700 10 monclient(hunting): start opening mon connection 2023-09-30T02:08:06.783 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.779+0000 7fb632e99700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb62c0af3f0 con 0x7fb62c142290 2023-09-30T02:08:06.783 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.779+0000 7fb632e99700 10 monclient(hunting): _renew_subs 2023-09-30T02:08:06.783 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.779+0000 7fb632e99700 10 monclient(hunting): authenticate will time out at 1932.082899s 2023-09-30T02:08:06.783 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.780+0000 7fb623fff700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb62c142290 0x7fb62c0b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:06.783 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.780+0000 7fb631436700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fb62c0a5f40 0x7fb62c13af30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.59:34886/0 (socket says 172.21.15.59:34886) 2023-09-30T02:08:06.783 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.780+0000 7fb631436700 1 -- 172.21.15.59:0/1705456831 learned_addr learned my addr 172.21.15.59:0/1705456831 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:08:06.783 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.780+0000 7fb623fff700 10 monclient(hunting): get_auth_request con 0x7fb62c142290 auth_method 0 2023-09-30T02:08:06.783 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.780+0000 7fb623fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:08:06.783 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.780+0000 7fb623fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:06.783 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.780+0000 7fb623fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:06.784 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.781+0000 7fb623fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:08:06.784 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.781+0000 7fb623fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:08:06.784 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.781+0000 7fb623fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:08:06.784 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.784+0000 7fb623fff700 10 monclient(hunting): handle_auth_done global_id 4472 payload 274 2023-09-30T02:08:06.784 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.784+0000 7fb623fff700 1 -- 172.21.15.59:0/1705456831 >> v1:172.21.15.110:6789/0 conn(0x7fb62c0a5f40 legacy=0x7fb62c13af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:06.784 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.784+0000 7fb623fff700 1 -- 172.21.15.59:0/1705456831 >> v1:172.21.15.59:6789/0 conn(0x7fb62c0a5b70 legacy=0x7fb62c141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:08:06.785 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.784+0000 7fb623fff700 10 monclient: _finish_hunting 0 2023-09-30T02:08:06.785 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.784+0000 7fb623fff700 1 monclient: found mon.noname-b 2023-09-30T02:08:06.785 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.784+0000 7fb623fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:08:06.785 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.784+0000 7fb623fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.59:3301/0 2023-09-30T02:08:06.785 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.784+0000 7fb623fff700 1 -- 172.21.15.59:0/1705456831 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb62c13b640 con 0x7fb62c142290 2023-09-30T02:08:06.786 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.786+0000 7fb623fff700 1 --2- 172.21.15.59:0/1705456831 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb62c142290 0x7fb62c0b1500 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb61c009910 tx=0x7fb61c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91969775250b07b3 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:08:06.791 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.791+0000 7fb6237fe700 1 -- 172.21.15.59:0/1705456831 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb61c00b820 con 0x7fb62c142290 2023-09-30T02:08:06.792 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.791+0000 7fb6237fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:08:06.792 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.791+0000 7fb6237fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:08:06.792 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.791+0000 7fb6237fe700 10 monclient: dump: 2023-09-30T02:08:06.792 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:08:06.792 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:06.792 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:06.792 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:06.792 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:08:06.792 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:06.792 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:08:06.792 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:08:06.793 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:08:06.793 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:06.793 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.791+0000 7fb6237fe700 10 monclient: _finish_auth 0 2023-09-30T02:08:06.793 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.791+0000 7fb6237fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:08:06.793 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.791+0000 7fb6237fe700 1 -- 172.21.15.59:0/1705456831 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb61c00b980 con 0x7fb62c142290 2023-09-30T02:08:06.793 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.791+0000 7fb632e99700 5 monclient: authenticate success, global_id 4472 2023-09-30T02:08:06.793 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.791+0000 7fb632e99700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:08:06.793 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.791+0000 7fb6237fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:08:06.793 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.791+0000 7fb632e99700 10 monclient: get_monmap_and_config success 2023-09-30T02:08:06.793 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.791+0000 7fb6237fe700 1 -- 172.21.15.59:0/1705456831 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb61c00bd20 con 0x7fb62c142290 2023-09-30T02:08:06.793 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.791+0000 7fb632e99700 4 set_mon_vals no callback set 2023-09-30T02:08:06.793 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.791+0000 7fb6237fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:08:06.794 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.791+0000 7fb6237fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:08:06.794 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.791+0000 7fb6237fe700 10 monclient: dump: 2023-09-30T02:08:06.794 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:08:06.794 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:06.794 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:06.794 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:06.794 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:08:06.794 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:06.794 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:08:06.794 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:08:06.794 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:08:06.794 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:06.797 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.792+0000 7fb632e99700 10 monclient: shutdown 2023-09-30T02:08:06.797 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.792+0000 7fb632e99700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:08:06.797 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.792+0000 7fb632e99700 1 -- 172.21.15.59:0/1705456831 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb62c142290 msgr2=0x7fb62c0b1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:06.797 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.792+0000 7fb632e99700 1 --2- 172.21.15.59:0/1705456831 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb62c142290 0x7fb62c0b1500 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb61c009910 tx=0x7fb61c0091a0 comp rx=0 tx=0).stop 2023-09-30T02:08:06.797 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.792+0000 7fb632e99700 1 -- 172.21.15.59:0/1705456831 shutdown_connections 2023-09-30T02:08:06.797 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.792+0000 7fb632e99700 1 --2- 172.21.15.59:0/1705456831 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb62c142290 0x7fb62c0b1500 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:06.798 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.792+0000 7fb632e99700 1 -- 172.21.15.59:0/1705456831 >> 172.21.15.59:0/1705456831 conn(0x7fb62c0631f0 msgr2=0x7fb62c0635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:08:06.798 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.792+0000 7fb632e99700 1 -- 172.21.15.59:0/1705456831 shutdown_connections 2023-09-30T02:08:06.798 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.792+0000 7fb632e99700 1 -- 172.21.15.59:0/1705456831 wait complete. 2023-09-30T02:08:06.798 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.793+0000 7fb632e99700 10 monclient: build_initial_monmap 2023-09-30T02:08:06.798 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.793+0000 7fb632e99700 10 monclient: monmap: 2023-09-30T02:08:06.798 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:08:06.798 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:08:06.798 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:08:06.798 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:08:06.798 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:08:06.798 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:08:06.799 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:08:06.799 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:08:06.799 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:08:06.799 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:06.799 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.793+0000 7fb632e99700 1 librados: starting msgr at 2023-09-30T02:08:06.799 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.793+0000 7fb632e99700 1 librados: starting objecter 2023-09-30T02:08:06.799 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.793+0000 7fb632e99700 1 Processor -- start 2023-09-30T02:08:06.799 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.794+0000 7fb632e99700 1 -- start start 2023-09-30T02:08:06.799 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.794+0000 7fb632e99700 1 librados: setting wanted keys 2023-09-30T02:08:06.800 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.794+0000 7fb632e99700 1 librados: calling monclient init 2023-09-30T02:08:06.800 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.794+0000 7fb632e99700 10 monclient: init 2023-09-30T02:08:06.800 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.794+0000 7fb632e99700 10 monclient: _reopen_session rank -1 2023-09-30T02:08:06.800 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.794+0000 7fb632e99700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T02:08:06.800 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.794+0000 7fb632e99700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb62c0a5b70 0x7fb62c1407c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:06.800 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.794+0000 7fb632e99700 10 monclient(hunting): picked mon.noname-b con 0x7fb62c0a5b70 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:08:06.800 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.794+0000 7fb632e99700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb62c0a5f40 0x7fb62c13dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:06.800 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.794+0000 7fb632e99700 10 monclient(hunting): picked mon.noname-a con 0x7fb62c0a5f40 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:08:06.800 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.794+0000 7fb632e99700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb62c142290 0x7fb62c14e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:06.800 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.794+0000 7fb632e99700 10 monclient(hunting): picked mon.noname-c con 0x7fb62c142290 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:08:06.800 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.794+0000 7fb632e99700 10 monclient(hunting): start opening mon connection 2023-09-30T02:08:06.800 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.794+0000 7fb632e99700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb62c13e4d0 con 0x7fb62c0a5f40 2023-09-30T02:08:06.801 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.794+0000 7fb632e99700 10 monclient(hunting): start opening mon connection 2023-09-30T02:08:06.801 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.794+0000 7fb632e99700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb62c0a8230 con 0x7fb62c0a5b70 2023-09-30T02:08:06.801 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.794+0000 7fb632e99700 10 monclient(hunting): start opening mon connection 2023-09-30T02:08:06.801 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.794+0000 7fb632e99700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb62c1410f0 con 0x7fb62c142290 2023-09-30T02:08:06.801 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.794+0000 7fb632e99700 10 monclient(hunting): _renew_subs 2023-09-30T02:08:06.801 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.794+0000 7fb632e99700 10 monclient(hunting): authenticate will time out at 1932.097942s 2023-09-30T02:08:06.801 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.795+0000 7fb623fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb62c0a5f40 0x7fb62c13dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:06.801 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.795+0000 7fb623fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb62c0a5f40 0x7fb62c13dd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:60576/0 (socket says 172.21.15.59:60576) 2023-09-30T02:08:06.801 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.795+0000 7fb623fff700 1 -- 172.21.15.59:0/4269708763 learned_addr learned my addr 172.21.15.59:0/4269708763 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:08:06.801 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.795+0000 7fb623fff700 10 monclient(hunting): get_auth_request con 0x7fb62c0a5f40 auth_method 0 2023-09-30T02:08:06.801 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.795+0000 7fb623fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:08:06.801 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.795+0000 7fb623fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:06.802 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.795+0000 7fb623fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:06.802 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.796+0000 7fb630c35700 1 --2- 172.21.15.59:0/4269708763 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb62c0a5b70 0x7fb62c1407c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:06.802 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.797+0000 7fb623fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:08:06.802 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.797+0000 7fb623fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:08:06.802 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.797+0000 7fb623fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:08:06.802 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.797+0000 7fb630c35700 10 monclient(hunting): get_auth_request con 0x7fb62c0a5b70 auth_method 0 2023-09-30T02:08:06.802 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.797+0000 7fb630c35700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:08:06.802 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.797+0000 7fb630c35700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:06.802 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.797+0000 7fb630c35700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:06.803 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.797+0000 7fb630c35700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:08:06.804 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.797+0000 7fb630c35700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:08:06.804 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.797+0000 7fb630c35700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:08:06.804 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.798+0000 7fb631436700 1 --2- 172.21.15.59:0/4269708763 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb62c142290 0x7fb62c14e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:06.804 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.799+0000 7fb631436700 10 monclient(hunting): get_auth_request con 0x7fb62c142290 auth_method 0 2023-09-30T02:08:06.804 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.799+0000 7fb631436700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:08:06.804 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.799+0000 7fb631436700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:06.804 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.799+0000 7fb631436700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:06.804 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.799+0000 7fb631436700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:08:06.804 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.799+0000 7fb631436700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:08:06.804 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.799+0000 7fb631436700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:08:06.804 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.802+0000 7fb630c35700 10 monclient(hunting): handle_auth_done global_id 4540 payload 824 2023-09-30T02:08:06.804 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.802+0000 7fb630c35700 1 -- 172.21.15.59:0/4269708763 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb62c142290 msgr2=0x7fb62c14e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:06.805 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.802+0000 7fb630c35700 1 --2- 172.21.15.59:0/4269708763 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb62c142290 0x7fb62c14e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:06.805 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.802+0000 7fb630c35700 1 -- 172.21.15.59:0/4269708763 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb62c0a5f40 msgr2=0x7fb62c13dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:06.805 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.802+0000 7fb630c35700 1 --2- 172.21.15.59:0/4269708763 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb62c0a5f40 0x7fb62c13dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:06.805 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.802+0000 7fb630c35700 10 monclient: _finish_hunting 0 2023-09-30T02:08:06.805 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.802+0000 7fb630c35700 1 monclient: found mon.noname-b 2023-09-30T02:08:06.805 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.802+0000 7fb630c35700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:08:06.805 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.802+0000 7fb630c35700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:08:06.805 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.802+0000 7fb630c35700 1 -- 172.21.15.59:0/4269708763 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb61c009420 con 0x7fb62c0a5b70 2023-09-30T02:08:06.806 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.805+0000 7fb630c35700 1 --2- 172.21.15.59:0/4269708763 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb62c0a5b70 0x7fb62c1407c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb62800b5b0 tx=0x7fb6280090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:08:06.806 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.805+0000 7fb620ff9700 10 client.?.objecter ms_handle_connect 0x7fb62c0a5b70 2023-09-30T02:08:06.806 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.805+0000 7fb620ff9700 10 client.?.objecter resend_mon_ops 2023-09-30T02:08:06.810 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.809+0000 7fb620ff9700 1 -- 172.21.15.59:0/4269708763 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb62800f510 con 0x7fb62c0a5b70 2023-09-30T02:08:06.810 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.809+0000 7fb620ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:08:06.810 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.809+0000 7fb620ff9700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:08:06.810 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.809+0000 7fb620ff9700 10 monclient: dump: 2023-09-30T02:08:06.810 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:08:06.811 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:06.811 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:06.811 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:06.811 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:08:06.811 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:06.811 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:08:06.812 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:08:06.812 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:08:06.812 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:06.812 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.809+0000 7fb620ff9700 10 monclient: _finish_auth 0 2023-09-30T02:08:06.812 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.809+0000 7fb620ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:08:06.812 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.809+0000 7fb620ff9700 1 -- 172.21.15.59:0/4269708763 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb62800f690 con 0x7fb62c0a5b70 2023-09-30T02:08:06.812 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.809+0000 7fb620ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:08:06.812 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.809+0000 7fb620ff9700 1 -- 172.21.15.59:0/4269708763 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb62800fa50 con 0x7fb62c0a5b70 2023-09-30T02:08:06.812 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.809+0000 7fb632e99700 5 monclient: authenticate success, global_id 4540 2023-09-30T02:08:06.812 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.809+0000 7fb632e99700 10 monclient: _renew_subs 2023-09-30T02:08:06.813 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.809+0000 7fb632e99700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:08:06.813 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.809+0000 7fb632e99700 1 -- 172.21.15.59:0/4269708763 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb62c13e960 con 0x7fb62c0a5b70 2023-09-30T02:08:06.813 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.809+0000 7fb620ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:08:06.813 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.809+0000 7fb622ffd700 4 set_mon_vals no callback set 2023-09-30T02:08:06.813 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.809+0000 7fb620ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:08:06.813 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.809+0000 7fb620ff9700 10 monclient: dump: 2023-09-30T02:08:06.813 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:08:06.813 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:06.813 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:06.813 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:06.813 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:08:06.813 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:06.814 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:08:06.814 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:08:06.814 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:08:06.814 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:06.814 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.809+0000 7fb632e99700 10 client.4540.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:08:06.814 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.809+0000 7fb632e99700 10 monclient: _renew_subs 2023-09-30T02:08:06.814 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.809+0000 7fb632e99700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:08:06.814 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.809+0000 7fb632e99700 1 -- 172.21.15.59:0/4269708763 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb62c14ea60 con 0x7fb62c0a5b70 2023-09-30T02:08:06.814 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.809+0000 7fb632e99700 1 librados: init done 2023-09-30T02:08:06.815 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.810+0000 7fb632e99700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:08:06.815 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.810+0000 7fb632e99700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:08:06.815 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.810+0000 7fb632e99700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:08:06.815 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.810+0000 7fb632e99700 1 -- 172.21.15.59:0/4269708763 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb62c14ecb0 con 0x7fb62c0a5b70 2023-09-30T02:08:06.908 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.908+0000 7fb620ff9700 1 -- 172.21.15.59:0/4269708763 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fb628024430 con 0x7fb62c0a5b70 2023-09-30T02:08:06.909 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.908+0000 7fb620ff9700 1 --2- 172.21.15.59:0/4269708763 >> v2:172.21.15.110:6800/115870 conn(0x7fb618068890 0x7fb61806ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:06.909 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.908+0000 7fb623fff700 1 --2- 172.21.15.59:0/4269708763 >> v2:172.21.15.110:6800/115870 conn(0x7fb618068890 0x7fb61806ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:06.909 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.909+0000 7fb623fff700 10 monclient: get_auth_request con 0x7fb618068890 auth_method 0 2023-09-30T02:08:06.909 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.909+0000 7fb623fff700 1 --2- 172.21.15.59:0/4269708763 >> v2:172.21.15.110:6800/115870 conn(0x7fb618068890 0x7fb61806ad40 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb62c140dc0 tx=0x7fb61c016000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:08:06.910 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.909+0000 7fb620ff9700 10 client.4540.objecter ms_handle_connect 0x7fb618068890 2023-09-30T02:08:06.913 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.912+0000 7fb620ff9700 1 -- 172.21.15.59:0/4269708763 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fb628055fd0 con 0x7fb62c0a5b70 2023-09-30T02:08:06.913 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.912+0000 7fb620ff9700 10 client.4540.objecter ms_dispatch 0x7fb62c001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:08:06.913 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.912+0000 7fb620ff9700 3 client.4540.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:08:06.913 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.912+0000 7fb620ff9700 3 client.4540.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:08:06.913 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:06.912+0000 7fb620ff9700 20 client.4540.objecter dump_active .. 0 homeless 2023-09-30T02:08:07.077 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:07.077+0000 7fb620ff9700 1 -- 172.21.15.59:0/4269708763 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb628051c10 con 0x7fb62c0a5b70 2023-09-30T02:08:07.078 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:07.077+0000 7fb620ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:08:07.078 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:07.077+0000 7fb620ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:08:07.332 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:07.331+0000 7fb632e99700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:08:07.332 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:07.331+0000 7fb632e99700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:08:07.332 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:07.331+0000 7fb632e99700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:08:07.332 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:07.331+0000 7fb632e99700 1 -- 172.21.15.59:0/4269708763 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fb62c0a8370 con 0x7fb62c0a5b70 2023-09-30T02:08:07.338 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:07.338+0000 7fb620ff9700 1 -- 172.21.15.59:0/4269708763 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fb62c0a8370 con 0x7fb62c0a5b70 2023-09-30T02:08:07.338 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:07.338+0000 7fb620ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:08:07.338 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:07.338+0000 7fb620ff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:08:07.338 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:08:07.339 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:08:07.341 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:07.340+0000 7fb632e99700 10 librados: watch_flush enter 2023-09-30T02:08:07.342 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:07.340+0000 7fb632e99700 10 librados: watch_flush exit 2023-09-30T02:08:07.342 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:07.340+0000 7fb632e99700 20 client.4540.objecter shutdown clearing up homeless session... 2023-09-30T02:08:07.342 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:07.340+0000 7fb632e99700 10 client.4540.objecter successfully canceled tick 2023-09-30T02:08:07.342 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:07.340+0000 7fb632e99700 1 -- 172.21.15.59:0/4269708763 >> v2:172.21.15.110:6800/115870 conn(0x7fb618068890 msgr2=0x7fb61806ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:07.342 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:07.340+0000 7fb632e99700 1 --2- 172.21.15.59:0/4269708763 >> v2:172.21.15.110:6800/115870 conn(0x7fb618068890 0x7fb61806ad40 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb62c140dc0 tx=0x7fb61c016000 comp rx=0 tx=0).stop 2023-09-30T02:08:07.342 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:07.340+0000 7fb632e99700 10 monclient: shutdown 2023-09-30T02:08:07.342 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:07.340+0000 7fb632e99700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:08:07.342 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:07.340+0000 7fb632e99700 1 -- 172.21.15.59:0/4269708763 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb62c0a5b70 msgr2=0x7fb62c1407c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:07.342 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:07.340+0000 7fb632e99700 1 --2- 172.21.15.59:0/4269708763 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb62c0a5b70 0x7fb62c1407c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb62800b5b0 tx=0x7fb6280090f0 comp rx=0 tx=0).stop 2023-09-30T02:08:07.342 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:07.340+0000 7fb632e99700 1 -- 172.21.15.59:0/4269708763 shutdown_connections 2023-09-30T02:08:07.342 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:07.340+0000 7fb632e99700 1 --2- 172.21.15.59:0/4269708763 >> v2:172.21.15.110:6800/115870 conn(0x7fb618068890 0x7fb61806ad40 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:07.343 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:07.340+0000 7fb632e99700 1 --2- 172.21.15.59:0/4269708763 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb62c0a5b70 0x7fb62c1407c0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:07.343 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:07.340+0000 7fb632e99700 1 --2- 172.21.15.59:0/4269708763 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb62c142290 0x7fb62c14e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:07.343 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:07.340+0000 7fb632e99700 1 --2- 172.21.15.59:0/4269708763 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb62c0a5f40 0x7fb62c13dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:07.343 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:07.340+0000 7fb632e99700 1 -- 172.21.15.59:0/4269708763 >> 172.21.15.59:0/4269708763 conn(0x7fb62c0631f0 msgr2=0x7fb62c0b0be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:08:07.343 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:07.341+0000 7fb632e99700 1 -- 172.21.15.59:0/4269708763 shutdown_connections 2023-09-30T02:08:07.343 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:07.341+0000 7fb632e99700 1 -- 172.21.15.59:0/4269708763 wait complete. 2023-09-30T02:08:07.343 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:07.341+0000 7fb632e99700 1 librados: shutdown 2023-09-30T02:08:10.357 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:08:10.498 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.497+0000 7f7ae6450700 10 monclient: get_monmap_and_config 2023-09-30T02:08:10.498 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.497+0000 7f7ae6450700 10 monclient: build_initial_monmap 2023-09-30T02:08:10.498 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.497+0000 7f7ae6450700 10 monclient: monmap: 2023-09-30T02:08:10.498 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:08:10.498 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:10.498 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:08:10.498974+0000 2023-09-30T02:08:10.498 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:08:10.498974+0000 2023-09-30T02:08:10.498 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:08:10.498 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:10.499 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:08:10.499 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:08:10.499 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:08:10.499 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:08:10.499 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:08:10.499 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:10.500 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.498+0000 7f7ae6450700 1 Processor -- start 2023-09-30T02:08:10.500 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.498+0000 7f7ae6450700 1 -- start start 2023-09-30T02:08:10.500 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.498+0000 7f7ae6450700 10 monclient: init 2023-09-30T02:08:10.500 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.499+0000 7f7ae6450700 10 monclient: _reopen_session rank -1 2023-09-30T02:08:10.500 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.499+0000 7f7ae6450700 10 monclient: _add_conns ranks=[4,1,0,2,3] 2023-09-30T02:08:10.501 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.499+0000 7f7ae6450700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f7ae00b16e0 0x7f7ae00a54e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:10.501 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.499+0000 7f7ae6450700 10 monclient(hunting): picked mon.noname-c con 0x7f7ae00b16e0 addr v2:172.21.15.110:3300/0 2023-09-30T02:08:10.501 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.499+0000 7f7ae6450700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7ae00a7650 addr v1:172.21.15.110:6789/0 2023-09-30T02:08:10.501 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.499+0000 7f7ae6450700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7ae013aef0 addr v1:172.21.15.59:6789/0 2023-09-30T02:08:10.501 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.499+0000 7f7ae6450700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ae00a5d40 con 0x7f7ae013aef0 2023-09-30T02:08:10.501 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.499+0000 7f7ae6450700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ae00b1ab0 con 0x7f7ae00a7650 2023-09-30T02:08:10.501 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.499+0000 7f7ae6450700 10 monclient(hunting): start opening mon connection 2023-09-30T02:08:10.501 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.499+0000 7f7ae6450700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ae00a5a20 con 0x7f7ae00b16e0 2023-09-30T02:08:10.501 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.499+0000 7f7ae6450700 10 monclient(hunting): _renew_subs 2023-09-30T02:08:10.501 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.499+0000 7f7ae6450700 10 monclient(hunting): authenticate will time out at 1935.802253s 2023-09-30T02:08:10.501 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.499+0000 7f7adffff700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f7ae00b16e0 0x7f7ae00a54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:10.501 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.499+0000 7f7ae49ed700 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f7ae013aef0 0x7f7ae013b390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:60654/0 (socket says 172.21.15.59:60654) 2023-09-30T02:08:10.502 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.499+0000 7f7ae49ed700 1 -- 172.21.15.59:0/4140745999 learned_addr learned my addr 172.21.15.59:0/4140745999 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:08:10.502 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.500+0000 7f7adffff700 10 monclient(hunting): get_auth_request con 0x7f7ae00b16e0 auth_method 0 2023-09-30T02:08:10.502 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.500+0000 7f7adffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:08:10.502 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.500+0000 7f7adffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:10.502 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.500+0000 7f7adffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:10.502 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.501+0000 7f7adffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:08:10.502 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.501+0000 7f7adffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:08:10.502 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.501+0000 7f7adffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:08:10.503 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.503+0000 7f7adeffd700 1 -- 172.21.15.59:0/4140745999 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 639510472 0 0) 0x7f7ae00a5d40 con 0x7f7ae013aef0 2023-09-30T02:08:10.503 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.503+0000 7f7adeffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:10.503 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.503+0000 7f7adeffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:10.504 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.503+0000 7f7adeffd700 10 monclient(hunting): my global_id is 4539 2023-09-30T02:08:10.504 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.503+0000 7f7adeffd700 1 -- 172.21.15.59:0/4140745999 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ac8003570 con 0x7f7ae013aef0 2023-09-30T02:08:10.505 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.504+0000 7f7adffff700 10 monclient(hunting): handle_auth_done global_id 4543 payload 274 2023-09-30T02:08:10.505 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.504+0000 7f7adffff700 1 -- 172.21.15.59:0/4140745999 >> v1:172.21.15.110:6789/0 conn(0x7f7ae00a7650 legacy=0x7f7ae00a7ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:08:10.505 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.504+0000 7f7adffff700 1 -- 172.21.15.59:0/4140745999 >> v1:172.21.15.59:6789/0 conn(0x7f7ae013aef0 legacy=0x7f7ae013b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:10.505 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.504+0000 7f7adffff700 10 monclient: _finish_hunting 0 2023-09-30T02:08:10.505 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.504+0000 7f7adffff700 1 monclient: found mon.noname-c 2023-09-30T02:08:10.505 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.504+0000 7f7adffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:08:10.505 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.504+0000 7f7adffff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3300/0 2023-09-30T02:08:10.505 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.504+0000 7f7adffff700 1 -- 172.21.15.59:0/4140745999 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ae00a6c70 con 0x7f7ae00b16e0 2023-09-30T02:08:10.509 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.509+0000 7f7adffff700 1 --2- 172.21.15.59:0/4140745999 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ae00b16e0 0x7f7ae00a54e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7ad00098e0 tx=0x7f7ad00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5e7931c2d38092df server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:08:10.514 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.513+0000 7f7adeffd700 1 -- 172.21.15.59:0/4140745999 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ad0011630 con 0x7f7ae00b16e0 2023-09-30T02:08:10.514 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.513+0000 7f7adeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:08:10.514 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.513+0000 7f7adeffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:08:10.514 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.513+0000 7f7adeffd700 10 monclient: dump: 2023-09-30T02:08:10.514 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:08:10.514 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:10.515 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:10.515 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:10.515 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:08:10.515 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:10.515 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:08:10.515 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:08:10.515 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:08:10.515 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:10.515 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.513+0000 7f7adeffd700 10 monclient: _finish_auth 0 2023-09-30T02:08:10.515 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.513+0000 7f7adeffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:08:10.515 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.513+0000 7f7adeffd700 1 -- 172.21.15.59:0/4140745999 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ad0011790 con 0x7f7ae00b16e0 2023-09-30T02:08:10.515 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.513+0000 7f7adeffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:08:10.516 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.513+0000 7f7adeffd700 1 -- 172.21.15.59:0/4140745999 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ad0011b30 con 0x7f7ae00b16e0 2023-09-30T02:08:10.516 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.513+0000 7f7adeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:08:10.516 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.513+0000 7f7adeffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:08:10.516 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.513+0000 7f7adeffd700 10 monclient: dump: 2023-09-30T02:08:10.516 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:08:10.516 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:10.516 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:10.516 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:10.516 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:08:10.516 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:10.516 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:08:10.516 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:08:10.517 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:08:10.517 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:10.517 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.513+0000 7f7ae6450700 5 monclient: authenticate success, global_id 4543 2023-09-30T02:08:10.517 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.513+0000 7f7ae6450700 10 monclient: get_monmap_and_config success 2023-09-30T02:08:10.517 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.514+0000 7f7ae6450700 4 set_mon_vals no callback set 2023-09-30T02:08:10.519 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.514+0000 7f7ae6450700 10 monclient: shutdown 2023-09-30T02:08:10.519 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.514+0000 7f7ae6450700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:08:10.519 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.514+0000 7f7ae6450700 1 -- 172.21.15.59:0/4140745999 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ae00b16e0 msgr2=0x7f7ae00a54e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:10.520 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.514+0000 7f7ae6450700 1 --2- 172.21.15.59:0/4140745999 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ae00b16e0 0x7f7ae00a54e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7ad00098e0 tx=0x7f7ad00091a0 comp rx=0 tx=0).stop 2023-09-30T02:08:10.520 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.514+0000 7f7ae6450700 1 -- 172.21.15.59:0/4140745999 shutdown_connections 2023-09-30T02:08:10.520 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.514+0000 7f7ae6450700 1 --2- 172.21.15.59:0/4140745999 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ae00b16e0 0x7f7ae00a54e0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:10.520 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.514+0000 7f7ae6450700 1 -- 172.21.15.59:0/4140745999 >> 172.21.15.59:0/4140745999 conn(0x7f7ae0068fe0 msgr2=0x7f7ae00631f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:08:10.520 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.514+0000 7f7ae6450700 1 -- 172.21.15.59:0/4140745999 shutdown_connections 2023-09-30T02:08:10.520 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.514+0000 7f7ae6450700 1 -- 172.21.15.59:0/4140745999 wait complete. 2023-09-30T02:08:10.520 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.515+0000 7f7ae6450700 10 monclient: build_initial_monmap 2023-09-30T02:08:10.520 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.515+0000 7f7ae6450700 10 monclient: monmap: 2023-09-30T02:08:10.520 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:08:10.520 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:08:10.520 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:08:10.521 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:08:10.521 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:08:10.521 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:08:10.521 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:08:10.521 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:08:10.521 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:08:10.521 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:10.521 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.515+0000 7f7ae6450700 1 librados: starting msgr at 2023-09-30T02:08:10.521 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.515+0000 7f7ae6450700 1 librados: starting objecter 2023-09-30T02:08:10.521 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.515+0000 7f7ae6450700 1 Processor -- start 2023-09-30T02:08:10.521 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.516+0000 7f7ae6450700 1 -- start start 2023-09-30T02:08:10.521 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.516+0000 7f7ae6450700 1 librados: setting wanted keys 2023-09-30T02:08:10.522 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.516+0000 7f7ae6450700 1 librados: calling monclient init 2023-09-30T02:08:10.522 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.516+0000 7f7ae6450700 10 monclient: init 2023-09-30T02:08:10.522 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.516+0000 7f7ae6450700 10 monclient: _reopen_session rank -1 2023-09-30T02:08:10.522 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.516+0000 7f7ae6450700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T02:08:10.522 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.516+0000 7f7ae6450700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ae00a7650 0x7f7ae013ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:10.522 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.516+0000 7f7ae6450700 10 monclient(hunting): picked mon.noname-b con 0x7f7ae00a7650 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:08:10.522 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.516+0000 7f7ae6450700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7ae00b16e0 0x7f7ae0140f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:10.522 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.516+0000 7f7ae6450700 10 monclient(hunting): picked mon.noname-c con 0x7f7ae00b16e0 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:08:10.522 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.516+0000 7f7ae6450700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ae013aef0 0x7f7ae0141480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:10.522 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.516+0000 7f7ae6450700 10 monclient(hunting): picked mon.noname-a con 0x7f7ae013aef0 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:08:10.522 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.516+0000 7f7ae6450700 10 monclient(hunting): start opening mon connection 2023-09-30T02:08:10.522 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.516+0000 7f7ae6450700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae00ab530 con 0x7f7ae013aef0 2023-09-30T02:08:10.523 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.516+0000 7f7ae6450700 10 monclient(hunting): start opening mon connection 2023-09-30T02:08:10.523 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.516+0000 7f7ae6450700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae0140b20 con 0x7f7ae00a7650 2023-09-30T02:08:10.523 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.516+0000 7f7ae6450700 10 monclient(hunting): start opening mon connection 2023-09-30T02:08:10.523 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.516+0000 7f7ae6450700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ae00ab010 con 0x7f7ae00b16e0 2023-09-30T02:08:10.523 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.516+0000 7f7ae6450700 10 monclient(hunting): _renew_subs 2023-09-30T02:08:10.523 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.516+0000 7f7ae6450700 10 monclient(hunting): authenticate will time out at 1935.819849s 2023-09-30T02:08:10.523 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.517+0000 7f7adffff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ae00a7650 0x7f7ae013ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:10.523 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.517+0000 7f7ae49ed700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ae013aef0 0x7f7ae0141480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:10.523 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.517+0000 7f7adf7fe700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7ae00b16e0 0x7f7ae0140f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:10.523 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.517+0000 7f7adffff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ae00a7650 0x7f7ae013ea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:47030/0 (socket says 172.21.15.59:47030) 2023-09-30T02:08:10.523 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.517+0000 7f7adffff700 1 -- 172.21.15.59:0/3977747903 learned_addr learned my addr 172.21.15.59:0/3977747903 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:08:10.523 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.517+0000 7f7adffff700 10 monclient(hunting): get_auth_request con 0x7f7ae00a7650 auth_method 0 2023-09-30T02:08:10.523 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.517+0000 7f7adffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:08:10.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.517+0000 7f7adffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:10.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.517+0000 7f7adffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:10.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.517+0000 7f7adf7fe700 10 monclient(hunting): get_auth_request con 0x7f7ae00b16e0 auth_method 0 2023-09-30T02:08:10.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.517+0000 7f7adf7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:08:10.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.517+0000 7f7adf7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:10.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.517+0000 7f7adf7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:10.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.517+0000 7f7ae49ed700 10 monclient(hunting): get_auth_request con 0x7f7ae013aef0 auth_method 0 2023-09-30T02:08:10.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.517+0000 7f7ae49ed700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:08:10.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.517+0000 7f7ae49ed700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:10.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.517+0000 7f7ae49ed700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:10.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.518+0000 7f7adffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:08:10.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.518+0000 7f7adffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:08:10.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.518+0000 7f7adffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:08:10.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.518+0000 7f7adf7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:08:10.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.518+0000 7f7adf7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:08:10.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.518+0000 7f7adf7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:08:10.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.518+0000 7f7ae49ed700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:08:10.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.518+0000 7f7ae49ed700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:08:10.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.518+0000 7f7ae49ed700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:08:10.526 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.522+0000 7f7adffff700 10 monclient(hunting): handle_auth_done global_id 4546 payload 824 2023-09-30T02:08:10.526 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.522+0000 7f7adffff700 1 -- 172.21.15.59:0/3977747903 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7ae00b16e0 msgr2=0x7f7ae0140f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:10.526 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.522+0000 7f7adffff700 1 --2- 172.21.15.59:0/3977747903 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7ae00b16e0 0x7f7ae0140f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:10.526 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.522+0000 7f7adffff700 1 -- 172.21.15.59:0/3977747903 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ae013aef0 msgr2=0x7f7ae0141480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:10.526 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.522+0000 7f7adffff700 1 --2- 172.21.15.59:0/3977747903 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ae013aef0 0x7f7ae0141480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:10.526 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.522+0000 7f7adffff700 10 monclient: _finish_hunting 0 2023-09-30T02:08:10.526 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.522+0000 7f7adffff700 1 monclient: found mon.noname-b 2023-09-30T02:08:10.526 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.522+0000 7f7adffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:08:10.526 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.522+0000 7f7adffff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:08:10.527 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.522+0000 7f7adffff700 1 -- 172.21.15.59:0/3977747903 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ad0009420 con 0x7f7ae00a7650 2023-09-30T02:08:10.527 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.524+0000 7f7adffff700 1 --2- 172.21.15.59:0/3977747903 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ae00a7650 0x7f7ae013ea90 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7ad00093f0 tx=0x7f7ad0016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:08:10.527 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.524+0000 7f7ac7fff700 10 client.?.objecter ms_handle_connect 0x7f7ae00a7650 2023-09-30T02:08:10.527 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.524+0000 7f7ac7fff700 10 client.?.objecter resend_mon_ops 2023-09-30T02:08:10.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.529+0000 7f7ac7fff700 1 -- 172.21.15.59:0/3977747903 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ad001bc70 con 0x7f7ae00a7650 2023-09-30T02:08:10.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.529+0000 7f7ac7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:08:10.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.529+0000 7f7ac7fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:08:10.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.529+0000 7f7ac7fff700 10 monclient: dump: 2023-09-30T02:08:10.530 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:08:10.530 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:10.530 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:10.530 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:10.530 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:08:10.530 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:10.531 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:08:10.531 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:08:10.531 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:08:10.531 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:10.531 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.529+0000 7f7ac7fff700 10 monclient: _finish_auth 0 2023-09-30T02:08:10.531 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.529+0000 7f7ac7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:08:10.531 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.529+0000 7f7ac7fff700 1 -- 172.21.15.59:0/3977747903 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ad0020070 con 0x7f7ae00a7650 2023-09-30T02:08:10.531 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.529+0000 7f7ae6450700 5 monclient: authenticate success, global_id 4546 2023-09-30T02:08:10.531 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.529+0000 7f7ac7fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:08:10.531 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.529+0000 7f7ac7fff700 1 -- 172.21.15.59:0/3977747903 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ad0027480 con 0x7f7ae00a7650 2023-09-30T02:08:10.531 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.529+0000 7f7ae6450700 10 monclient: _renew_subs 2023-09-30T02:08:10.531 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.529+0000 7f7ae6450700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:08:10.531 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.529+0000 7f7ae6450700 1 -- 172.21.15.59:0/3977747903 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ae013efd0 con 0x7f7ae00a7650 2023-09-30T02:08:10.532 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.529+0000 7f7ade7fc700 4 set_mon_vals no callback set 2023-09-30T02:08:10.532 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.529+0000 7f7ac7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:08:10.532 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.529+0000 7f7ac7fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:08:10.532 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.529+0000 7f7ac7fff700 10 monclient: dump: 2023-09-30T02:08:10.532 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:08:10.532 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:10.532 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:10.532 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:10.532 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:08:10.532 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:10.532 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:08:10.532 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:08:10.533 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:08:10.533 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:10.533 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.529+0000 7f7ae6450700 10 client.4546.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:08:10.533 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.529+0000 7f7ae6450700 10 monclient: _renew_subs 2023-09-30T02:08:10.533 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.529+0000 7f7ae6450700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:08:10.533 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.529+0000 7f7ae6450700 1 -- 172.21.15.59:0/3977747903 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ae01420f0 con 0x7f7ae00a7650 2023-09-30T02:08:10.533 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.529+0000 7f7ae6450700 1 librados: init done 2023-09-30T02:08:10.534 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.530+0000 7f7ae6450700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:08:10.534 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.530+0000 7f7ae6450700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:08:10.534 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.530+0000 7f7ae6450700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:08:10.534 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.530+0000 7f7ae6450700 1 -- 172.21.15.59:0/3977747903 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ae00ab230 con 0x7f7ae00a7650 2023-09-30T02:08:10.781 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.780+0000 7f7ac7fff700 1 -- 172.21.15.59:0/3977747903 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f7ad001c450 con 0x7f7ae00a7650 2023-09-30T02:08:10.782 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.781+0000 7f7ac7fff700 1 --2- 172.21.15.59:0/3977747903 >> v2:172.21.15.110:6800/115870 conn(0x7f7acc0689e0 0x7f7acc06ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:10.782 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.781+0000 7f7ac7fff700 1 -- 172.21.15.59:0/3977747903 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f7ad0022070 con 0x7f7ae00a7650 2023-09-30T02:08:10.782 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.781+0000 7f7ac7fff700 10 client.4546.objecter ms_dispatch 0x7f7ae0001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:08:10.782 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.781+0000 7f7ac7fff700 3 client.4546.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:08:10.782 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.781+0000 7f7ac7fff700 3 client.4546.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:08:10.782 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.781+0000 7f7adf7fe700 1 --2- 172.21.15.59:0/3977747903 >> v2:172.21.15.110:6800/115870 conn(0x7f7acc0689e0 0x7f7acc06ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:10.782 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.781+0000 7f7ac7fff700 20 client.4546.objecter dump_active .. 0 homeless 2023-09-30T02:08:10.782 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.781+0000 7f7adf7fe700 10 monclient: get_auth_request con 0x7f7acc0689e0 auth_method 0 2023-09-30T02:08:10.783 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.782+0000 7f7ac7fff700 10 client.4546.objecter ms_handle_connect 0x7f7acc0689e0 2023-09-30T02:08:10.783 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.782+0000 7f7adf7fe700 1 --2- 172.21.15.59:0/3977747903 >> v2:172.21.15.110:6800/115870 conn(0x7f7acc0689e0 0x7f7acc06ae90 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f7ae013c0d0 tx=0x7f7ad4007ea0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:08:10.790 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.790+0000 7f7ac7fff700 1 -- 172.21.15.59:0/3977747903 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7ad0053970 con 0x7f7ae00a7650 2023-09-30T02:08:10.791 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.790+0000 7f7ac7fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:08:10.791 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:10.790+0000 7f7ac7fff700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:08:11.075 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:11.074+0000 7f7ae6450700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:08:11.075 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:11.074+0000 7f7ae6450700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:08:11.075 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:11.074+0000 7f7ae6450700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:08:11.075 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:11.074+0000 7f7ae6450700 1 -- 172.21.15.59:0/3977747903 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f7ae0072b30 con 0x7f7ae00a7650 2023-09-30T02:08:11.082 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:11.082+0000 7f7ac7fff700 1 -- 172.21.15.59:0/3977747903 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f7ad0056f90 con 0x7f7ae00a7650 2023-09-30T02:08:11.082 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:11.082+0000 7f7ac7fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:08:11.082 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:11.082+0000 7f7ac7fff700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:08:11.082 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:08:11.082 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:08:11.084 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:11.084+0000 7f7ae6450700 10 librados: watch_flush enter 2023-09-30T02:08:11.085 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:11.084+0000 7f7ae6450700 10 librados: watch_flush exit 2023-09-30T02:08:11.085 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:11.084+0000 7f7ae6450700 20 client.4546.objecter shutdown clearing up homeless session... 2023-09-30T02:08:11.085 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:11.084+0000 7f7ae6450700 10 client.4546.objecter successfully canceled tick 2023-09-30T02:08:11.085 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:11.084+0000 7f7ae6450700 1 -- 172.21.15.59:0/3977747903 >> v2:172.21.15.110:6800/115870 conn(0x7f7acc0689e0 msgr2=0x7f7acc06ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:11.085 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:11.084+0000 7f7ae6450700 1 --2- 172.21.15.59:0/3977747903 >> v2:172.21.15.110:6800/115870 conn(0x7f7acc0689e0 0x7f7acc06ae90 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f7ae013c0d0 tx=0x7f7ad4007ea0 comp rx=0 tx=0).stop 2023-09-30T02:08:11.085 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:11.084+0000 7f7ae6450700 10 monclient: shutdown 2023-09-30T02:08:11.085 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:11.084+0000 7f7ae6450700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:08:11.085 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:11.084+0000 7f7ae6450700 1 -- 172.21.15.59:0/3977747903 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ae00a7650 msgr2=0x7f7ae013ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:11.085 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:11.084+0000 7f7ae6450700 1 --2- 172.21.15.59:0/3977747903 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ae00a7650 0x7f7ae013ea90 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7ad00093f0 tx=0x7f7ad0016000 comp rx=0 tx=0).stop 2023-09-30T02:08:11.086 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:11.084+0000 7f7ae6450700 1 -- 172.21.15.59:0/3977747903 shutdown_connections 2023-09-30T02:08:11.086 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:11.084+0000 7f7ae6450700 1 --2- 172.21.15.59:0/3977747903 >> v2:172.21.15.110:6800/115870 conn(0x7f7acc0689e0 0x7f7acc06ae90 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:11.086 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:11.084+0000 7f7ae6450700 1 --2- 172.21.15.59:0/3977747903 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7ae00a7650 0x7f7ae013ea90 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:11.086 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:11.084+0000 7f7ae6450700 1 --2- 172.21.15.59:0/3977747903 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7ae013aef0 0x7f7ae0141480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:11.086 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:11.084+0000 7f7ae6450700 1 --2- 172.21.15.59:0/3977747903 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7ae00b16e0 0x7f7ae0140f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:11.086 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:11.084+0000 7f7ae6450700 1 -- 172.21.15.59:0/3977747903 >> 172.21.15.59:0/3977747903 conn(0x7f7ae0068fe0 msgr2=0x7f7ae00b28d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:08:11.086 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:11.084+0000 7f7ae6450700 1 -- 172.21.15.59:0/3977747903 shutdown_connections 2023-09-30T02:08:11.086 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:11.084+0000 7f7ae6450700 1 -- 172.21.15.59:0/3977747903 wait complete. 2023-09-30T02:08:11.086 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:11.084+0000 7f7ae6450700 1 librados: shutdown 2023-09-30T02:08:13.688 INFO:tasks.ceph.mon.a.smithi059.stderr:==00:00:03:23.554 116256== Warning: unimplemented fcntl command: 1036 2023-09-30T02:08:13.756 INFO:tasks.ceph.mon.a.smithi059.stderr:==00:00:03:23.622 116256== Warning: unimplemented fcntl command: 1036 2023-09-30T02:08:13.787 INFO:tasks.ceph.mon.a.smithi059.stderr:==00:00:03:23.652 116256== Warning: unimplemented fcntl command: 1036 2023-09-30T02:08:13.807 INFO:tasks.ceph.mon.b.smithi110.stderr:==00:00:03:23.654 115869== Warning: unimplemented fcntl command: 1036 2023-09-30T02:08:13.811 INFO:tasks.ceph.mon.b.smithi110.stderr:==00:00:03:23.659 115869== Warning: unimplemented fcntl command: 1036 2023-09-30T02:08:13.821 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:03:23.687 116257== Warning: unimplemented fcntl command: 1036 2023-09-30T02:08:13.829 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:03:23.694 116257== Warning: unimplemented fcntl command: 1036 2023-09-30T02:08:13.833 INFO:tasks.ceph.mon.b.smithi110.stderr:==00:00:03:23.680 115869== Warning: unimplemented fcntl command: 1036 2023-09-30T02:08:13.864 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:03:23.729 116257== Warning: unimplemented fcntl command: 1036 2023-09-30T02:08:14.098 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:08:14.232 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.232+0000 7f12a33c2700 10 monclient: get_monmap_and_config 2023-09-30T02:08:14.232 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.232+0000 7f12a33c2700 10 monclient: build_initial_monmap 2023-09-30T02:08:14.233 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.232+0000 7f12a33c2700 10 monclient: monmap: 2023-09-30T02:08:14.233 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:08:14.233 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:14.233 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:08:14.233727+0000 2023-09-30T02:08:14.233 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:08:14.233727+0000 2023-09-30T02:08:14.233 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:08:14.233 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:14.233 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:08:14.233 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:08:14.233 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:08:14.233 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:08:14.234 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:08:14.234 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:14.235 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.232+0000 7f12a33c2700 1 Processor -- start 2023-09-30T02:08:14.236 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.233+0000 7f12a33c2700 1 -- start start 2023-09-30T02:08:14.236 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.233+0000 7f12a33c2700 10 monclient: init 2023-09-30T02:08:14.236 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.233+0000 7f12a33c2700 10 monclient: _reopen_session rank -1 2023-09-30T02:08:14.236 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.233+0000 7f12a33c2700 10 monclient: _add_conns ranks=[1,3,4,0,2] 2023-09-30T02:08:14.236 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.233+0000 7f12a33c2700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f129c0a5b70 addr v1:172.21.15.110:6789/0 2023-09-30T02:08:14.236 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.233+0000 7f12a33c2700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f129c142290 0x7f129c0b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:14.236 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.233+0000 7f12a33c2700 10 monclient(hunting): picked mon.noname-a con 0x7f129c142290 addr v2:172.21.15.59:3300/0 2023-09-30T02:08:14.236 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.233+0000 7f12a33c2700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f129c0a5f40 0x7f129c13af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:14.236 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.233+0000 7f12a33c2700 10 monclient(hunting): picked mon.noname-c con 0x7f129c0a5f40 addr v2:172.21.15.110:3300/0 2023-09-30T02:08:14.236 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.233+0000 7f12a33c2700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f129c0a75a0 con 0x7f129c0a5b70 2023-09-30T02:08:14.237 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.233+0000 7f12a33c2700 10 monclient(hunting): start opening mon connection 2023-09-30T02:08:14.237 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.233+0000 7f12a33c2700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f129c0a6360 con 0x7f129c142290 2023-09-30T02:08:14.237 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.233+0000 7f12a33c2700 10 monclient(hunting): start opening mon connection 2023-09-30T02:08:14.237 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.233+0000 7f12a33c2700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f129c0af3f0 con 0x7f129c0a5f40 2023-09-30T02:08:14.237 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.233+0000 7f12a33c2700 10 monclient(hunting): _renew_subs 2023-09-30T02:08:14.237 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.234+0000 7f12a33c2700 10 monclient(hunting): authenticate will time out at 1939.537007s 2023-09-30T02:08:14.237 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.234+0000 7f12a195f700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f129c0a5f40 0x7f129c13af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:14.237 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.235+0000 7f12a195f700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f129c0a5f40 0x7f129c13af30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:47044/0 (socket says 172.21.15.59:47044) 2023-09-30T02:08:14.237 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.235+0000 7f12a195f700 1 -- 172.21.15.59:0/3489992869 learned_addr learned my addr 172.21.15.59:0/3489992869 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:08:14.237 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.235+0000 7f12a195f700 10 monclient(hunting): get_auth_request con 0x7f129c0a5f40 auth_method 0 2023-09-30T02:08:14.237 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.235+0000 7f12a195f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:08:14.237 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.235+0000 7f12a195f700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:14.237 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.235+0000 7f12a195f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:14.238 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.235+0000 7f12a195f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:08:14.238 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.235+0000 7f12a195f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:08:14.238 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.235+0000 7f12a195f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:08:14.239 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.238+0000 7f12a195f700 10 monclient(hunting): handle_auth_done global_id 4549 payload 274 2023-09-30T02:08:14.239 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.238+0000 7f12a195f700 1 -- 172.21.15.59:0/3489992869 >> v2:172.21.15.59:3300/0 conn(0x7f129c142290 msgr2=0x7f129c0b1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:08:14.239 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.238+0000 7f12a195f700 1 --2- 172.21.15.59:0/3489992869 >> v2:172.21.15.59:3300/0 conn(0x7f129c142290 0x7f129c0b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:14.239 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.238+0000 7f12a195f700 1 -- 172.21.15.59:0/3489992869 >> v1:172.21.15.110:6789/0 conn(0x7f129c0a5b70 legacy=0x7f129c141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:08:14.239 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.238+0000 7f12a195f700 10 monclient: _finish_hunting 0 2023-09-30T02:08:14.239 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.238+0000 7f12a195f700 1 monclient: found mon.noname-c 2023-09-30T02:08:14.239 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.238+0000 7f12a195f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:08:14.239 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.238+0000 7f12a195f700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3300/0 2023-09-30T02:08:14.239 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.238+0000 7f12a195f700 1 -- 172.21.15.59:0/3489992869 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f129c13b470 con 0x7f129c0a5f40 2023-09-30T02:08:14.241 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.241+0000 7f12a195f700 1 --2- 172.21.15.59:0/3489992869 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f129c0a5f40 0x7f129c13af30 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f129c0b0640 tx=0x7f1294009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=11aff33626514367 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:08:14.245 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.245+0000 7f1293fff700 1 -- 172.21.15.59:0/3489992869 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f129400e070 con 0x7f129c0a5f40 2023-09-30T02:08:14.245 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.245+0000 7f1293fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:08:14.245 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.245+0000 7f1293fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:08:14.246 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.245+0000 7f1293fff700 10 monclient: dump: 2023-09-30T02:08:14.246 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:08:14.246 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:14.246 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:14.246 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:14.246 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:08:14.246 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:14.246 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:08:14.246 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:08:14.246 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:08:14.246 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:14.246 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.245+0000 7f1293fff700 10 monclient: _finish_auth 0 2023-09-30T02:08:14.247 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.245+0000 7f1293fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:08:14.247 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.245+0000 7f12a33c2700 5 monclient: authenticate success, global_id 4549 2023-09-30T02:08:14.247 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.245+0000 7f12a33c2700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:08:14.250 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.249+0000 7f1293fff700 1 -- 172.21.15.59:0/3489992869 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f129400ba10 con 0x7f129c0a5f40 2023-09-30T02:08:14.250 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.249+0000 7f1293fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:08:14.250 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.249+0000 7f1293fff700 1 -- 172.21.15.59:0/3489992869 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12940092e0 con 0x7f129c0a5f40 2023-09-30T02:08:14.250 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.250+0000 7f1293fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:08:14.250 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.250+0000 7f1293fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:08:14.250 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.250+0000 7f1293fff700 10 monclient: dump: 2023-09-30T02:08:14.250 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:08:14.250 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:14.251 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:14.251 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:14.251 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:08:14.251 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:14.251 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:08:14.251 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:08:14.251 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:08:14.251 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:14.251 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.250+0000 7f12a33c2700 10 monclient: get_monmap_and_config success 2023-09-30T02:08:14.251 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.250+0000 7f12a33c2700 4 set_mon_vals no callback set 2023-09-30T02:08:14.252 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.250+0000 7f12a33c2700 10 monclient: shutdown 2023-09-30T02:08:14.253 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.250+0000 7f12a33c2700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:08:14.253 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.250+0000 7f12a33c2700 1 -- 172.21.15.59:0/3489992869 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f129c0a5f40 msgr2=0x7f129c13af30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:14.253 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.250+0000 7f12a33c2700 1 --2- 172.21.15.59:0/3489992869 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f129c0a5f40 0x7f129c13af30 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f129c0b0640 tx=0x7f1294009840 comp rx=0 tx=0).stop 2023-09-30T02:08:14.253 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.250+0000 7f12a33c2700 1 -- 172.21.15.59:0/3489992869 shutdown_connections 2023-09-30T02:08:14.253 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.250+0000 7f12a33c2700 1 --2- 172.21.15.59:0/3489992869 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f129c0a5f40 0x7f129c13af30 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:14.253 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.250+0000 7f12a33c2700 1 --2- 172.21.15.59:0/3489992869 >> v2:172.21.15.59:3300/0 conn(0x7f129c142290 0x7f129c0b1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:14.253 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.250+0000 7f12a33c2700 1 -- 172.21.15.59:0/3489992869 >> 172.21.15.59:0/3489992869 conn(0x7f129c0631f0 msgr2=0x7f129c0635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:08:14.253 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.251+0000 7f12a33c2700 1 -- 172.21.15.59:0/3489992869 shutdown_connections 2023-09-30T02:08:14.253 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.251+0000 7f12a33c2700 1 -- 172.21.15.59:0/3489992869 wait complete. 2023-09-30T02:08:14.253 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.251+0000 7f12a33c2700 10 monclient: build_initial_monmap 2023-09-30T02:08:14.254 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.251+0000 7f12a33c2700 10 monclient: monmap: 2023-09-30T02:08:14.254 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:08:14.254 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:08:14.254 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:08:14.254 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:08:14.254 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:08:14.254 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:08:14.254 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:08:14.254 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:08:14.254 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:08:14.254 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:14.255 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.252+0000 7f12a33c2700 1 librados: starting msgr at 2023-09-30T02:08:14.255 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.252+0000 7f12a33c2700 1 librados: starting objecter 2023-09-30T02:08:14.255 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.252+0000 7f12a33c2700 1 Processor -- start 2023-09-30T02:08:14.255 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.252+0000 7f12a33c2700 1 -- start start 2023-09-30T02:08:14.255 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.252+0000 7f12a33c2700 1 librados: setting wanted keys 2023-09-30T02:08:14.255 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.252+0000 7f12a33c2700 1 librados: calling monclient init 2023-09-30T02:08:14.255 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.252+0000 7f12a33c2700 10 monclient: init 2023-09-30T02:08:14.257 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.253+0000 7f12a33c2700 10 monclient: _reopen_session rank -1 2023-09-30T02:08:14.257 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.253+0000 7f12a33c2700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T02:08:14.257 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.253+0000 7f12a33c2700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f129c0a5b70 0x7f129c0a7d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:14.257 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.253+0000 7f12a33c2700 10 monclient(hunting): picked mon.noname-b con 0x7f129c0a5b70 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:08:14.257 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.253+0000 7f12a33c2700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f129c0a5f40 0x7f129c13e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:14.258 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.253+0000 7f12a33c2700 10 monclient(hunting): picked mon.noname-a con 0x7f129c0a5f40 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:08:14.258 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.253+0000 7f12a33c2700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f129c142290 0x7f129c137880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:14.258 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.253+0000 7f12a33c2700 10 monclient(hunting): picked mon.noname-c con 0x7f129c142290 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:08:14.258 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.253+0000 7f12a33c2700 10 monclient(hunting): start opening mon connection 2023-09-30T02:08:14.258 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.253+0000 7f12a33c2700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f129c0a91c0 con 0x7f129c0a5f40 2023-09-30T02:08:14.258 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.253+0000 7f12a33c2700 10 monclient(hunting): start opening mon connection 2023-09-30T02:08:14.258 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.253+0000 7f12a33c2700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f129c13c030 con 0x7f129c0a5b70 2023-09-30T02:08:14.258 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.253+0000 7f12a33c2700 10 monclient(hunting): start opening mon connection 2023-09-30T02:08:14.258 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.253+0000 7f12a33c2700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f129c0a82b0 con 0x7f129c142290 2023-09-30T02:08:14.258 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.253+0000 7f12a33c2700 10 monclient(hunting): _renew_subs 2023-09-30T02:08:14.258 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.253+0000 7f12a33c2700 10 monclient(hunting): authenticate will time out at 1939.556175s 2023-09-30T02:08:14.258 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.253+0000 7f12a195f700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f129c142290 0x7f129c137880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:14.259 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.253+0000 7f12a115e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f129c0a5b70 0x7f129c0a7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:14.259 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.254+0000 7f12a115e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f129c0a5b70 0x7f129c0a7d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:47056/0 (socket says 172.21.15.59:47056) 2023-09-30T02:08:14.259 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.254+0000 7f12a115e700 1 -- 172.21.15.59:0/4091028415 learned_addr learned my addr 172.21.15.59:0/4091028415 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:08:14.259 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.254+0000 7f12a115e700 10 monclient(hunting): get_auth_request con 0x7f129c0a5b70 auth_method 0 2023-09-30T02:08:14.259 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.254+0000 7f12a115e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:08:14.259 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.254+0000 7f12a115e700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:14.259 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.254+0000 7f12a115e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:14.259 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.254+0000 7f12a195f700 10 monclient(hunting): get_auth_request con 0x7f129c142290 auth_method 0 2023-09-30T02:08:14.259 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.254+0000 7f12a195f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:08:14.259 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.254+0000 7f12a195f700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:14.259 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.254+0000 7f12a195f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:14.259 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.254+0000 7f12a115e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:08:14.260 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.254+0000 7f12a115e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:08:14.260 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.254+0000 7f12a115e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:08:14.260 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.255+0000 7f12a195f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:08:14.260 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.255+0000 7f12a195f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:08:14.260 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.255+0000 7f12a195f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:08:14.260 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.255+0000 7f12a095d700 1 --2- 172.21.15.59:0/4091028415 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f129c0a5f40 0x7f129c13e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:14.260 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.255+0000 7f12a095d700 10 monclient(hunting): get_auth_request con 0x7f129c0a5f40 auth_method 0 2023-09-30T02:08:14.260 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.255+0000 7f12a095d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:08:14.260 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.255+0000 7f12a095d700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:14.260 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.255+0000 7f12a095d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:14.260 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.256+0000 7f12a095d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:08:14.260 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.256+0000 7f12a095d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:08:14.261 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.256+0000 7f12a095d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:08:14.261 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.258+0000 7f12a115e700 10 monclient(hunting): handle_auth_done global_id 4552 payload 824 2023-09-30T02:08:14.262 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.258+0000 7f12a115e700 1 -- 172.21.15.59:0/4091028415 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f129c142290 msgr2=0x7f129c137880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:14.262 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.258+0000 7f12a115e700 1 --2- 172.21.15.59:0/4091028415 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f129c142290 0x7f129c137880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:14.262 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.258+0000 7f12a115e700 1 -- 172.21.15.59:0/4091028415 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f129c0a5f40 msgr2=0x7f129c13e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:14.262 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.258+0000 7f12a115e700 1 --2- 172.21.15.59:0/4091028415 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f129c0a5f40 0x7f129c13e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:14.262 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.258+0000 7f12a115e700 10 monclient: _finish_hunting 0 2023-09-30T02:08:14.262 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.258+0000 7f12a115e700 1 monclient: found mon.noname-b 2023-09-30T02:08:14.262 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.258+0000 7f12a115e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:08:14.262 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.258+0000 7f12a115e700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:08:14.262 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.258+0000 7f12a115e700 1 -- 172.21.15.59:0/4091028415 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1294009ae0 con 0x7f129c0a5b70 2023-09-30T02:08:14.263 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.260+0000 7f12a115e700 1 --2- 172.21.15.59:0/4091028415 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f129c0a5b70 0x7f129c0a7d70 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f129800c5d0 tx=0x7f1298007f60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:08:14.263 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.260+0000 7f12917fa700 10 client.?.objecter ms_handle_connect 0x7f129c0a5b70 2023-09-30T02:08:14.263 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.260+0000 7f12917fa700 10 client.?.objecter resend_mon_ops 2023-09-30T02:08:14.264 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.262+0000 7f12917fa700 1 -- 172.21.15.59:0/4091028415 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1298011590 con 0x7f129c0a5b70 2023-09-30T02:08:14.264 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.262+0000 7f12917fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:08:14.264 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.262+0000 7f12917fa700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:08:14.264 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.262+0000 7f12917fa700 10 monclient: dump: 2023-09-30T02:08:14.264 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:08:14.264 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:14.264 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:14.264 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:14.265 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:08:14.265 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:14.265 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:08:14.265 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:08:14.265 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:08:14.265 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:14.265 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.262+0000 7f12917fa700 10 monclient: _finish_auth 0 2023-09-30T02:08:14.265 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.262+0000 7f12917fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:08:14.265 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.262+0000 7f12a33c2700 5 monclient: authenticate success, global_id 4552 2023-09-30T02:08:14.265 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.262+0000 7f12a33c2700 10 monclient: _renew_subs 2023-09-30T02:08:14.265 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.262+0000 7f12a33c2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:08:14.265 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.262+0000 7f12a33c2700 1 -- 172.21.15.59:0/4091028415 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f129c13edb0 con 0x7f129c0a5b70 2023-09-30T02:08:14.265 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.262+0000 7f12a33c2700 10 client.4552.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:08:14.266 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.262+0000 7f12a33c2700 10 monclient: _renew_subs 2023-09-30T02:08:14.266 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.262+0000 7f12a33c2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:08:14.266 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.262+0000 7f12a33c2700 1 -- 172.21.15.59:0/4091028415 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f129c137f80 con 0x7f129c0a5b70 2023-09-30T02:08:14.266 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.262+0000 7f12a33c2700 1 librados: init done 2023-09-30T02:08:14.266 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.263+0000 7f12a33c2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:08:14.266 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.263+0000 7f12a33c2700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:08:14.266 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.263+0000 7f12a33c2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:08:14.266 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.263+0000 7f12a33c2700 1 -- 172.21.15.59:0/4091028415 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f129c13c270 con 0x7f129c0a5b70 2023-09-30T02:08:14.267 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.264+0000 7f12917fa700 1 -- 172.21.15.59:0/4091028415 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12980021c0 con 0x7f129c0a5b70 2023-09-30T02:08:14.267 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.264+0000 7f12917fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:08:14.267 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.264+0000 7f12917fa700 1 -- 172.21.15.59:0/4091028415 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1298011900 con 0x7f129c0a5b70 2023-09-30T02:08:14.267 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.264+0000 7f12917fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:08:14.267 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.264+0000 7f12917fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:08:14.267 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.264+0000 7f12917fa700 10 monclient: dump: 2023-09-30T02:08:14.267 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:08:14.267 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:14.267 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:14.267 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:14.268 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:08:14.268 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:14.268 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:08:14.268 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:08:14.268 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:08:14.268 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:14.268 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.264+0000 7f12937fe700 4 set_mon_vals no callback set 2023-09-30T02:08:14.475 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.475+0000 7f12917fa700 1 -- 172.21.15.59:0/4091028415 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f1298011b20 con 0x7f129c0a5b70 2023-09-30T02:08:14.476 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.475+0000 7f12917fa700 1 --2- 172.21.15.59:0/4091028415 >> v2:172.21.15.110:6800/115870 conn(0x7f1280068890 0x7f128006ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:14.476 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.475+0000 7f12a095d700 1 --2- 172.21.15.59:0/4091028415 >> v2:172.21.15.110:6800/115870 conn(0x7f1280068890 0x7f128006ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:14.476 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.475+0000 7f12a095d700 10 monclient: get_auth_request con 0x7f1280068890 auth_method 0 2023-09-30T02:08:14.476 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.476+0000 7f12a095d700 1 --2- 172.21.15.59:0/4091028415 >> v2:172.21.15.110:6800/115870 conn(0x7f1280068890 0x7f128006ad40 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f128c009b40 tx=0x7f128c0090f0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:08:14.476 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.476+0000 7f12917fa700 10 client.4552.objecter ms_handle_connect 0x7f1280068890 2023-09-30T02:08:14.533 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.533+0000 7f12917fa700 1 -- 172.21.15.59:0/4091028415 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f1298019310 con 0x7f129c0a5b70 2023-09-30T02:08:14.533 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.533+0000 7f12917fa700 10 client.4552.objecter ms_dispatch 0x7f129c001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:08:14.533 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.533+0000 7f12917fa700 3 client.4552.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:08:14.534 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.533+0000 7f12917fa700 3 client.4552.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:08:14.534 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.533+0000 7f12917fa700 20 client.4552.objecter dump_active .. 0 homeless 2023-09-30T02:08:14.542 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.542+0000 7f12917fa700 1 -- 172.21.15.59:0/4091028415 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1298051930 con 0x7f129c0a5b70 2023-09-30T02:08:14.542 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.542+0000 7f12917fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:08:14.543 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.542+0000 7f12917fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:08:14.798 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.798+0000 7f12a33c2700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:08:14.799 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.798+0000 7f12a33c2700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:08:14.799 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.798+0000 7f12a33c2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:08:14.799 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.798+0000 7f12a33c2700 1 -- 172.21.15.59:0/4091028415 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f129c064a00 con 0x7f129c0a5b70 2023-09-30T02:08:14.803 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.803+0000 7f12917fa700 1 -- 172.21.15.59:0/4091028415 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f1298054f50 con 0x7f129c0a5b70 2023-09-30T02:08:14.803 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.803+0000 7f12917fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:08:14.804 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.803+0000 7f12917fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:08:14.804 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:08:14.804 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:08:14.806 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.805+0000 7f12a33c2700 10 librados: watch_flush enter 2023-09-30T02:08:14.806 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.805+0000 7f12a33c2700 10 librados: watch_flush exit 2023-09-30T02:08:14.806 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.805+0000 7f12a33c2700 20 client.4552.objecter shutdown clearing up homeless session... 2023-09-30T02:08:14.807 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.805+0000 7f12a33c2700 10 client.4552.objecter successfully canceled tick 2023-09-30T02:08:14.807 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.805+0000 7f12a33c2700 1 -- 172.21.15.59:0/4091028415 >> v2:172.21.15.110:6800/115870 conn(0x7f1280068890 msgr2=0x7f128006ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:14.807 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.805+0000 7f12a33c2700 1 --2- 172.21.15.59:0/4091028415 >> v2:172.21.15.110:6800/115870 conn(0x7f1280068890 0x7f128006ad40 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f128c009b40 tx=0x7f128c0090f0 comp rx=0 tx=0).stop 2023-09-30T02:08:14.807 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.805+0000 7f12a33c2700 10 monclient: shutdown 2023-09-30T02:08:14.807 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.805+0000 7f12a33c2700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:08:14.807 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.805+0000 7f12a33c2700 1 -- 172.21.15.59:0/4091028415 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f129c0a5b70 msgr2=0x7f129c0a7d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:14.807 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.805+0000 7f12a33c2700 1 --2- 172.21.15.59:0/4091028415 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f129c0a5b70 0x7f129c0a7d70 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f129800c5d0 tx=0x7f1298007f60 comp rx=0 tx=0).stop 2023-09-30T02:08:14.807 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.805+0000 7f12a33c2700 1 -- 172.21.15.59:0/4091028415 shutdown_connections 2023-09-30T02:08:14.807 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.805+0000 7f12a33c2700 1 --2- 172.21.15.59:0/4091028415 >> v2:172.21.15.110:6800/115870 conn(0x7f1280068890 0x7f128006ad40 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:14.808 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.806+0000 7f12a33c2700 1 --2- 172.21.15.59:0/4091028415 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f129c0a5b70 0x7f129c0a7d70 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:14.808 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.806+0000 7f12a33c2700 1 --2- 172.21.15.59:0/4091028415 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f129c142290 0x7f129c137880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:14.808 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.806+0000 7f12a33c2700 1 --2- 172.21.15.59:0/4091028415 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f129c0a5f40 0x7f129c13e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:14.808 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.806+0000 7f12a33c2700 1 -- 172.21.15.59:0/4091028415 >> 172.21.15.59:0/4091028415 conn(0x7f129c0631f0 msgr2=0x7f129c0b3360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:08:14.808 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.806+0000 7f12a33c2700 1 -- 172.21.15.59:0/4091028415 shutdown_connections 2023-09-30T02:08:14.808 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.806+0000 7f12a33c2700 1 -- 172.21.15.59:0/4091028415 wait complete. 2023-09-30T02:08:14.808 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:14.806+0000 7f12a33c2700 1 librados: shutdown 2023-09-30T02:08:17.820 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:08:17.963 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.958+0000 7fcc36cf4700 10 monclient: get_monmap_and_config 2023-09-30T02:08:17.964 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.958+0000 7fcc36cf4700 10 monclient: build_initial_monmap 2023-09-30T02:08:17.964 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.958+0000 7fcc36cf4700 10 monclient: monmap: 2023-09-30T02:08:17.964 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:08:17.964 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:17.964 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:08:17.959860+0000 2023-09-30T02:08:17.964 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:08:17.959860+0000 2023-09-30T02:08:17.964 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:08:17.964 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:17.964 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:08:17.964 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:08:17.964 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:08:17.964 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:08:17.965 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:08:17.965 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:17.965 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.959+0000 7fcc36cf4700 1 Processor -- start 2023-09-30T02:08:17.965 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.959+0000 7fcc36cf4700 1 -- start start 2023-09-30T02:08:17.965 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.959+0000 7fcc36cf4700 10 monclient: init 2023-09-30T02:08:17.965 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.959+0000 7fcc36cf4700 10 monclient: _reopen_session rank -1 2023-09-30T02:08:17.965 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.959+0000 7fcc36cf4700 10 monclient: _add_conns ranks=[2,3,1,0,4] 2023-09-30T02:08:17.965 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.959+0000 7fcc36cf4700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fcc300a5b70 0x7fcc300b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:17.965 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.959+0000 7fcc36cf4700 10 monclient(hunting): picked mon.noname-b con 0x7fcc300a5b70 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:08:17.965 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.959+0000 7fcc36cf4700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fcc300a5f40 0x7fcc30141b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:17.966 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.959+0000 7fcc36cf4700 10 monclient(hunting): picked mon.noname-a con 0x7fcc300a5f40 addr v2:172.21.15.59:3300/0 2023-09-30T02:08:17.966 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.960+0000 7fcc36cf4700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcc301420c0 addr v1:172.21.15.110:6789/0 2023-09-30T02:08:17.966 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.960+0000 7fcc36cf4700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc300a63b0 con 0x7fcc301420c0 2023-09-30T02:08:17.966 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.960+0000 7fcc36cf4700 10 monclient(hunting): start opening mon connection 2023-09-30T02:08:17.966 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.960+0000 7fcc36cf4700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc300a75a0 con 0x7fcc300a5f40 2023-09-30T02:08:17.966 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.960+0000 7fcc36cf4700 10 monclient(hunting): start opening mon connection 2023-09-30T02:08:17.966 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.960+0000 7fcc36cf4700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc300af3f0 con 0x7fcc300a5b70 2023-09-30T02:08:17.966 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.960+0000 7fcc36cf4700 10 monclient(hunting): _renew_subs 2023-09-30T02:08:17.966 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.960+0000 7fcc36cf4700 10 monclient(hunting): authenticate will time out at 1943.263040s 2023-09-30T02:08:17.966 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.960+0000 7fcc34a90700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fcc300a5b70 0x7fcc300b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:17.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.960+0000 7fcc2ffff700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fcc300a5f40 0x7fcc30141b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:17.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.960+0000 7fcc35291700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fcc301420c0 0x7fcc3013e230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.59:33418/0 (socket says 172.21.15.59:33418) 2023-09-30T02:08:17.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.960+0000 7fcc35291700 1 -- 172.21.15.59:0/1895631132 learned_addr learned my addr 172.21.15.59:0/1895631132 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:08:17.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.961+0000 7fcc34a90700 10 monclient(hunting): get_auth_request con 0x7fcc300a5b70 auth_method 0 2023-09-30T02:08:17.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.961+0000 7fcc34a90700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:08:17.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.961+0000 7fcc34a90700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:17.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.961+0000 7fcc34a90700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:17.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.961+0000 7fcc2ffff700 10 monclient(hunting): get_auth_request con 0x7fcc300a5f40 auth_method 0 2023-09-30T02:08:17.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.961+0000 7fcc2ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:08:17.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.961+0000 7fcc2ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:17.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.961+0000 7fcc2ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:17.968 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.961+0000 7fcc2ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:08:17.968 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.961+0000 7fcc2ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:08:17.968 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.961+0000 7fcc2ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:08:17.968 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.962+0000 7fcc34a90700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:08:17.968 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.962+0000 7fcc34a90700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:08:17.968 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.962+0000 7fcc34a90700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:08:17.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.964+0000 7fcc2f7fe700 1 -- 172.21.15.59:0/1895631132 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1717620545 0 0) 0x7fcc300a63b0 con 0x7fcc301420c0 2023-09-30T02:08:17.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.964+0000 7fcc2f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:17.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.964+0000 7fcc2f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:17.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.964+0000 7fcc2f7fe700 10 monclient(hunting): my global_id is 4555 2023-09-30T02:08:17.970 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.964+0000 7fcc2f7fe700 1 -- 172.21.15.59:0/1895631132 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc20003570 con 0x7fcc301420c0 2023-09-30T02:08:17.970 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.964+0000 7fcc2ffff700 10 monclient(hunting): handle_auth_done global_id 4548 payload 274 2023-09-30T02:08:17.970 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.964+0000 7fcc2ffff700 1 -- 172.21.15.59:0/1895631132 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fcc300a5b70 msgr2=0x7fcc300b1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:08:17.970 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.964+0000 7fcc2ffff700 1 --2- 172.21.15.59:0/1895631132 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fcc300a5b70 0x7fcc300b1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:17.970 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.964+0000 7fcc2ffff700 1 -- 172.21.15.59:0/1895631132 >> v1:172.21.15.110:6789/0 conn(0x7fcc301420c0 legacy=0x7fcc3013e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:17.970 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.964+0000 7fcc2ffff700 10 monclient: _finish_hunting 0 2023-09-30T02:08:17.970 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.964+0000 7fcc2ffff700 1 monclient: found mon.noname-a 2023-09-30T02:08:17.970 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.964+0000 7fcc2ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:08:17.970 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.964+0000 7fcc2ffff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.59:3300/0 2023-09-30T02:08:17.970 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.964+0000 7fcc2ffff700 1 -- 172.21.15.59:0/1895631132 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc30142500 con 0x7fcc300a5f40 2023-09-30T02:08:17.970 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.966+0000 7fcc2ffff700 1 --2- 172.21.15.59:0/1895631132 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc300a5f40 0x7fcc30141b80 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fcc1c0098e0 tx=0x7fcc1c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b75fbdf2ffe38de server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:08:17.971 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.969+0000 7fcc2f7fe700 1 -- 172.21.15.59:0/1895631132 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc1c011630 con 0x7fcc300a5f40 2023-09-30T02:08:17.971 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.969+0000 7fcc2f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:08:17.971 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.969+0000 7fcc2f7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T02:08:17.971 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.969+0000 7fcc2f7fe700 10 monclient: dump: 2023-09-30T02:08:17.971 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:08:17.971 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:17.971 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:17.971 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:17.971 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:08:17.971 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:17.971 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:08:17.971 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:08:17.972 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:08:17.972 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:17.972 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.969+0000 7fcc2f7fe700 10 monclient: _finish_auth 0 2023-09-30T02:08:17.972 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.969+0000 7fcc2f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:08:17.972 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.969+0000 7fcc36cf4700 5 monclient: authenticate success, global_id 4548 2023-09-30T02:08:17.972 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.969+0000 7fcc36cf4700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:08:17.973 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.971+0000 7fcc2f7fe700 1 -- 172.21.15.59:0/1895631132 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc1c011790 con 0x7fcc300a5f40 2023-09-30T02:08:17.973 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.971+0000 7fcc2f7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:08:17.973 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.971+0000 7fcc2f7fe700 1 -- 172.21.15.59:0/1895631132 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc1c011b10 con 0x7fcc300a5f40 2023-09-30T02:08:17.974 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.971+0000 7fcc2f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:08:17.974 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.971+0000 7fcc2f7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T02:08:17.974 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.971+0000 7fcc2f7fe700 10 monclient: dump: 2023-09-30T02:08:17.974 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:08:17.974 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:17.974 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:17.974 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:17.974 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:08:17.974 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:17.974 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:08:17.974 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:08:17.975 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:08:17.975 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:17.975 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.971+0000 7fcc36cf4700 10 monclient: get_monmap_and_config success 2023-09-30T02:08:17.975 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.971+0000 7fcc36cf4700 4 set_mon_vals no callback set 2023-09-30T02:08:17.975 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.971+0000 7fcc36cf4700 10 monclient: shutdown 2023-09-30T02:08:17.975 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.971+0000 7fcc36cf4700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:08:17.975 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.971+0000 7fcc36cf4700 1 -- 172.21.15.59:0/1895631132 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc300a5f40 msgr2=0x7fcc30141b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:17.975 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.971+0000 7fcc36cf4700 1 --2- 172.21.15.59:0/1895631132 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc300a5f40 0x7fcc30141b80 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fcc1c0098e0 tx=0x7fcc1c0091a0 comp rx=0 tx=0).stop 2023-09-30T02:08:17.975 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.972+0000 7fcc36cf4700 1 -- 172.21.15.59:0/1895631132 shutdown_connections 2023-09-30T02:08:17.975 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.972+0000 7fcc36cf4700 1 --2- 172.21.15.59:0/1895631132 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fcc300a5b70 0x7fcc300b1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:17.975 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.972+0000 7fcc36cf4700 1 --2- 172.21.15.59:0/1895631132 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc300a5f40 0x7fcc30141b80 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:17.975 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.972+0000 7fcc36cf4700 1 -- 172.21.15.59:0/1895631132 >> 172.21.15.59:0/1895631132 conn(0x7fcc300631f0 msgr2=0x7fcc300635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:08:17.976 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.972+0000 7fcc36cf4700 1 -- 172.21.15.59:0/1895631132 shutdown_connections 2023-09-30T02:08:17.976 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.972+0000 7fcc36cf4700 1 -- 172.21.15.59:0/1895631132 wait complete. 2023-09-30T02:08:17.976 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.972+0000 7fcc36cf4700 10 monclient: build_initial_monmap 2023-09-30T02:08:17.976 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.972+0000 7fcc36cf4700 10 monclient: monmap: 2023-09-30T02:08:17.976 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:08:17.976 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:08:17.976 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:08:17.976 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:08:17.976 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:08:17.976 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:08:17.976 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:08:17.976 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:08:17.976 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:08:17.977 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:17.977 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.973+0000 7fcc36cf4700 1 librados: starting msgr at 2023-09-30T02:08:17.977 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.973+0000 7fcc36cf4700 1 librados: starting objecter 2023-09-30T02:08:17.977 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.973+0000 7fcc36cf4700 1 Processor -- start 2023-09-30T02:08:17.979 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.973+0000 7fcc36cf4700 1 -- start start 2023-09-30T02:08:17.979 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.973+0000 7fcc36cf4700 1 librados: setting wanted keys 2023-09-30T02:08:17.979 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.973+0000 7fcc36cf4700 1 librados: calling monclient init 2023-09-30T02:08:17.979 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.973+0000 7fcc36cf4700 10 monclient: init 2023-09-30T02:08:17.979 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.974+0000 7fcc36cf4700 10 monclient: _reopen_session rank -1 2023-09-30T02:08:17.980 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.974+0000 7fcc36cf4700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T02:08:17.980 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.974+0000 7fcc36cf4700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcc300a5b70 0x7fcc30151780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:17.980 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.974+0000 7fcc36cf4700 10 monclient(hunting): picked mon.noname-b con 0x7fcc300a5b70 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:08:17.980 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.974+0000 7fcc36cf4700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc300a5f40 0x7fcc300a78a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:17.980 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.974+0000 7fcc36cf4700 10 monclient(hunting): picked mon.noname-a con 0x7fcc300a5f40 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:08:17.980 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.974+0000 7fcc36cf4700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fcc301420c0 0x7fcc300a7de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:17.980 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.974+0000 7fcc36cf4700 10 monclient(hunting): picked mon.noname-c con 0x7fcc301420c0 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:08:17.980 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.974+0000 7fcc36cf4700 10 monclient(hunting): start opening mon connection 2023-09-30T02:08:17.980 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.974+0000 7fcc36cf4700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc30152110 con 0x7fcc300a5f40 2023-09-30T02:08:17.981 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.974+0000 7fcc36cf4700 10 monclient(hunting): start opening mon connection 2023-09-30T02:08:17.981 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.974+0000 7fcc36cf4700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc300a98f0 con 0x7fcc300a5b70 2023-09-30T02:08:17.981 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.974+0000 7fcc36cf4700 10 monclient(hunting): start opening mon connection 2023-09-30T02:08:17.981 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.974+0000 7fcc36cf4700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc300a9bd0 con 0x7fcc301420c0 2023-09-30T02:08:17.981 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.974+0000 7fcc36cf4700 10 monclient(hunting): _renew_subs 2023-09-30T02:08:17.981 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.974+0000 7fcc36cf4700 10 monclient(hunting): authenticate will time out at 1943.277688s 2023-09-30T02:08:17.981 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.975+0000 7fcc2ffff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc300a5f40 0x7fcc300a78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:17.981 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.975+0000 7fcc34a90700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcc300a5b70 0x7fcc30151780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:17.981 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.975+0000 7fcc35291700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fcc301420c0 0x7fcc300a7de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:17.981 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.975+0000 7fcc2ffff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc300a5f40 0x7fcc300a78a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44710/0 (socket says 172.21.15.59:44710) 2023-09-30T02:08:17.981 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.975+0000 7fcc2ffff700 1 -- 172.21.15.59:0/1506980209 learned_addr learned my addr 172.21.15.59:0/1506980209 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:08:17.981 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.975+0000 7fcc2ffff700 10 monclient(hunting): get_auth_request con 0x7fcc300a5f40 auth_method 0 2023-09-30T02:08:17.982 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.975+0000 7fcc2ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:08:17.982 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.975+0000 7fcc2ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:17.982 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.975+0000 7fcc2ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:17.982 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.975+0000 7fcc34a90700 10 monclient(hunting): get_auth_request con 0x7fcc300a5b70 auth_method 0 2023-09-30T02:08:17.982 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.975+0000 7fcc34a90700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:08:17.982 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.975+0000 7fcc34a90700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:17.982 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.975+0000 7fcc34a90700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:17.982 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.975+0000 7fcc35291700 10 monclient(hunting): get_auth_request con 0x7fcc301420c0 auth_method 0 2023-09-30T02:08:17.982 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.975+0000 7fcc35291700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:08:17.982 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.975+0000 7fcc35291700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:17.982 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.975+0000 7fcc35291700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:17.982 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.976+0000 7fcc2ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:08:17.983 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.976+0000 7fcc2ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:08:17.983 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.976+0000 7fcc2ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:08:17.983 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.976+0000 7fcc35291700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:08:17.983 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.976+0000 7fcc35291700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:08:17.983 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.976+0000 7fcc35291700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:08:17.983 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.976+0000 7fcc34a90700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:08:17.983 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.976+0000 7fcc34a90700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:08:17.983 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.976+0000 7fcc34a90700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:08:17.984 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.980+0000 7fcc35291700 10 monclient(hunting): handle_auth_done global_id 4487 payload 824 2023-09-30T02:08:17.985 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.980+0000 7fcc35291700 1 -- 172.21.15.59:0/1506980209 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcc300a5b70 msgr2=0x7fcc30151780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:17.985 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.980+0000 7fcc35291700 1 --2- 172.21.15.59:0/1506980209 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcc300a5b70 0x7fcc30151780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:17.985 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.980+0000 7fcc35291700 1 -- 172.21.15.59:0/1506980209 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc300a5f40 msgr2=0x7fcc300a78a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:17.985 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.980+0000 7fcc35291700 1 --2- 172.21.15.59:0/1506980209 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc300a5f40 0x7fcc300a78a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:17.985 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.980+0000 7fcc35291700 10 monclient: _finish_hunting 0 2023-09-30T02:08:17.985 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.980+0000 7fcc35291700 1 monclient: found mon.noname-c 2023-09-30T02:08:17.985 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.980+0000 7fcc35291700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:08:17.986 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.980+0000 7fcc35291700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.59:3301/0 2023-09-30T02:08:17.986 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.980+0000 7fcc35291700 1 -- 172.21.15.59:0/1506980209 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc1c009420 con 0x7fcc301420c0 2023-09-30T02:08:17.986 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.980+0000 7fcc2ffff700 1 --2- 172.21.15.59:0/1506980209 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc300a5f40 0x7fcc300a78a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T02:08:17.986 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.983+0000 7fcc35291700 1 --2- 172.21.15.59:0/1506980209 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fcc301420c0 0x7fcc300a7de0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fcc2800cb90 tx=0x7fcc28001980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:08:17.986 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.983+0000 7fcc2cff9700 10 client.?.objecter ms_handle_connect 0x7fcc301420c0 2023-09-30T02:08:17.986 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.983+0000 7fcc2cff9700 10 client.?.objecter resend_mon_ops 2023-09-30T02:08:17.987 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.985+0000 7fcc2cff9700 1 -- 172.21.15.59:0/1506980209 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc2800a7e0 con 0x7fcc301420c0 2023-09-30T02:08:17.987 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.985+0000 7fcc2cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:08:17.987 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.985+0000 7fcc2cff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:08:17.987 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.985+0000 7fcc2cff9700 10 monclient: dump: 2023-09-30T02:08:17.988 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:08:17.988 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:17.988 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:17.988 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:17.988 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:08:17.988 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:17.988 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:08:17.988 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:08:17.988 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:08:17.988 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:17.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.985+0000 7fcc2cff9700 10 monclient: _finish_auth 0 2023-09-30T02:08:17.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.985+0000 7fcc2cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:08:17.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.985+0000 7fcc36cf4700 5 monclient: authenticate success, global_id 4487 2023-09-30T02:08:17.989 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.985+0000 7fcc36cf4700 10 monclient: _renew_subs 2023-09-30T02:08:17.989 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.985+0000 7fcc36cf4700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:08:17.989 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.985+0000 7fcc36cf4700 1 -- 172.21.15.59:0/1506980209 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc300b4c70 con 0x7fcc301420c0 2023-09-30T02:08:17.989 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.985+0000 7fcc36cf4700 10 client.4487.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:08:17.989 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.985+0000 7fcc36cf4700 10 monclient: _renew_subs 2023-09-30T02:08:17.989 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.985+0000 7fcc36cf4700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:08:17.989 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.985+0000 7fcc36cf4700 1 -- 172.21.15.59:0/1506980209 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc30151cc0 con 0x7fcc301420c0 2023-09-30T02:08:17.989 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.986+0000 7fcc36cf4700 1 librados: init done 2023-09-30T02:08:17.989 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.986+0000 7fcc36cf4700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:08:17.989 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.986+0000 7fcc36cf4700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:08:17.989 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.986+0000 7fcc36cf4700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:08:17.989 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.986+0000 7fcc36cf4700 1 -- 172.21.15.59:0/1506980209 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc30141b80 con 0x7fcc301420c0 2023-09-30T02:08:17.990 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.988+0000 7fcc2cff9700 1 -- 172.21.15.59:0/1506980209 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc2800ab50 con 0x7fcc301420c0 2023-09-30T02:08:17.990 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.988+0000 7fcc2cff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:08:17.990 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.988+0000 7fcc2cff9700 1 -- 172.21.15.59:0/1506980209 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc2800ace0 con 0x7fcc301420c0 2023-09-30T02:08:17.990 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.988+0000 7fcc2cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:08:17.990 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.988+0000 7fcc2cff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:08:17.990 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.988+0000 7fcc2cff9700 10 monclient: dump: 2023-09-30T02:08:17.991 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:08:17.991 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:17.991 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:17.991 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:17.991 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:08:17.991 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:17.991 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:08:17.991 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:08:17.991 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:08:17.991 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:17.991 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:17.988+0000 7fcc2effd700 4 set_mon_vals no callback set 2023-09-30T02:08:18.290 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:18.290+0000 7fcc2cff9700 1 -- 172.21.15.59:0/1506980209 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fcc280070f0 con 0x7fcc301420c0 2023-09-30T02:08:18.292 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:18.290+0000 7fcc2cff9700 1 --2- 172.21.15.59:0/1506980209 >> v2:172.21.15.110:6800/115870 conn(0x7fcc14068890 0x7fcc1406ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:18.292 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:18.291+0000 7fcc2cff9700 1 -- 172.21.15.59:0/1506980209 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fcc280840e0 con 0x7fcc301420c0 2023-09-30T02:08:18.292 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:18.291+0000 7fcc2cff9700 10 client.4487.objecter ms_dispatch 0x7fcc30001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:08:18.292 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:18.291+0000 7fcc2cff9700 3 client.4487.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:08:18.292 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:18.291+0000 7fcc2cff9700 3 client.4487.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:08:18.292 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:18.291+0000 7fcc2cff9700 20 client.4487.objecter dump_active .. 0 homeless 2023-09-30T02:08:18.292 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:18.291+0000 7fcc2cff9700 1 -- 172.21.15.59:0/1506980209 <== mon.2 v2:172.21.15.59:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcc280845e0 con 0x7fcc301420c0 2023-09-30T02:08:18.292 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:18.291+0000 7fcc2cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:08:18.292 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:18.291+0000 7fcc2cff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:08:18.292 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:18.291+0000 7fcc34a90700 1 --2- 172.21.15.59:0/1506980209 >> v2:172.21.15.110:6800/115870 conn(0x7fcc14068890 0x7fcc1406ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:18.292 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:18.291+0000 7fcc34a90700 10 monclient: get_auth_request con 0x7fcc14068890 auth_method 0 2023-09-30T02:08:18.292 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:18.291+0000 7fcc34a90700 1 --2- 172.21.15.59:0/1506980209 >> v2:172.21.15.110:6800/115870 conn(0x7fcc14068890 0x7fcc1406ad40 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fcc24009db0 tx=0x7fcc240091a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:08:18.293 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:18.292+0000 7fcc2cff9700 10 client.4487.objecter ms_handle_connect 0x7fcc14068890 2023-09-30T02:08:18.549 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:18.548+0000 7fcc36cf4700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:08:18.549 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:18.548+0000 7fcc36cf4700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:08:18.549 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:18.548+0000 7fcc36cf4700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:08:18.549 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:18.548+0000 7fcc36cf4700 1 -- 172.21.15.59:0/1506980209 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fcc300a9d10 con 0x7fcc301420c0 2023-09-30T02:08:18.557 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:18.557+0000 7fcc2cff9700 1 -- 172.21.15.59:0/1506980209 <== mon.2 v2:172.21.15.59:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fcc300a9d10 con 0x7fcc301420c0 2023-09-30T02:08:18.557 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:18.557+0000 7fcc2cff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:08:18.557 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:18.557+0000 7fcc2cff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:08:18.558 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:08:18.558 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:08:18.560 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:18.559+0000 7fcc36cf4700 10 librados: watch_flush enter 2023-09-30T02:08:18.561 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:18.559+0000 7fcc36cf4700 10 librados: watch_flush exit 2023-09-30T02:08:18.561 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:18.559+0000 7fcc36cf4700 20 client.4487.objecter shutdown clearing up homeless session... 2023-09-30T02:08:18.561 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:18.559+0000 7fcc36cf4700 10 client.4487.objecter successfully canceled tick 2023-09-30T02:08:18.561 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:18.559+0000 7fcc36cf4700 1 -- 172.21.15.59:0/1506980209 >> v2:172.21.15.110:6800/115870 conn(0x7fcc14068890 msgr2=0x7fcc1406ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:18.561 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:18.559+0000 7fcc36cf4700 1 --2- 172.21.15.59:0/1506980209 >> v2:172.21.15.110:6800/115870 conn(0x7fcc14068890 0x7fcc1406ad40 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fcc24009db0 tx=0x7fcc240091a0 comp rx=0 tx=0).stop 2023-09-30T02:08:18.561 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:18.559+0000 7fcc36cf4700 10 monclient: shutdown 2023-09-30T02:08:18.561 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:18.559+0000 7fcc36cf4700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:08:18.561 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:18.559+0000 7fcc36cf4700 1 -- 172.21.15.59:0/1506980209 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fcc301420c0 msgr2=0x7fcc300a7de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:18.561 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:18.559+0000 7fcc36cf4700 1 --2- 172.21.15.59:0/1506980209 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fcc301420c0 0x7fcc300a7de0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fcc2800cb90 tx=0x7fcc28001980 comp rx=0 tx=0).stop 2023-09-30T02:08:18.561 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:18.560+0000 7fcc36cf4700 1 -- 172.21.15.59:0/1506980209 shutdown_connections 2023-09-30T02:08:18.561 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:18.560+0000 7fcc36cf4700 1 --2- 172.21.15.59:0/1506980209 >> v2:172.21.15.110:6800/115870 conn(0x7fcc14068890 0x7fcc1406ad40 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:18.562 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:18.560+0000 7fcc36cf4700 1 --2- 172.21.15.59:0/1506980209 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcc300a5b70 0x7fcc30151780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:18.562 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:18.560+0000 7fcc36cf4700 1 --2- 172.21.15.59:0/1506980209 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fcc301420c0 0x7fcc300a7de0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:18.562 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:18.560+0000 7fcc36cf4700 1 --2- 172.21.15.59:0/1506980209 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc300a5f40 0x7fcc300a78a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:18.562 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:18.560+0000 7fcc36cf4700 1 -- 172.21.15.59:0/1506980209 >> 172.21.15.59:0/1506980209 conn(0x7fcc300631f0 msgr2=0x7fcc30138d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:08:18.562 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:18.560+0000 7fcc36cf4700 1 -- 172.21.15.59:0/1506980209 shutdown_connections 2023-09-30T02:08:18.562 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:18.560+0000 7fcc36cf4700 1 -- 172.21.15.59:0/1506980209 wait complete. 2023-09-30T02:08:18.562 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:18.560+0000 7fcc36cf4700 1 librados: shutdown 2023-09-30T02:08:21.575 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:08:21.715 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.715+0000 7f8b387ff700 10 monclient: get_monmap_and_config 2023-09-30T02:08:21.716 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.715+0000 7f8b387ff700 10 monclient: build_initial_monmap 2023-09-30T02:08:21.716 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.715+0000 7f8b387ff700 10 monclient: monmap: 2023-09-30T02:08:21.716 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:08:21.716 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:21.716 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:08:21.716774+0000 2023-09-30T02:08:21.716 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:08:21.716774+0000 2023-09-30T02:08:21.716 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:08:21.716 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:21.716 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:08:21.716 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:08:21.717 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:08:21.717 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:08:21.717 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:08:21.717 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:21.718 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.716+0000 7f8b387ff700 1 Processor -- start 2023-09-30T02:08:21.718 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.716+0000 7f8b387ff700 1 -- start start 2023-09-30T02:08:21.718 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.716+0000 7f8b387ff700 10 monclient: init 2023-09-30T02:08:21.719 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.716+0000 7f8b387ff700 10 monclient: _reopen_session rank -1 2023-09-30T02:08:21.719 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.716+0000 7f8b387ff700 10 monclient: _add_conns ranks=[2,0,1,3,4] 2023-09-30T02:08:21.719 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.716+0000 7f8b387ff700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8b300a5b70 0x7f8b300b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:21.719 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.716+0000 7f8b387ff700 10 monclient(hunting): picked mon.noname-b con 0x7f8b300a5b70 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:08:21.719 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.716+0000 7f8b387ff700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8b300a5f40 addr v1:172.21.15.59:6789/0 2023-09-30T02:08:21.719 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.716+0000 7f8b387ff700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8b30142290 addr v1:172.21.15.110:6789/0 2023-09-30T02:08:21.719 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.716+0000 7f8b387ff700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b300a63b0 con 0x7f8b300a5f40 2023-09-30T02:08:21.719 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.716+0000 7f8b387ff700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b300a75a0 con 0x7f8b30142290 2023-09-30T02:08:21.719 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.716+0000 7f8b387ff700 10 monclient(hunting): start opening mon connection 2023-09-30T02:08:21.719 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.716+0000 7f8b387ff700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b301426d0 con 0x7f8b300a5b70 2023-09-30T02:08:21.719 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.716+0000 7f8b387ff700 10 monclient(hunting): _renew_subs 2023-09-30T02:08:21.719 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.716+0000 7f8b387ff700 10 monclient(hunting): authenticate will time out at 1947.019989s 2023-09-30T02:08:21.720 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.717+0000 7f8b3659b700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8b300a5b70 0x7f8b300b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:21.720 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.717+0000 7f8b36d9c700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f8b30142290 0x7f8b3013af30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.59:33428/0 (socket says 172.21.15.59:33428) 2023-09-30T02:08:21.720 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.717+0000 7f8b36d9c700 1 -- 172.21.15.59:0/298977711 learned_addr learned my addr 172.21.15.59:0/298977711 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:08:21.720 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.718+0000 7f8b3659b700 10 monclient(hunting): get_auth_request con 0x7f8b300a5b70 auth_method 0 2023-09-30T02:08:21.720 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.718+0000 7f8b3659b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:08:21.720 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.718+0000 7f8b3659b700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:21.720 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.718+0000 7f8b3659b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:21.721 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.718+0000 7f8b3659b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:08:21.721 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.718+0000 7f8b3659b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:08:21.721 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.718+0000 7f8b3659b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:08:21.721 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.721+0000 7f8b3659b700 10 monclient(hunting): handle_auth_done global_id 4490 payload 274 2023-09-30T02:08:21.721 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.721+0000 7f8b3659b700 1 -- 172.21.15.59:0/298977711 >> v1:172.21.15.110:6789/0 conn(0x7f8b30142290 legacy=0x7f8b3013af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:21.721 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.721+0000 7f8b3659b700 1 -- 172.21.15.59:0/298977711 >> v1:172.21.15.59:6789/0 conn(0x7f8b300a5f40 legacy=0x7f8b30141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:08:21.722 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.721+0000 7f8b3659b700 10 monclient: _finish_hunting 0 2023-09-30T02:08:21.722 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.721+0000 7f8b3659b700 1 monclient: found mon.noname-b 2023-09-30T02:08:21.722 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.721+0000 7f8b3659b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:08:21.722 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.721+0000 7f8b3659b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.59:3301/0 2023-09-30T02:08:21.722 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.721+0000 7f8b3659b700 1 -- 172.21.15.59:0/298977711 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b300af3f0 con 0x7f8b300a5b70 2023-09-30T02:08:21.723 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.723+0000 7f8b3659b700 1 --2- 172.21.15.59:0/298977711 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8b300a5b70 0x7f8b300b1500 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8b20009910 tx=0x7f8b200091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=99c4aa1515e45257 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:08:21.729 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.728+0000 7f8b35599700 1 -- 172.21.15.59:0/298977711 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b2000b820 con 0x7f8b300a5b70 2023-09-30T02:08:21.729 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.728+0000 7f8b35599700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:08:21.729 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.728+0000 7f8b35599700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:08:21.729 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.728+0000 7f8b35599700 10 monclient: dump: 2023-09-30T02:08:21.729 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:08:21.729 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:21.729 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:21.729 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:21.729 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:08:21.729 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:21.730 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:08:21.730 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:08:21.730 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:08:21.730 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:21.730 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.728+0000 7f8b35599700 10 monclient: _finish_auth 0 2023-09-30T02:08:21.730 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.728+0000 7f8b35599700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:08:21.730 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.728+0000 7f8b35599700 1 -- 172.21.15.59:0/298977711 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b2000b980 con 0x7f8b300a5b70 2023-09-30T02:08:21.730 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.728+0000 7f8b387ff700 5 monclient: authenticate success, global_id 4490 2023-09-30T02:08:21.730 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.728+0000 7f8b387ff700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:08:21.730 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.728+0000 7f8b35599700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:08:21.730 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.728+0000 7f8b387ff700 10 monclient: get_monmap_and_config success 2023-09-30T02:08:21.730 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.728+0000 7f8b35599700 1 -- 172.21.15.59:0/298977711 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b2000bd20 con 0x7f8b300a5b70 2023-09-30T02:08:21.731 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.728+0000 7f8b387ff700 4 set_mon_vals no callback set 2023-09-30T02:08:21.731 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.728+0000 7f8b35599700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:08:21.731 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.728+0000 7f8b35599700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:08:21.731 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.728+0000 7f8b35599700 10 monclient: dump: 2023-09-30T02:08:21.731 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:08:21.731 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:21.731 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:21.731 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:21.731 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:08:21.731 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:21.731 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:08:21.731 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:08:21.732 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:08:21.732 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:21.734 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.729+0000 7f8b387ff700 10 monclient: shutdown 2023-09-30T02:08:21.734 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.729+0000 7f8b387ff700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:08:21.734 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.729+0000 7f8b387ff700 1 -- 172.21.15.59:0/298977711 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8b300a5b70 msgr2=0x7f8b300b1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:21.734 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.729+0000 7f8b387ff700 1 --2- 172.21.15.59:0/298977711 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8b300a5b70 0x7f8b300b1500 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f8b20009910 tx=0x7f8b200091a0 comp rx=0 tx=0).stop 2023-09-30T02:08:21.735 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.729+0000 7f8b387ff700 1 -- 172.21.15.59:0/298977711 shutdown_connections 2023-09-30T02:08:21.735 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.729+0000 7f8b387ff700 1 --2- 172.21.15.59:0/298977711 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8b300a5b70 0x7f8b300b1500 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:21.735 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.729+0000 7f8b387ff700 1 -- 172.21.15.59:0/298977711 >> 172.21.15.59:0/298977711 conn(0x7f8b300631f0 msgr2=0x7f8b300635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:08:21.735 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.729+0000 7f8b387ff700 1 -- 172.21.15.59:0/298977711 shutdown_connections 2023-09-30T02:08:21.735 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.729+0000 7f8b387ff700 1 -- 172.21.15.59:0/298977711 wait complete. 2023-09-30T02:08:21.735 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.730+0000 7f8b387ff700 10 monclient: build_initial_monmap 2023-09-30T02:08:21.735 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.730+0000 7f8b387ff700 10 monclient: monmap: 2023-09-30T02:08:21.735 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:08:21.735 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:08:21.735 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:08:21.735 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:08:21.735 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:08:21.736 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:08:21.736 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:08:21.736 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:08:21.736 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:08:21.736 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:21.736 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.730+0000 7f8b387ff700 1 librados: starting msgr at 2023-09-30T02:08:21.736 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.730+0000 7f8b387ff700 1 librados: starting objecter 2023-09-30T02:08:21.736 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.730+0000 7f8b387ff700 1 Processor -- start 2023-09-30T02:08:21.736 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.730+0000 7f8b387ff700 1 -- start start 2023-09-30T02:08:21.736 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.730+0000 7f8b387ff700 1 librados: setting wanted keys 2023-09-30T02:08:21.736 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.730+0000 7f8b387ff700 1 librados: calling monclient init 2023-09-30T02:08:21.736 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.730+0000 7f8b387ff700 10 monclient: init 2023-09-30T02:08:21.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.731+0000 7f8b387ff700 10 monclient: _reopen_session rank -1 2023-09-30T02:08:21.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.731+0000 7f8b387ff700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T02:08:21.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.731+0000 7f8b387ff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b300a5b70 0x7f8b3013c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:21.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.731+0000 7f8b387ff700 10 monclient(hunting): picked mon.noname-a con 0x7f8b300a5b70 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:08:21.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.731+0000 7f8b387ff700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8b300a5f40 0x7f8b30139ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:21.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.731+0000 7f8b387ff700 10 monclient(hunting): picked mon.noname-c con 0x7f8b300a5f40 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:08:21.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.731+0000 7f8b387ff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8b30142290 0x7f8b3013e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:21.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.731+0000 7f8b387ff700 10 monclient(hunting): picked mon.noname-b con 0x7f8b30142290 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:08:21.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.731+0000 7f8b387ff700 10 monclient(hunting): start opening mon connection 2023-09-30T02:08:21.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.731+0000 7f8b387ff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b30140ec0 con 0x7f8b300a5b70 2023-09-30T02:08:21.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.731+0000 7f8b387ff700 10 monclient(hunting): start opening mon connection 2023-09-30T02:08:21.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.731+0000 7f8b387ff700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3013a980 con 0x7f8b30142290 2023-09-30T02:08:21.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.731+0000 7f8b387ff700 10 monclient(hunting): start opening mon connection 2023-09-30T02:08:21.738 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.731+0000 7f8b387ff700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b30139ff0 con 0x7f8b300a5f40 2023-09-30T02:08:21.738 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.731+0000 7f8b387ff700 10 monclient(hunting): _renew_subs 2023-09-30T02:08:21.738 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.731+0000 7f8b387ff700 10 monclient(hunting): authenticate will time out at 1947.034486s 2023-09-30T02:08:21.738 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.731+0000 7f8b35d9a700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8b300a5f40 0x7f8b30139ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:21.738 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.732+0000 7f8b36d9c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8b30142290 0x7f8b3013e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:21.738 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.732+0000 7f8b35d9a700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8b300a5f40 0x7f8b30139ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:44398/0 (socket says 172.21.15.59:44398) 2023-09-30T02:08:21.738 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.732+0000 7f8b35d9a700 1 -- 172.21.15.59:0/1822871378 learned_addr learned my addr 172.21.15.59:0/1822871378 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:08:21.738 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.732+0000 7f8b35d9a700 10 monclient(hunting): get_auth_request con 0x7f8b300a5f40 auth_method 0 2023-09-30T02:08:21.738 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.732+0000 7f8b35d9a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:08:21.738 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.732+0000 7f8b35d9a700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:21.738 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.732+0000 7f8b35d9a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:21.739 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.732+0000 7f8b36d9c700 10 monclient(hunting): get_auth_request con 0x7f8b30142290 auth_method 0 2023-09-30T02:08:21.739 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.732+0000 7f8b36d9c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:08:21.739 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.732+0000 7f8b36d9c700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:21.739 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.732+0000 7f8b36d9c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:21.739 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.732+0000 7f8b35d9a700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:08:21.739 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.733+0000 7f8b35d9a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:08:21.739 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.733+0000 7f8b35d9a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:08:21.739 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.733+0000 7f8b36d9c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:08:21.739 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.733+0000 7f8b36d9c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:08:21.739 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.733+0000 7f8b36d9c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:08:21.740 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.736+0000 7f8b35d9a700 10 monclient(hunting): handle_auth_done global_id 4493 payload 824 2023-09-30T02:08:21.740 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.736+0000 7f8b35d9a700 1 -- 172.21.15.59:0/1822871378 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8b30142290 msgr2=0x7f8b3013e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:21.740 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.736+0000 7f8b35d9a700 1 --2- 172.21.15.59:0/1822871378 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8b30142290 0x7f8b3013e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:21.740 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.736+0000 7f8b35d9a700 1 -- 172.21.15.59:0/1822871378 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b300a5b70 msgr2=0x7f8b3013c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:21.740 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.736+0000 7f8b35d9a700 1 --2- 172.21.15.59:0/1822871378 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b300a5b70 0x7f8b3013c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:21.740 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.736+0000 7f8b35d9a700 10 monclient: _finish_hunting 0 2023-09-30T02:08:21.741 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.736+0000 7f8b35d9a700 1 monclient: found mon.noname-c 2023-09-30T02:08:21.741 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.736+0000 7f8b35d9a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:08:21.741 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.736+0000 7f8b35d9a700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.59:3301/0 2023-09-30T02:08:21.741 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.736+0000 7f8b35d9a700 1 -- 172.21.15.59:0/1822871378 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b20009420 con 0x7f8b300a5f40 2023-09-30T02:08:21.741 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.739+0000 7f8b35d9a700 1 --2- 172.21.15.59:0/1822871378 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8b300a5f40 0x7f8b30139ab0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8b2c00b600 tx=0x7f8b2c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:08:21.741 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.739+0000 7f8b267fc700 10 client.?.objecter ms_handle_connect 0x7f8b300a5f40 2023-09-30T02:08:21.741 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.739+0000 7f8b267fc700 10 client.?.objecter resend_mon_ops 2023-09-30T02:08:21.744 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.743+0000 7f8b267fc700 1 -- 172.21.15.59:0/1822871378 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b2c00f510 con 0x7f8b300a5f40 2023-09-30T02:08:21.744 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.743+0000 7f8b267fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:08:21.744 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.743+0000 7f8b267fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:08:21.744 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.743+0000 7f8b267fc700 10 monclient: dump: 2023-09-30T02:08:21.744 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:08:21.744 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:21.744 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:21.744 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:21.744 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:08:21.744 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:21.744 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:08:21.745 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:08:21.745 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:08:21.745 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:21.745 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.743+0000 7f8b267fc700 10 monclient: _finish_auth 0 2023-09-30T02:08:21.745 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.743+0000 7f8b267fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:08:21.745 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.743+0000 7f8b267fc700 1 -- 172.21.15.59:0/1822871378 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b2c00f690 con 0x7f8b300a5f40 2023-09-30T02:08:21.745 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.743+0000 7f8b267fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:08:21.745 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.743+0000 7f8b267fc700 1 -- 172.21.15.59:0/1822871378 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b2c00fa50 con 0x7f8b300a5f40 2023-09-30T02:08:21.745 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.743+0000 7f8b387ff700 5 monclient: authenticate success, global_id 4493 2023-09-30T02:08:21.745 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.743+0000 7f8b387ff700 10 monclient: _renew_subs 2023-09-30T02:08:21.745 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.743+0000 7f8b387ff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:08:21.745 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.743+0000 7f8b387ff700 1 -- 172.21.15.59:0/1822871378 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b30141350 con 0x7f8b300a5f40 2023-09-30T02:08:21.745 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.743+0000 7f8b34d98700 4 set_mon_vals no callback set 2023-09-30T02:08:21.746 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.743+0000 7f8b267fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:08:21.746 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.743+0000 7f8b267fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:08:21.746 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.743+0000 7f8b267fc700 10 monclient: dump: 2023-09-30T02:08:21.746 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:08:21.746 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:21.746 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:21.746 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:21.746 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:08:21.746 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:21.746 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:08:21.746 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:08:21.746 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:08:21.747 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:21.747 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.743+0000 7f8b387ff700 10 client.4493.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:08:21.747 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.743+0000 7f8b387ff700 10 monclient: _renew_subs 2023-09-30T02:08:21.747 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.743+0000 7f8b387ff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:08:21.747 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.743+0000 7f8b387ff700 1 -- 172.21.15.59:0/1822871378 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b3013e8f0 con 0x7f8b300a5f40 2023-09-30T02:08:21.747 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.743+0000 7f8b387ff700 1 librados: init done 2023-09-30T02:08:21.747 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.744+0000 7f8b387ff700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:08:21.748 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.744+0000 7f8b387ff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:08:21.748 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.744+0000 7f8b387ff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:08:21.748 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.744+0000 7f8b387ff700 1 -- 172.21.15.59:0/1822871378 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b30141570 con 0x7f8b300a5f40 2023-09-30T02:08:21.816 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.816+0000 7f8b267fc700 1 -- 172.21.15.59:0/1822871378 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f8b2c024430 con 0x7f8b300a5f40 2023-09-30T02:08:21.817 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.816+0000 7f8b267fc700 1 --2- 172.21.15.59:0/1822871378 >> v2:172.21.15.110:6800/115870 conn(0x7f8b14068890 0x7f8b1406ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:21.817 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.816+0000 7f8b267fc700 1 -- 172.21.15.59:0/1822871378 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f8b2c084420 con 0x7f8b300a5f40 2023-09-30T02:08:21.817 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.816+0000 7f8b267fc700 10 client.4493.objecter ms_dispatch 0x7f8b30001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:08:21.817 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.816+0000 7f8b267fc700 3 client.4493.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:08:21.817 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.816+0000 7f8b267fc700 3 client.4493.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:08:21.818 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.817+0000 7f8b3659b700 1 --2- 172.21.15.59:0/1822871378 >> v2:172.21.15.110:6800/115870 conn(0x7f8b14068890 0x7f8b1406ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:21.818 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.817+0000 7f8b3659b700 10 monclient: get_auth_request con 0x7f8b14068890 auth_method 0 2023-09-30T02:08:21.818 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.817+0000 7f8b267fc700 20 client.4493.objecter dump_active .. 0 homeless 2023-09-30T02:08:21.818 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.817+0000 7f8b267fc700 10 client.4493.objecter ms_handle_connect 0x7f8b14068890 2023-09-30T02:08:21.818 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:21.817+0000 7f8b3659b700 1 --2- 172.21.15.59:0/1822871378 >> v2:172.21.15.110:6800/115870 conn(0x7f8b14068890 0x7f8b1406ad40 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8b200093a0 tx=0x7f8b20016000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:08:22.045 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:22.045+0000 7f8b267fc700 1 -- 172.21.15.59:0/1822871378 <== mon.2 v2:172.21.15.59:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8b2c051d40 con 0x7f8b300a5f40 2023-09-30T02:08:22.045 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:22.045+0000 7f8b267fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:08:22.045 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:22.045+0000 7f8b267fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:08:22.319 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:22.318+0000 7f8b387ff700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:08:22.319 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:22.318+0000 7f8b387ff700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:08:22.319 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:22.318+0000 7f8b387ff700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:08:22.319 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:22.318+0000 7f8b387ff700 1 -- 172.21.15.59:0/1822871378 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f8b3013eb70 con 0x7f8b300a5f40 2023-09-30T02:08:22.324 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:22.324+0000 7f8b267fc700 1 -- 172.21.15.59:0/1822871378 <== mon.2 v2:172.21.15.59:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f8b2c055360 con 0x7f8b300a5f40 2023-09-30T02:08:22.325 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:22.324+0000 7f8b267fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:08:22.325 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:22.324+0000 7f8b267fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:08:22.325 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:08:22.325 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:08:22.328 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:22.326+0000 7f8b387ff700 10 librados: watch_flush enter 2023-09-30T02:08:22.328 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:22.327+0000 7f8b387ff700 10 librados: watch_flush exit 2023-09-30T02:08:22.328 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:22.327+0000 7f8b387ff700 20 client.4493.objecter shutdown clearing up homeless session... 2023-09-30T02:08:22.328 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:22.327+0000 7f8b387ff700 10 client.4493.objecter successfully canceled tick 2023-09-30T02:08:22.328 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:22.327+0000 7f8b387ff700 1 -- 172.21.15.59:0/1822871378 >> v2:172.21.15.110:6800/115870 conn(0x7f8b14068890 msgr2=0x7f8b1406ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:22.328 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:22.327+0000 7f8b387ff700 1 --2- 172.21.15.59:0/1822871378 >> v2:172.21.15.110:6800/115870 conn(0x7f8b14068890 0x7f8b1406ad40 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8b200093a0 tx=0x7f8b20016000 comp rx=0 tx=0).stop 2023-09-30T02:08:22.328 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:22.327+0000 7f8b387ff700 10 monclient: shutdown 2023-09-30T02:08:22.328 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:22.327+0000 7f8b387ff700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:08:22.329 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:22.327+0000 7f8b387ff700 1 -- 172.21.15.59:0/1822871378 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8b300a5f40 msgr2=0x7f8b30139ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:22.329 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:22.327+0000 7f8b387ff700 1 --2- 172.21.15.59:0/1822871378 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8b300a5f40 0x7f8b30139ab0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8b2c00b600 tx=0x7f8b2c0091a0 comp rx=0 tx=0).stop 2023-09-30T02:08:22.329 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:22.327+0000 7f8b387ff700 1 -- 172.21.15.59:0/1822871378 shutdown_connections 2023-09-30T02:08:22.329 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:22.327+0000 7f8b387ff700 1 --2- 172.21.15.59:0/1822871378 >> v2:172.21.15.110:6800/115870 conn(0x7f8b14068890 0x7f8b1406ad40 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:22.329 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:22.327+0000 7f8b387ff700 1 --2- 172.21.15.59:0/1822871378 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8b30142290 0x7f8b3013e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:22.329 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:22.327+0000 7f8b387ff700 1 --2- 172.21.15.59:0/1822871378 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8b300a5b70 0x7f8b3013c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:22.329 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:22.327+0000 7f8b387ff700 1 --2- 172.21.15.59:0/1822871378 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8b300a5f40 0x7f8b30139ab0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:22.329 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:22.327+0000 7f8b387ff700 1 -- 172.21.15.59:0/1822871378 >> 172.21.15.59:0/1822871378 conn(0x7f8b300631f0 msgr2=0x7f8b30070f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:08:22.329 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:22.327+0000 7f8b387ff700 1 -- 172.21.15.59:0/1822871378 shutdown_connections 2023-09-30T02:08:22.329 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:22.327+0000 7f8b387ff700 1 -- 172.21.15.59:0/1822871378 wait complete. 2023-09-30T02:08:22.330 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:22.327+0000 7f8b387ff700 1 librados: shutdown 2023-09-30T02:08:25.342 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:08:25.479 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.479+0000 7f3848bc8700 10 monclient: get_monmap_and_config 2023-09-30T02:08:25.480 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.479+0000 7f3848bc8700 10 monclient: build_initial_monmap 2023-09-30T02:08:25.480 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.479+0000 7f3848bc8700 10 monclient: monmap: 2023-09-30T02:08:25.480 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:08:25.480 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:25.480 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:08:25.480749+0000 2023-09-30T02:08:25.480 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:08:25.480749+0000 2023-09-30T02:08:25.480 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:08:25.480 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:25.480 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:08:25.480 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:08:25.480 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:08:25.481 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:08:25.481 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:08:25.481 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:25.482 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.480+0000 7f3848bc8700 1 Processor -- start 2023-09-30T02:08:25.482 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.480+0000 7f3848bc8700 1 -- start start 2023-09-30T02:08:25.482 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.480+0000 7f3848bc8700 10 monclient: init 2023-09-30T02:08:25.482 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.480+0000 7f3848bc8700 10 monclient: _reopen_session rank -1 2023-09-30T02:08:25.482 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.480+0000 7f3848bc8700 10 monclient: _add_conns ranks=[2,1,0,3,4] 2023-09-30T02:08:25.483 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.480+0000 7f3848bc8700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f38440a5b70 0x7f38440b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:25.483 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.480+0000 7f3848bc8700 10 monclient(hunting): picked mon.noname-b con 0x7f38440a5b70 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:08:25.483 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.480+0000 7f3848bc8700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f38440a5f40 addr v1:172.21.15.110:6789/0 2023-09-30T02:08:25.483 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.480+0000 7f3848bc8700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3844142290 addr v1:172.21.15.59:6789/0 2023-09-30T02:08:25.483 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.480+0000 7f3848bc8700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38440a63b0 con 0x7f3844142290 2023-09-30T02:08:25.483 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.480+0000 7f3848bc8700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38440a75a0 con 0x7f38440a5f40 2023-09-30T02:08:25.483 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.480+0000 7f3848bc8700 10 monclient(hunting): start opening mon connection 2023-09-30T02:08:25.483 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.481+0000 7f3848bc8700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38441426d0 con 0x7f38440a5b70 2023-09-30T02:08:25.483 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.481+0000 7f3848bc8700 10 monclient(hunting): _renew_subs 2023-09-30T02:08:25.483 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.481+0000 7f3848bc8700 10 monclient(hunting): authenticate will time out at 1950.784022s 2023-09-30T02:08:25.483 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.481+0000 7f384259c700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f38440a5b70 0x7f38440b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:25.484 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.481+0000 7f3842d9d700 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f3844142290 0x7f384413af30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:36182/0 (socket says 172.21.15.59:36182) 2023-09-30T02:08:25.484 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.481+0000 7f3842d9d700 1 -- 172.21.15.59:0/1426857683 learned_addr learned my addr 172.21.15.59:0/1426857683 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:08:25.484 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.482+0000 7f384259c700 10 monclient(hunting): get_auth_request con 0x7f38440a5b70 auth_method 0 2023-09-30T02:08:25.484 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.482+0000 7f384259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:08:25.484 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.482+0000 7f384259c700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:25.484 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.482+0000 7f384259c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:25.484 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.482+0000 7f384259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:08:25.484 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.482+0000 7f384259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:08:25.484 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.483+0000 7f384259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:08:25.485 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.485+0000 7f384159a700 1 -- 172.21.15.59:0/1426857683 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 124634387 0 0) 0x7f38440a75a0 con 0x7f38440a5f40 2023-09-30T02:08:25.485 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.485+0000 7f384159a700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:25.485 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.485+0000 7f384159a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:25.485 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.485+0000 7f384159a700 10 monclient(hunting): my global_id is 4564 2023-09-30T02:08:25.485 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.485+0000 7f384159a700 1 -- 172.21.15.59:0/1426857683 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f382c003570 con 0x7f38440a5f40 2023-09-30T02:08:25.486 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.485+0000 7f384159a700 1 -- 172.21.15.59:0/1426857683 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1191762153 0 0) 0x7f38440a63b0 con 0x7f3844142290 2023-09-30T02:08:25.486 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.485+0000 7f384159a700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:25.486 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.485+0000 7f384159a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:25.487 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.485+0000 7f384159a700 10 monclient(hunting): my global_id is 4554 2023-09-30T02:08:25.487 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.485+0000 7f384159a700 1 -- 172.21.15.59:0/1426857683 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38440a75a0 con 0x7f3844142290 2023-09-30T02:08:25.487 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.485+0000 7f384259c700 10 monclient(hunting): handle_auth_done global_id 4496 payload 274 2023-09-30T02:08:25.487 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.485+0000 7f384259c700 1 -- 172.21.15.59:0/1426857683 >> v1:172.21.15.110:6789/0 conn(0x7f38440a5f40 legacy=0x7f3844141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:25.487 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.485+0000 7f384259c700 1 -- 172.21.15.59:0/1426857683 >> v1:172.21.15.59:6789/0 conn(0x7f3844142290 legacy=0x7f384413af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:25.487 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.486+0000 7f384259c700 10 monclient: _finish_hunting 0 2023-09-30T02:08:25.487 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.486+0000 7f384259c700 1 monclient: found mon.noname-b 2023-09-30T02:08:25.487 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.486+0000 7f384259c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:08:25.487 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.486+0000 7f384259c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.59:3301/0 2023-09-30T02:08:25.487 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.486+0000 7f384259c700 1 -- 172.21.15.59:0/1426857683 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38440af3f0 con 0x7f38440a5b70 2023-09-30T02:08:25.488 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.488+0000 7f384259c700 1 --2- 172.21.15.59:0/1426857683 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f38440a5b70 0x7f38440b1500 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3834009910 tx=0x7f38340091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f65c7a0a89327560 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:08:25.493 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.492+0000 7f384159a700 1 -- 172.21.15.59:0/1426857683 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3834011630 con 0x7f38440a5b70 2023-09-30T02:08:25.493 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.492+0000 7f384159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:08:25.493 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.492+0000 7f384159a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:08:25.493 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.492+0000 7f384159a700 10 monclient: dump: 2023-09-30T02:08:25.493 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:08:25.493 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:25.493 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:25.493 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:25.494 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:08:25.494 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:25.494 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:08:25.494 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:08:25.494 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:08:25.494 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:25.494 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.492+0000 7f384159a700 10 monclient: _finish_auth 0 2023-09-30T02:08:25.494 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.492+0000 7f384159a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:08:25.494 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.492+0000 7f384159a700 1 -- 172.21.15.59:0/1426857683 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3834011790 con 0x7f38440a5b70 2023-09-30T02:08:25.494 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.492+0000 7f384159a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:08:25.494 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.492+0000 7f384159a700 1 -- 172.21.15.59:0/1426857683 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3834011b30 con 0x7f38440a5b70 2023-09-30T02:08:25.494 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.492+0000 7f384159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:08:25.495 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.492+0000 7f384159a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:08:25.495 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.492+0000 7f384159a700 10 monclient: dump: 2023-09-30T02:08:25.495 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:08:25.495 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:25.495 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:25.495 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:25.495 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:08:25.495 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:25.495 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:08:25.495 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:08:25.495 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:08:25.495 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:25.495 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.492+0000 7f3848bc8700 5 monclient: authenticate success, global_id 4496 2023-09-30T02:08:25.496 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.492+0000 7f3848bc8700 10 monclient: get_monmap_and_config success 2023-09-30T02:08:25.496 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.492+0000 7f3848bc8700 4 set_mon_vals no callback set 2023-09-30T02:08:25.498 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.493+0000 7f3848bc8700 10 monclient: shutdown 2023-09-30T02:08:25.499 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.493+0000 7f3848bc8700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:08:25.499 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.493+0000 7f3848bc8700 1 -- 172.21.15.59:0/1426857683 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f38440a5b70 msgr2=0x7f38440b1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:25.499 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.493+0000 7f3848bc8700 1 --2- 172.21.15.59:0/1426857683 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f38440a5b70 0x7f38440b1500 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3834009910 tx=0x7f38340091a0 comp rx=0 tx=0).stop 2023-09-30T02:08:25.499 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.493+0000 7f3848bc8700 1 -- 172.21.15.59:0/1426857683 shutdown_connections 2023-09-30T02:08:25.499 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.493+0000 7f3848bc8700 1 --2- 172.21.15.59:0/1426857683 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f38440a5b70 0x7f38440b1500 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:25.499 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.493+0000 7f3848bc8700 1 -- 172.21.15.59:0/1426857683 >> 172.21.15.59:0/1426857683 conn(0x7f38440631f0 msgr2=0x7f38440635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:08:25.499 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.493+0000 7f3848bc8700 1 -- 172.21.15.59:0/1426857683 shutdown_connections 2023-09-30T02:08:25.499 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.493+0000 7f3848bc8700 1 -- 172.21.15.59:0/1426857683 wait complete. 2023-09-30T02:08:25.499 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.494+0000 7f3848bc8700 10 monclient: build_initial_monmap 2023-09-30T02:08:25.499 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.494+0000 7f3848bc8700 10 monclient: monmap: 2023-09-30T02:08:25.499 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:08:25.499 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:08:25.500 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:08:25.500 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:08:25.500 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:08:25.500 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:08:25.500 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:08:25.500 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:08:25.500 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:08:25.500 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:25.500 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.494+0000 7f3848bc8700 1 librados: starting msgr at 2023-09-30T02:08:25.500 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.494+0000 7f3848bc8700 1 librados: starting objecter 2023-09-30T02:08:25.500 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.494+0000 7f3848bc8700 1 Processor -- start 2023-09-30T02:08:25.500 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.494+0000 7f3848bc8700 1 -- start start 2023-09-30T02:08:25.501 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.494+0000 7f3848bc8700 1 librados: setting wanted keys 2023-09-30T02:08:25.501 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.494+0000 7f3848bc8700 1 librados: calling monclient init 2023-09-30T02:08:25.501 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.494+0000 7f3848bc8700 10 monclient: init 2023-09-30T02:08:25.501 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.495+0000 7f3848bc8700 10 monclient: _reopen_session rank -1 2023-09-30T02:08:25.501 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.495+0000 7f3848bc8700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T02:08:25.501 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.495+0000 7f3848bc8700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f38440a5b70 0x7f38441514c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:25.501 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.495+0000 7f3848bc8700 10 monclient(hunting): picked mon.noname-b con 0x7f38440a5b70 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:08:25.501 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.495+0000 7f3848bc8700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f38440a5f40 0x7f3844151a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:25.501 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.495+0000 7f3848bc8700 10 monclient(hunting): picked mon.noname-c con 0x7f38440a5f40 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:08:25.501 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.495+0000 7f3848bc8700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3844142290 0x7f384413ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:25.501 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.495+0000 7f3848bc8700 10 monclient(hunting): picked mon.noname-a con 0x7f3844142290 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:08:25.501 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.495+0000 7f3848bc8700 10 monclient(hunting): start opening mon connection 2023-09-30T02:08:25.501 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.495+0000 7f3848bc8700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f384413c030 con 0x7f3844142290 2023-09-30T02:08:25.502 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.495+0000 7f3848bc8700 10 monclient(hunting): start opening mon connection 2023-09-30T02:08:25.502 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.495+0000 7f3848bc8700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3844151f40 con 0x7f38440a5b70 2023-09-30T02:08:25.502 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.495+0000 7f3848bc8700 10 monclient(hunting): start opening mon connection 2023-09-30T02:08:25.502 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.495+0000 7f3848bc8700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f384413e1c0 con 0x7f38440a5f40 2023-09-30T02:08:25.502 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.495+0000 7f3848bc8700 10 monclient(hunting): _renew_subs 2023-09-30T02:08:25.502 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.495+0000 7f3848bc8700 10 monclient(hunting): authenticate will time out at 1950.798568s 2023-09-30T02:08:25.502 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.496+0000 7f384259c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f38440a5b70 0x7f38441514c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:25.502 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.496+0000 7f3842d9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3844142290 0x7f384413ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:25.502 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.496+0000 7f3841d9b700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f38440a5f40 0x7f3844151a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:25.502 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.496+0000 7f384259c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f38440a5b70 0x7f38441514c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:51690/0 (socket says 172.21.15.59:51690) 2023-09-30T02:08:25.502 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.496+0000 7f384259c700 1 -- 172.21.15.59:0/1146902870 learned_addr learned my addr 172.21.15.59:0/1146902870 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:08:25.502 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.496+0000 7f384259c700 10 monclient(hunting): get_auth_request con 0x7f38440a5b70 auth_method 0 2023-09-30T02:08:25.503 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.496+0000 7f384259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:08:25.503 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.496+0000 7f384259c700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:25.503 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.496+0000 7f384259c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:25.503 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.496+0000 7f3842d9d700 10 monclient(hunting): get_auth_request con 0x7f3844142290 auth_method 0 2023-09-30T02:08:25.503 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.496+0000 7f3842d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:08:25.503 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.496+0000 7f3842d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:25.503 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.496+0000 7f3842d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:25.503 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.496+0000 7f3841d9b700 10 monclient(hunting): get_auth_request con 0x7f38440a5f40 auth_method 0 2023-09-30T02:08:25.503 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.496+0000 7f3841d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:08:25.503 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.496+0000 7f3841d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:25.503 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.496+0000 7f3841d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:25.503 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.497+0000 7f384259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:08:25.504 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.497+0000 7f384259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:08:25.504 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.497+0000 7f384259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:08:25.504 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.497+0000 7f3842d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:08:25.504 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.497+0000 7f3842d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:08:25.504 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.497+0000 7f3842d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:08:25.504 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.497+0000 7f3841d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:08:25.504 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.497+0000 7f3841d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:08:25.504 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.497+0000 7f3841d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:08:25.505 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.501+0000 7f384259c700 10 monclient(hunting): handle_auth_done global_id 4567 payload 824 2023-09-30T02:08:25.505 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.501+0000 7f384259c700 1 -- 172.21.15.59:0/1146902870 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f38440a5f40 msgr2=0x7f3844151a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:25.505 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.501+0000 7f384259c700 1 --2- 172.21.15.59:0/1146902870 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f38440a5f40 0x7f3844151a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:25.505 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.501+0000 7f384259c700 1 -- 172.21.15.59:0/1146902870 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3844142290 msgr2=0x7f384413ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:25.505 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.501+0000 7f384259c700 1 --2- 172.21.15.59:0/1146902870 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3844142290 0x7f384413ff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:25.505 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.501+0000 7f384259c700 10 monclient: _finish_hunting 0 2023-09-30T02:08:25.505 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.501+0000 7f384259c700 1 monclient: found mon.noname-b 2023-09-30T02:08:25.505 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.501+0000 7f384259c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:08:25.506 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.501+0000 7f384259c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:08:25.506 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.501+0000 7f384259c700 1 -- 172.21.15.59:0/1146902870 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3834009420 con 0x7f38440a5b70 2023-09-30T02:08:25.506 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.503+0000 7f384259c700 1 --2- 172.21.15.59:0/1146902870 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f38440a5b70 0x7f38441514c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f38440b1ff0 tx=0x7f3834017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:08:25.506 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.503+0000 7f38327fc700 10 client.?.objecter ms_handle_connect 0x7f38440a5b70 2023-09-30T02:08:25.506 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.503+0000 7f38327fc700 10 client.?.objecter resend_mon_ops 2023-09-30T02:08:25.508 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.507+0000 7f38327fc700 1 -- 172.21.15.59:0/1146902870 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3834011900 con 0x7f38440a5b70 2023-09-30T02:08:25.508 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.507+0000 7f38327fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:08:25.508 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.507+0000 7f38327fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:08:25.508 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.507+0000 7f38327fc700 10 monclient: dump: 2023-09-30T02:08:25.508 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:08:25.508 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:25.508 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:25.508 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:25.509 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:08:25.509 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:25.509 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:08:25.509 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:08:25.509 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:08:25.509 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:25.509 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.507+0000 7f38327fc700 10 monclient: _finish_auth 0 2023-09-30T02:08:25.509 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.507+0000 7f38327fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:08:25.509 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.507+0000 7f3848bc8700 5 monclient: authenticate success, global_id 4567 2023-09-30T02:08:25.509 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.507+0000 7f3848bc8700 10 monclient: _renew_subs 2023-09-30T02:08:25.509 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.507+0000 7f3848bc8700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:08:25.509 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.507+0000 7f3848bc8700 1 -- 172.21.15.59:0/1146902870 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3844140480 con 0x7f38440a5b70 2023-09-30T02:08:25.510 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.507+0000 7f3848bc8700 10 client.4567.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:08:25.510 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.507+0000 7f3848bc8700 10 monclient: _renew_subs 2023-09-30T02:08:25.510 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.507+0000 7f3848bc8700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:08:25.510 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.507+0000 7f3848bc8700 1 -- 172.21.15.59:0/1146902870 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3844140920 con 0x7f38440a5b70 2023-09-30T02:08:25.510 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.507+0000 7f3848bc8700 1 librados: init done 2023-09-30T02:08:25.510 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.507+0000 7f38327fc700 1 -- 172.21.15.59:0/1146902870 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f383402c3f0 con 0x7f38440a5b70 2023-09-30T02:08:25.510 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.507+0000 7f38327fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:08:25.510 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.507+0000 7f3840d99700 4 set_mon_vals no callback set 2023-09-30T02:08:25.511 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.508+0000 7f38327fc700 1 -- 172.21.15.59:0/1146902870 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f383402c560 con 0x7f38440a5b70 2023-09-30T02:08:25.511 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.508+0000 7f38327fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:08:25.511 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.508+0000 7f38327fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:08:25.511 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.508+0000 7f38327fc700 10 monclient: dump: 2023-09-30T02:08:25.511 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:08:25.511 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:25.511 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:25.512 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:25.512 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:08:25.512 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:25.512 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:08:25.512 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:08:25.512 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:08:25.512 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:25.512 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.508+0000 7f3848bc8700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:08:25.512 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.508+0000 7f3848bc8700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:08:25.512 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.508+0000 7f3848bc8700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:08:25.512 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.508+0000 7f3848bc8700 1 -- 172.21.15.59:0/1146902870 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38280050b0 con 0x7f38440a5b70 2023-09-30T02:08:25.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.529+0000 7f38327fc700 1 -- 172.21.15.59:0/1146902870 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f383402c950 con 0x7f38440a5b70 2023-09-30T02:08:25.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.530+0000 7f38327fc700 1 --2- 172.21.15.59:0/1146902870 >> v2:172.21.15.110:6800/115870 conn(0x7f3824068890 0x7f382406ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:25.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.530+0000 7f3841d9b700 1 --2- 172.21.15.59:0/1146902870 >> v2:172.21.15.110:6800/115870 conn(0x7f3824068890 0x7f382406ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:25.531 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.530+0000 7f3841d9b700 10 monclient: get_auth_request con 0x7f3824068890 auth_method 0 2023-09-30T02:08:25.531 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.531+0000 7f3841d9b700 1 --2- 172.21.15.59:0/1146902870 >> v2:172.21.15.110:6800/115870 conn(0x7f3824068890 0x7f382406ad40 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f3844063a20 tx=0x7f3838007d40 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:08:25.531 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.531+0000 7f38327fc700 10 client.4567.objecter ms_handle_connect 0x7f3824068890 2023-09-30T02:08:25.534 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.533+0000 7f38327fc700 1 -- 172.21.15.59:0/1146902870 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f38340576b0 con 0x7f38440a5b70 2023-09-30T02:08:25.534 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.533+0000 7f38327fc700 10 client.4567.objecter ms_dispatch 0x7f3844001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:08:25.534 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.533+0000 7f38327fc700 3 client.4567.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:08:25.534 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.533+0000 7f38327fc700 3 client.4567.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:08:25.534 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.534+0000 7f38327fc700 20 client.4567.objecter dump_active .. 0 homeless 2023-09-30T02:08:25.777 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.776+0000 7f38327fc700 1 -- 172.21.15.59:0/1146902870 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f38340532f0 con 0x7f38440a5b70 2023-09-30T02:08:25.777 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.776+0000 7f38327fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:08:25.777 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:25.776+0000 7f38327fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:08:26.052 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:26.051+0000 7f3848bc8700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:08:26.052 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:26.051+0000 7f3848bc8700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:08:26.052 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:26.051+0000 7f3848bc8700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:08:26.052 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:26.051+0000 7f3848bc8700 1 -- 172.21.15.59:0/1146902870 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f3828005630 con 0x7f38440a5b70 2023-09-30T02:08:26.057 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:26.057+0000 7f38327fc700 1 -- 172.21.15.59:0/1146902870 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f3834015100 con 0x7f38440a5b70 2023-09-30T02:08:26.057 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:26.057+0000 7f38327fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:08:26.057 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:26.057+0000 7f38327fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:08:26.057 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:08:26.057 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:08:26.060 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:26.059+0000 7f3848bc8700 10 librados: watch_flush enter 2023-09-30T02:08:26.060 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:26.059+0000 7f3848bc8700 10 librados: watch_flush exit 2023-09-30T02:08:26.061 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:26.059+0000 7f3848bc8700 20 client.4567.objecter shutdown clearing up homeless session... 2023-09-30T02:08:26.061 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:26.059+0000 7f3848bc8700 10 client.4567.objecter successfully canceled tick 2023-09-30T02:08:26.061 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:26.059+0000 7f3848bc8700 1 -- 172.21.15.59:0/1146902870 >> v2:172.21.15.110:6800/115870 conn(0x7f3824068890 msgr2=0x7f382406ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:26.061 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:26.059+0000 7f3848bc8700 1 --2- 172.21.15.59:0/1146902870 >> v2:172.21.15.110:6800/115870 conn(0x7f3824068890 0x7f382406ad40 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f3844063a20 tx=0x7f3838007d40 comp rx=0 tx=0).stop 2023-09-30T02:08:26.061 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:26.059+0000 7f3848bc8700 10 monclient: shutdown 2023-09-30T02:08:26.061 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:26.059+0000 7f3848bc8700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:08:26.061 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:26.059+0000 7f3848bc8700 1 -- 172.21.15.59:0/1146902870 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f38440a5b70 msgr2=0x7f38441514c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:26.061 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:26.059+0000 7f3848bc8700 1 --2- 172.21.15.59:0/1146902870 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f38440a5b70 0x7f38441514c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f38440b1ff0 tx=0x7f3834017000 comp rx=0 tx=0).stop 2023-09-30T02:08:26.061 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:26.059+0000 7f3848bc8700 1 -- 172.21.15.59:0/1146902870 shutdown_connections 2023-09-30T02:08:26.061 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:26.059+0000 7f3848bc8700 1 --2- 172.21.15.59:0/1146902870 >> v2:172.21.15.110:6800/115870 conn(0x7f3824068890 0x7f382406ad40 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:26.061 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:26.059+0000 7f3848bc8700 1 --2- 172.21.15.59:0/1146902870 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f38440a5b70 0x7f38441514c0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:26.061 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:26.059+0000 7f3848bc8700 1 --2- 172.21.15.59:0/1146902870 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3844142290 0x7f384413ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:26.062 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:26.059+0000 7f3848bc8700 1 --2- 172.21.15.59:0/1146902870 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f38440a5f40 0x7f3844151a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:26.062 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:26.059+0000 7f3848bc8700 1 -- 172.21.15.59:0/1146902870 >> 172.21.15.59:0/1146902870 conn(0x7f38440631f0 msgr2=0x7f38440b0be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:08:26.062 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:26.060+0000 7f3848bc8700 1 -- 172.21.15.59:0/1146902870 shutdown_connections 2023-09-30T02:08:26.062 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:26.060+0000 7f3848bc8700 1 -- 172.21.15.59:0/1146902870 wait complete. 2023-09-30T02:08:26.062 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:26.060+0000 7f3848bc8700 1 librados: shutdown 2023-09-30T02:08:29.075 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:08:29.212 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.211+0000 7f311568b700 10 monclient: get_monmap_and_config 2023-09-30T02:08:29.212 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.211+0000 7f311568b700 10 monclient: build_initial_monmap 2023-09-30T02:08:29.212 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.211+0000 7f311568b700 10 monclient: monmap: 2023-09-30T02:08:29.212 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:08:29.212 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:29.212 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:08:29.213092+0000 2023-09-30T02:08:29.212 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:08:29.213092+0000 2023-09-30T02:08:29.212 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:08:29.213 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:29.213 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:08:29.213 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:08:29.213 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:08:29.213 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:08:29.213 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:08:29.213 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:29.214 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.212+0000 7f311568b700 1 Processor -- start 2023-09-30T02:08:29.215 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.212+0000 7f311568b700 1 -- start start 2023-09-30T02:08:29.215 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.212+0000 7f311568b700 10 monclient: init 2023-09-30T02:08:29.215 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.212+0000 7f311568b700 10 monclient: _reopen_session rank -1 2023-09-30T02:08:29.215 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.212+0000 7f311568b700 10 monclient: _add_conns ranks=[1,0,3,2,4] 2023-09-30T02:08:29.215 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.213+0000 7f311568b700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f31100aeca0 addr v1:172.21.15.110:6789/0 2023-09-30T02:08:29.215 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.213+0000 7f311568b700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f31100a54e0 addr v1:172.21.15.59:6789/0 2023-09-30T02:08:29.215 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.213+0000 7f311568b700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f31100ae210 0x7f311013e150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:29.215 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.213+0000 7f311568b700 10 monclient(hunting): picked mon.noname-a con 0x7f31100ae210 addr v2:172.21.15.59:3300/0 2023-09-30T02:08:29.215 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.213+0000 7f311568b700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31100af070 con 0x7f31100a54e0 2023-09-30T02:08:29.215 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.213+0000 7f311568b700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31100ae620 con 0x7f31100aeca0 2023-09-30T02:08:29.216 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.213+0000 7f311568b700 10 monclient(hunting): start opening mon connection 2023-09-30T02:08:29.216 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.213+0000 7f311568b700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31100a7230 con 0x7f31100ae210 2023-09-30T02:08:29.216 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.213+0000 7f311568b700 10 monclient(hunting): _renew_subs 2023-09-30T02:08:29.216 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.213+0000 7f311568b700 10 monclient(hunting): authenticate will time out at 1954.516107s 2023-09-30T02:08:29.216 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.213+0000 7f310f7fe700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f31100ae210 0x7f311013e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:29.216 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.214+0000 7f310e7fc700 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f31100a54e0 0x7f31100adb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:55824/0 (socket says 172.21.15.59:55824) 2023-09-30T02:08:29.216 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.214+0000 7f310e7fc700 1 -- 172.21.15.59:0/1251763791 learned_addr learned my addr 172.21.15.59:0/1251763791 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:08:29.216 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.214+0000 7f310f7fe700 10 monclient(hunting): get_auth_request con 0x7f31100ae210 auth_method 0 2023-09-30T02:08:29.216 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.214+0000 7f310f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:08:29.216 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.214+0000 7f310f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:29.216 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.214+0000 7f310f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:29.217 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.215+0000 7f310f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:08:29.217 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.215+0000 7f310f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:08:29.217 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.215+0000 7f310f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:08:29.218 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.218+0000 7f310f7fe700 10 monclient(hunting): handle_auth_done global_id 4560 payload 274 2023-09-30T02:08:29.218 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.218+0000 7f310f7fe700 1 -- 172.21.15.59:0/1251763791 >> v1:172.21.15.110:6789/0 conn(0x7f31100aeca0 legacy=0x7f31100b1400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:29.218 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.218+0000 7f310f7fe700 1 -- 172.21.15.59:0/1251763791 >> v1:172.21.15.59:6789/0 conn(0x7f31100a54e0 legacy=0x7f31100adb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:29.219 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.218+0000 7f310f7fe700 10 monclient: _finish_hunting 0 2023-09-30T02:08:29.219 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.218+0000 7f310f7fe700 1 monclient: found mon.noname-a 2023-09-30T02:08:29.219 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.218+0000 7f310f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:08:29.219 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.218+0000 7f310f7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.59:3300/0 2023-09-30T02:08:29.219 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.218+0000 7f310f7fe700 1 -- 172.21.15.59:0/1251763791 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31100a7410 con 0x7f31100ae210 2023-09-30T02:08:29.220 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.220+0000 7f310f7fe700 1 --2- 172.21.15.59:0/1251763791 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31100ae210 0x7f311013e150 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3108009e60 tx=0x7f3108009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ced81774de7b0b7b server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:08:29.228 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.227+0000 7f310dffb700 1 -- 172.21.15.59:0/1251763791 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f310800e070 con 0x7f31100ae210 2023-09-30T02:08:29.228 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.227+0000 7f310dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:08:29.228 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.227+0000 7f310dffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T02:08:29.228 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.227+0000 7f310dffb700 10 monclient: dump: 2023-09-30T02:08:29.228 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:08:29.228 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:29.229 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:29.229 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:29.229 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:08:29.229 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:29.229 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:08:29.229 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:08:29.229 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:08:29.229 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:29.229 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.227+0000 7f310dffb700 10 monclient: _finish_auth 0 2023-09-30T02:08:29.229 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.227+0000 7f310dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:08:29.229 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.227+0000 7f310dffb700 1 -- 172.21.15.59:0/1251763791 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f310800bc30 con 0x7f31100ae210 2023-09-30T02:08:29.229 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.227+0000 7f310dffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:08:29.230 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.227+0000 7f310dffb700 1 -- 172.21.15.59:0/1251763791 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3108012430 con 0x7f31100ae210 2023-09-30T02:08:29.230 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.227+0000 7f310dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:08:29.230 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.227+0000 7f310dffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T02:08:29.230 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.227+0000 7f310dffb700 10 monclient: dump: 2023-09-30T02:08:29.230 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:08:29.230 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:29.230 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:29.230 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:29.230 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:08:29.230 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:29.230 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:08:29.231 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:08:29.231 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:08:29.231 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:29.231 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.227+0000 7f311568b700 5 monclient: authenticate success, global_id 4560 2023-09-30T02:08:29.231 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.227+0000 7f311568b700 10 monclient: get_monmap_and_config success 2023-09-30T02:08:29.231 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.227+0000 7f311568b700 4 set_mon_vals no callback set 2023-09-30T02:08:29.234 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.228+0000 7f311568b700 10 monclient: shutdown 2023-09-30T02:08:29.234 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.228+0000 7f311568b700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:08:29.234 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.228+0000 7f311568b700 1 -- 172.21.15.59:0/1251763791 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31100ae210 msgr2=0x7f311013e150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:29.234 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.228+0000 7f311568b700 1 --2- 172.21.15.59:0/1251763791 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31100ae210 0x7f311013e150 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3108009e60 tx=0x7f3108009780 comp rx=0 tx=0).stop 2023-09-30T02:08:29.237 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.228+0000 7f311568b700 1 -- 172.21.15.59:0/1251763791 shutdown_connections 2023-09-30T02:08:29.237 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.228+0000 7f311568b700 1 --2- 172.21.15.59:0/1251763791 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31100ae210 0x7f311013e150 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:29.237 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.228+0000 7f311568b700 1 -- 172.21.15.59:0/1251763791 >> 172.21.15.59:0/1251763791 conn(0x7f31100631f0 msgr2=0x7f31100635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:08:29.237 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.228+0000 7f311568b700 1 -- 172.21.15.59:0/1251763791 shutdown_connections 2023-09-30T02:08:29.237 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.228+0000 7f311568b700 1 -- 172.21.15.59:0/1251763791 wait complete. 2023-09-30T02:08:29.237 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.229+0000 7f311568b700 10 monclient: build_initial_monmap 2023-09-30T02:08:29.237 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.229+0000 7f311568b700 10 monclient: monmap: 2023-09-30T02:08:29.237 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:08:29.237 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:08:29.238 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:08:29.238 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:08:29.238 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:08:29.238 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:08:29.238 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:08:29.238 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:08:29.238 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:08:29.238 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:29.238 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.229+0000 7f311568b700 1 librados: starting msgr at 2023-09-30T02:08:29.238 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.229+0000 7f311568b700 1 librados: starting objecter 2023-09-30T02:08:29.238 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.229+0000 7f311568b700 1 Processor -- start 2023-09-30T02:08:29.238 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.230+0000 7f311568b700 1 -- start start 2023-09-30T02:08:29.239 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.230+0000 7f311568b700 1 librados: setting wanted keys 2023-09-30T02:08:29.239 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.230+0000 7f311568b700 1 librados: calling monclient init 2023-09-30T02:08:29.239 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.230+0000 7f311568b700 10 monclient: init 2023-09-30T02:08:29.239 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.230+0000 7f311568b700 10 monclient: _reopen_session rank -1 2023-09-30T02:08:29.239 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.230+0000 7f311568b700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T02:08:29.239 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.230+0000 7f311568b700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f31100a54e0 0x7f31100a95b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:29.239 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.230+0000 7f311568b700 10 monclient(hunting): picked mon.noname-c con 0x7f31100a54e0 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:08:29.239 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.230+0000 7f311568b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f31100ae210 0x7f311014e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:29.239 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.230+0000 7f311568b700 10 monclient(hunting): picked mon.noname-b con 0x7f31100ae210 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:08:29.239 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.230+0000 7f311568b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31100aeca0 0x7f311014ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:29.239 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.230+0000 7f311568b700 10 monclient(hunting): picked mon.noname-a con 0x7f31100aeca0 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:08:29.240 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.230+0000 7f311568b700 10 monclient(hunting): start opening mon connection 2023-09-30T02:08:29.240 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.230+0000 7f311568b700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31100b1b10 con 0x7f31100aeca0 2023-09-30T02:08:29.240 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.230+0000 7f311568b700 10 monclient(hunting): start opening mon connection 2023-09-30T02:08:29.240 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.230+0000 7f311568b700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f311014e2c0 con 0x7f31100ae210 2023-09-30T02:08:29.240 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.230+0000 7f311568b700 10 monclient(hunting): start opening mon connection 2023-09-30T02:08:29.240 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.230+0000 7f311568b700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f311014f1d0 con 0x7f31100a54e0 2023-09-30T02:08:29.240 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.230+0000 7f311568b700 10 monclient(hunting): _renew_subs 2023-09-30T02:08:29.240 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.230+0000 7f311568b700 10 monclient(hunting): authenticate will time out at 1954.533822s 2023-09-30T02:08:29.240 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.231+0000 7f310f7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31100aeca0 0x7f311014ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:29.240 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.231+0000 7f310effd700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f31100a54e0 0x7f31100a95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:29.240 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.231+0000 7f310e7fc700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f31100ae210 0x7f311014e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:29.241 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.231+0000 7f310f7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31100aeca0 0x7f311014ec90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58336/0 (socket says 172.21.15.59:58336) 2023-09-30T02:08:29.241 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.231+0000 7f310f7fe700 1 -- 172.21.15.59:0/1836154178 learned_addr learned my addr 172.21.15.59:0/1836154178 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:08:29.241 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.231+0000 7f310f7fe700 10 monclient(hunting): get_auth_request con 0x7f31100aeca0 auth_method 0 2023-09-30T02:08:29.241 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.231+0000 7f310f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:08:29.241 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.231+0000 7f310f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:29.241 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.231+0000 7f310f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:29.241 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.232+0000 7f310e7fc700 10 monclient(hunting): get_auth_request con 0x7f31100ae210 auth_method 0 2023-09-30T02:08:29.241 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.232+0000 7f310e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:08:29.241 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.232+0000 7f310e7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:29.241 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.232+0000 7f310e7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:29.241 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.232+0000 7f310effd700 10 monclient(hunting): get_auth_request con 0x7f31100a54e0 auth_method 0 2023-09-30T02:08:29.241 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.232+0000 7f310effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:08:29.242 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.232+0000 7f310effd700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:29.242 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.232+0000 7f310effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:29.242 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.232+0000 7f310f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:08:29.242 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.232+0000 7f310f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:08:29.242 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.232+0000 7f310f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:08:29.242 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.232+0000 7f310e7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:08:29.242 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.232+0000 7f310e7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:08:29.242 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.232+0000 7f310e7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:08:29.242 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.233+0000 7f310effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:08:29.242 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.233+0000 7f310effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:08:29.242 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.233+0000 7f310effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:08:29.243 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.236+0000 7f310e7fc700 10 monclient(hunting): handle_auth_done global_id 4573 payload 824 2023-09-30T02:08:29.243 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.236+0000 7f310e7fc700 1 -- 172.21.15.59:0/1836154178 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f31100a54e0 msgr2=0x7f31100a95b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:29.244 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.236+0000 7f310e7fc700 1 --2- 172.21.15.59:0/1836154178 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f31100a54e0 0x7f31100a95b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:29.244 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.236+0000 7f310e7fc700 1 -- 172.21.15.59:0/1836154178 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31100aeca0 msgr2=0x7f311014ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:29.244 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.236+0000 7f310e7fc700 1 --2- 172.21.15.59:0/1836154178 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31100aeca0 0x7f311014ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:29.244 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.236+0000 7f310e7fc700 10 monclient: _finish_hunting 0 2023-09-30T02:08:29.244 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.236+0000 7f310e7fc700 1 monclient: found mon.noname-b 2023-09-30T02:08:29.244 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.236+0000 7f310e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:08:29.244 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.236+0000 7f310e7fc700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:08:29.244 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.236+0000 7f310e7fc700 1 -- 172.21.15.59:0/1836154178 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3108009a20 con 0x7f31100ae210 2023-09-30T02:08:29.244 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.239+0000 7f310e7fc700 1 --2- 172.21.15.59:0/1836154178 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f31100ae210 0x7f311014e750 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f30f800c7c0 tx=0x7f30f8007eb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:08:29.244 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.239+0000 7f3106ffd700 10 client.?.objecter ms_handle_connect 0x7f31100ae210 2023-09-30T02:08:29.244 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.239+0000 7f3106ffd700 10 client.?.objecter resend_mon_ops 2023-09-30T02:08:29.246 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.244+0000 7f3106ffd700 1 -- 172.21.15.59:0/1836154178 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30f8005b30 con 0x7f31100ae210 2023-09-30T02:08:29.246 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.244+0000 7f3106ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:08:29.246 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.244+0000 7f3106ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:08:29.246 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.244+0000 7f3106ffd700 10 monclient: dump: 2023-09-30T02:08:29.246 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:08:29.246 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:29.246 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:29.246 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:29.246 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:08:29.247 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:29.247 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:08:29.247 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:08:29.247 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:08:29.247 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:29.247 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.244+0000 7f3106ffd700 10 monclient: _finish_auth 0 2023-09-30T02:08:29.247 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.244+0000 7f3106ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:08:29.247 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.244+0000 7f3106ffd700 1 -- 172.21.15.59:0/1836154178 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30f8005c90 con 0x7f31100ae210 2023-09-30T02:08:29.247 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.244+0000 7f311568b700 5 monclient: authenticate success, global_id 4573 2023-09-30T02:08:29.247 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.244+0000 7f311568b700 10 monclient: _renew_subs 2023-09-30T02:08:29.247 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.244+0000 7f311568b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:08:29.248 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.244+0000 7f311568b700 1 -- 172.21.15.59:0/1836154178 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3110140de0 con 0x7f31100ae210 2023-09-30T02:08:29.248 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.244+0000 7f3106ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:08:29.248 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.244+0000 7f3106ffd700 1 -- 172.21.15.59:0/1836154178 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30f8004a80 con 0x7f31100ae210 2023-09-30T02:08:29.248 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.244+0000 7f3106ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:08:29.248 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.244+0000 7f3106ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:08:29.248 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.244+0000 7f3106ffd700 10 monclient: dump: 2023-09-30T02:08:29.248 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:08:29.248 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:29.248 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:29.248 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:29.248 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:08:29.248 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:29.249 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:08:29.249 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:08:29.249 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:08:29.249 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:29.249 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.244+0000 7f311568b700 10 client.4573.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:08:29.249 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.244+0000 7f310d7fa700 4 set_mon_vals no callback set 2023-09-30T02:08:29.249 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.244+0000 7f311568b700 10 monclient: _renew_subs 2023-09-30T02:08:29.249 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.244+0000 7f311568b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:08:29.249 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.244+0000 7f311568b700 1 -- 172.21.15.59:0/1836154178 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31100a7ee0 con 0x7f31100ae210 2023-09-30T02:08:29.249 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.244+0000 7f311568b700 1 librados: init done 2023-09-30T02:08:29.249 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.245+0000 7f311568b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:08:29.250 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.245+0000 7f311568b700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:08:29.250 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.245+0000 7f311568b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:08:29.250 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.245+0000 7f311568b700 1 -- 172.21.15.59:0/1836154178 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3110072b30 con 0x7f31100ae210 2023-09-30T02:08:29.498 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.498+0000 7f3106ffd700 1 -- 172.21.15.59:0/1836154178 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f30f80070d0 con 0x7f31100ae210 2023-09-30T02:08:29.499 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.498+0000 7f3106ffd700 1 --2- 172.21.15.59:0/1836154178 >> v2:172.21.15.110:6800/115870 conn(0x7f30f0068910 0x7f30f006adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:29.499 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.498+0000 7f3106ffd700 1 -- 172.21.15.59:0/1836154178 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f30f8083c60 con 0x7f31100ae210 2023-09-30T02:08:29.500 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.498+0000 7f3106ffd700 10 client.4573.objecter ms_dispatch 0x7f3110001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:08:29.500 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.498+0000 7f3106ffd700 3 client.4573.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:08:29.500 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.498+0000 7f3106ffd700 3 client.4573.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:08:29.500 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.499+0000 7f3106ffd700 20 client.4573.objecter dump_active .. 0 homeless 2023-09-30T02:08:29.500 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.499+0000 7f310effd700 1 --2- 172.21.15.59:0/1836154178 >> v2:172.21.15.110:6800/115870 conn(0x7f30f0068910 0x7f30f006adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:29.500 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.499+0000 7f310effd700 10 monclient: get_auth_request con 0x7f30f0068910 auth_method 0 2023-09-30T02:08:29.500 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.499+0000 7f310effd700 1 --2- 172.21.15.59:0/1836154178 >> v2:172.21.15.110:6800/115870 conn(0x7f30f0068910 0x7f30f006adc0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f3100000c00 tx=0x7f3100007ea0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:08:29.500 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.499+0000 7f3106ffd700 10 client.4573.objecter ms_handle_connect 0x7f30f0068910 2023-09-30T02:08:29.507 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.507+0000 7f3106ffd700 1 -- 172.21.15.59:0/1836154178 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f30f8051500 con 0x7f31100ae210 2023-09-30T02:08:29.507 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.507+0000 7f3106ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:08:29.507 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.507+0000 7f3106ffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:08:29.762 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.761+0000 7f311568b700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:08:29.762 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.761+0000 7f311568b700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:08:29.762 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.761+0000 7f311568b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:08:29.762 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.761+0000 7f311568b700 1 -- 172.21.15.59:0/1836154178 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f31100a7ee0 con 0x7f31100ae210 2023-09-30T02:08:29.766 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.767+0000 7f3106ffd700 1 -- 172.21.15.59:0/1836154178 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f31100a7ee0 con 0x7f31100ae210 2023-09-30T02:08:29.766 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.767+0000 7f3106ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:08:29.766 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.767+0000 7f3106ffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:08:29.767 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:08:29.767 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:08:29.769 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.768+0000 7f311568b700 10 librados: watch_flush enter 2023-09-30T02:08:29.769 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.768+0000 7f311568b700 10 librados: watch_flush exit 2023-09-30T02:08:29.769 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.768+0000 7f311568b700 20 client.4573.objecter shutdown clearing up homeless session... 2023-09-30T02:08:29.769 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.768+0000 7f311568b700 10 client.4573.objecter successfully canceled tick 2023-09-30T02:08:29.770 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.768+0000 7f311568b700 1 -- 172.21.15.59:0/1836154178 >> v2:172.21.15.110:6800/115870 conn(0x7f30f0068910 msgr2=0x7f30f006adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:29.770 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.768+0000 7f311568b700 1 --2- 172.21.15.59:0/1836154178 >> v2:172.21.15.110:6800/115870 conn(0x7f30f0068910 0x7f30f006adc0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f3100000c00 tx=0x7f3100007ea0 comp rx=0 tx=0).stop 2023-09-30T02:08:29.770 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.768+0000 7f311568b700 10 monclient: shutdown 2023-09-30T02:08:29.770 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.768+0000 7f311568b700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:08:29.770 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.768+0000 7f311568b700 1 -- 172.21.15.59:0/1836154178 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f31100ae210 msgr2=0x7f311014e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:29.770 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.768+0000 7f311568b700 1 --2- 172.21.15.59:0/1836154178 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f31100ae210 0x7f311014e750 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f30f800c7c0 tx=0x7f30f8007eb0 comp rx=0 tx=0).stop 2023-09-30T02:08:29.770 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.768+0000 7f311568b700 1 -- 172.21.15.59:0/1836154178 shutdown_connections 2023-09-30T02:08:29.770 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.768+0000 7f311568b700 1 --2- 172.21.15.59:0/1836154178 >> v2:172.21.15.110:6800/115870 conn(0x7f30f0068910 0x7f30f006adc0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:29.770 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.768+0000 7f311568b700 1 --2- 172.21.15.59:0/1836154178 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f31100aeca0 0x7f311014ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:29.770 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.768+0000 7f311568b700 1 --2- 172.21.15.59:0/1836154178 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f31100a54e0 0x7f31100a95b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:29.770 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.768+0000 7f311568b700 1 --2- 172.21.15.59:0/1836154178 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f31100ae210 0x7f311014e750 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:29.771 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.768+0000 7f311568b700 1 -- 172.21.15.59:0/1836154178 >> 172.21.15.59:0/1836154178 conn(0x7f31100631f0 msgr2=0x7f3110070f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:08:29.771 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.768+0000 7f311568b700 1 -- 172.21.15.59:0/1836154178 shutdown_connections 2023-09-30T02:08:29.771 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.768+0000 7f311568b700 1 -- 172.21.15.59:0/1836154178 wait complete. 2023-09-30T02:08:29.771 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:29.768+0000 7f311568b700 1 librados: shutdown 2023-09-30T02:08:32.784 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:08:32.921 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.920+0000 7f65daa59700 10 monclient: get_monmap_and_config 2023-09-30T02:08:32.921 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.920+0000 7f65daa59700 10 monclient: build_initial_monmap 2023-09-30T02:08:32.921 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.920+0000 7f65daa59700 10 monclient: monmap: 2023-09-30T02:08:32.921 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:08:32.921 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:32.921 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:08:32.922076+0000 2023-09-30T02:08:32.922 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:08:32.922076+0000 2023-09-30T02:08:32.922 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:08:32.922 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:32.922 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:08:32.922 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:08:32.922 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:08:32.922 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:08:32.922 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:08:32.922 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:32.924 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.921+0000 7f65daa59700 1 Processor -- start 2023-09-30T02:08:32.925 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.921+0000 7f65daa59700 1 -- start start 2023-09-30T02:08:32.925 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.921+0000 7f65daa59700 10 monclient: init 2023-09-30T02:08:32.925 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.922+0000 7f65daa59700 10 monclient: _reopen_session rank -1 2023-09-30T02:08:32.925 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.922+0000 7f65daa59700 10 monclient: _add_conns ranks=[3,1,2,4,0] 2023-09-30T02:08:32.925 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.922+0000 7f65daa59700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f65d40a5b70 0x7f65d40b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:32.925 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.922+0000 7f65daa59700 10 monclient(hunting): picked mon.noname-a con 0x7f65d40a5b70 addr v2:172.21.15.59:3300/0 2023-09-30T02:08:32.925 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.922+0000 7f65daa59700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f65d40a5f40 addr v1:172.21.15.110:6789/0 2023-09-30T02:08:32.925 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.922+0000 7f65daa59700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f65d4142290 0x7f65d413af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:32.925 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.922+0000 7f65daa59700 10 monclient(hunting): picked mon.noname-b con 0x7f65d4142290 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:08:32.925 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.922+0000 7f65daa59700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65d40a75a0 con 0x7f65d40a5f40 2023-09-30T02:08:32.925 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.922+0000 7f65daa59700 10 monclient(hunting): start opening mon connection 2023-09-30T02:08:32.926 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.922+0000 7f65daa59700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65d41426d0 con 0x7f65d40a5b70 2023-09-30T02:08:32.926 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.922+0000 7f65daa59700 10 monclient(hunting): start opening mon connection 2023-09-30T02:08:32.926 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.922+0000 7f65daa59700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65d40a63e0 con 0x7f65d4142290 2023-09-30T02:08:32.926 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.922+0000 7f65daa59700 10 monclient(hunting): _renew_subs 2023-09-30T02:08:32.926 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.922+0000 7f65daa59700 10 monclient(hunting): authenticate will time out at 1958.225408s 2023-09-30T02:08:32.926 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.923+0000 7f65d3fff700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f65d40a5b70 0x7f65d40b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:32.926 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.923+0000 7f65d8ff6700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f65d4142290 0x7f65d413af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:32.926 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.923+0000 7f65d37fe700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f65d40a5f40 0x7f65d4141b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.59:41782/0 (socket says 172.21.15.59:41782) 2023-09-30T02:08:32.926 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.923+0000 7f65d37fe700 1 -- 172.21.15.59:0/631113135 learned_addr learned my addr 172.21.15.59:0/631113135 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:08:32.926 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.923+0000 7f65d8ff6700 10 monclient(hunting): get_auth_request con 0x7f65d4142290 auth_method 0 2023-09-30T02:08:32.926 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.923+0000 7f65d8ff6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:08:32.926 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.923+0000 7f65d8ff6700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:32.927 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.923+0000 7f65d8ff6700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:32.927 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.923+0000 7f65d3fff700 10 monclient(hunting): get_auth_request con 0x7f65d40a5b70 auth_method 0 2023-09-30T02:08:32.927 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.923+0000 7f65d3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:08:32.927 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.923+0000 7f65d3fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:32.927 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.923+0000 7f65d3fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:32.927 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.924+0000 7f65d8ff6700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:08:32.927 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.924+0000 7f65d8ff6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:08:32.929 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.924+0000 7f65d8ff6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:08:32.929 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.924+0000 7f65d3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:08:32.929 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.924+0000 7f65d3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:08:32.929 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.924+0000 7f65d3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:08:32.930 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.926+0000 7f65d2ffd700 1 -- 172.21.15.59:0/631113135 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 404280676 0 0) 0x7f65d40a75a0 con 0x7f65d40a5f40 2023-09-30T02:08:32.930 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.926+0000 7f65d2ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:32.930 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.926+0000 7f65d2ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:32.930 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.926+0000 7f65d2ffd700 10 monclient(hunting): my global_id is 4576 2023-09-30T02:08:32.930 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.926+0000 7f65d2ffd700 1 -- 172.21.15.59:0/631113135 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65bc003570 con 0x7f65d40a5f40 2023-09-30T02:08:32.931 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.928+0000 7f65d8ff6700 10 monclient(hunting): handle_auth_done global_id 4505 payload 274 2023-09-30T02:08:32.931 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.928+0000 7f65d8ff6700 1 -- 172.21.15.59:0/631113135 >> v2:172.21.15.59:3300/0 conn(0x7f65d40a5b70 msgr2=0x7f65d40b1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:08:32.931 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.928+0000 7f65d8ff6700 1 --2- 172.21.15.59:0/631113135 >> v2:172.21.15.59:3300/0 conn(0x7f65d40a5b70 0x7f65d40b1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:32.931 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.928+0000 7f65d8ff6700 1 -- 172.21.15.59:0/631113135 >> v1:172.21.15.110:6789/0 conn(0x7f65d40a5f40 legacy=0x7f65d4141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:32.931 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.928+0000 7f65d8ff6700 10 monclient: _finish_hunting 0 2023-09-30T02:08:32.931 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.928+0000 7f65d8ff6700 1 monclient: found mon.noname-b 2023-09-30T02:08:32.931 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.928+0000 7f65d8ff6700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:08:32.931 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.928+0000 7f65d8ff6700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.59:3301/0 2023-09-30T02:08:32.931 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.928+0000 7f65d8ff6700 1 -- 172.21.15.59:0/631113135 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65d40af3f0 con 0x7f65d4142290 2023-09-30T02:08:32.932 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.930+0000 7f65d8ff6700 1 --2- 172.21.15.59:0/631113135 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f65d4142290 0x7f65d413af30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f65cc009e60 tx=0x7f65cc009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc6f50042e22e167 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:08:32.935 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.934+0000 7f65d2ffd700 1 -- 172.21.15.59:0/631113135 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65cc00e070 con 0x7f65d4142290 2023-09-30T02:08:32.935 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.934+0000 7f65d2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:08:32.935 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.934+0000 7f65d2ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:08:32.935 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.934+0000 7f65d2ffd700 10 monclient: dump: 2023-09-30T02:08:32.936 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:08:32.936 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:32.936 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:32.936 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:32.936 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:08:32.936 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:32.936 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:08:32.936 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:08:32.936 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:08:32.936 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:32.936 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.934+0000 7f65d2ffd700 10 monclient: _finish_auth 0 2023-09-30T02:08:32.937 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.934+0000 7f65d2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:08:32.937 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.934+0000 7f65d2ffd700 1 -- 172.21.15.59:0/631113135 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65cc00bab0 con 0x7f65d4142290 2023-09-30T02:08:32.937 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.934+0000 7f65d2ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:08:32.937 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.934+0000 7f65d2ffd700 1 -- 172.21.15.59:0/631113135 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65cc00be70 con 0x7f65d4142290 2023-09-30T02:08:32.937 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.934+0000 7f65d2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:08:32.937 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.934+0000 7f65d2ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:08:32.937 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.934+0000 7f65d2ffd700 10 monclient: dump: 2023-09-30T02:08:32.937 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:08:32.937 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:32.937 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:32.937 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:32.938 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:08:32.938 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:32.938 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:08:32.938 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:08:32.938 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:08:32.938 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:32.938 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.935+0000 7f65daa59700 5 monclient: authenticate success, global_id 4505 2023-09-30T02:08:32.938 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.935+0000 7f65daa59700 10 monclient: get_monmap_and_config success 2023-09-30T02:08:32.938 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.935+0000 7f65daa59700 4 set_mon_vals no callback set 2023-09-30T02:08:32.941 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.935+0000 7f65daa59700 10 monclient: shutdown 2023-09-30T02:08:32.941 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.935+0000 7f65daa59700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:08:32.941 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.935+0000 7f65daa59700 1 -- 172.21.15.59:0/631113135 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f65d4142290 msgr2=0x7f65d413af30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:32.941 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.935+0000 7f65daa59700 1 --2- 172.21.15.59:0/631113135 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f65d4142290 0x7f65d413af30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f65cc009e60 tx=0x7f65cc009780 comp rx=0 tx=0).stop 2023-09-30T02:08:32.941 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.935+0000 7f65daa59700 1 -- 172.21.15.59:0/631113135 shutdown_connections 2023-09-30T02:08:32.942 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.935+0000 7f65daa59700 1 --2- 172.21.15.59:0/631113135 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f65d4142290 0x7f65d413af30 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:32.942 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.935+0000 7f65daa59700 1 --2- 172.21.15.59:0/631113135 >> v2:172.21.15.59:3300/0 conn(0x7f65d40a5b70 0x7f65d40b1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:32.942 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.935+0000 7f65daa59700 1 -- 172.21.15.59:0/631113135 >> 172.21.15.59:0/631113135 conn(0x7f65d40631f0 msgr2=0x7f65d40635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:08:32.942 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.935+0000 7f65daa59700 1 -- 172.21.15.59:0/631113135 shutdown_connections 2023-09-30T02:08:32.942 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.935+0000 7f65daa59700 1 -- 172.21.15.59:0/631113135 wait complete. 2023-09-30T02:08:32.942 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.936+0000 7f65daa59700 10 monclient: build_initial_monmap 2023-09-30T02:08:32.942 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.936+0000 7f65daa59700 10 monclient: monmap: 2023-09-30T02:08:32.942 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:08:32.942 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:08:32.942 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:08:32.942 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:08:32.943 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:08:32.943 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:08:32.943 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:08:32.943 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:08:32.943 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:08:32.943 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:32.943 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.936+0000 7f65daa59700 1 librados: starting msgr at 2023-09-30T02:08:32.943 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.936+0000 7f65daa59700 1 librados: starting objecter 2023-09-30T02:08:32.943 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.937+0000 7f65daa59700 1 Processor -- start 2023-09-30T02:08:32.943 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.937+0000 7f65daa59700 1 -- start start 2023-09-30T02:08:32.943 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.937+0000 7f65daa59700 1 librados: setting wanted keys 2023-09-30T02:08:32.943 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.937+0000 7f65daa59700 1 librados: calling monclient init 2023-09-30T02:08:32.944 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.937+0000 7f65daa59700 10 monclient: init 2023-09-30T02:08:32.944 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.937+0000 7f65daa59700 10 monclient: _reopen_session rank -1 2023-09-30T02:08:32.944 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.937+0000 7f65daa59700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T02:08:32.944 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.937+0000 7f65daa59700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65d40a5b70 0x7f65d4150dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:32.944 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.937+0000 7f65daa59700 10 monclient(hunting): picked mon.noname-a con 0x7f65d40a5b70 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:08:32.944 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.937+0000 7f65daa59700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f65d40a5f40 0x7f65d4151300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:32.944 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.937+0000 7f65daa59700 10 monclient(hunting): picked mon.noname-b con 0x7f65d40a5f40 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:08:32.944 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.937+0000 7f65daa59700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f65d4142290 0x7f65d40aabd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:32.944 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.937+0000 7f65daa59700 10 monclient(hunting): picked mon.noname-c con 0x7f65d4142290 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:08:32.944 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.937+0000 7f65daa59700 10 monclient(hunting): start opening mon connection 2023-09-30T02:08:32.944 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.937+0000 7f65daa59700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65d40aca20 con 0x7f65d40a5b70 2023-09-30T02:08:32.945 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.937+0000 7f65daa59700 10 monclient(hunting): start opening mon connection 2023-09-30T02:08:32.945 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.937+0000 7f65daa59700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65d4151990 con 0x7f65d40a5f40 2023-09-30T02:08:32.945 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.937+0000 7f65daa59700 10 monclient(hunting): start opening mon connection 2023-09-30T02:08:32.945 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.937+0000 7f65daa59700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65d414b110 con 0x7f65d4142290 2023-09-30T02:08:32.945 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.937+0000 7f65daa59700 10 monclient(hunting): _renew_subs 2023-09-30T02:08:32.945 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.937+0000 7f65daa59700 10 monclient(hunting): authenticate will time out at 1958.240931s 2023-09-30T02:08:32.945 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.938+0000 7f65d3fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65d40a5b70 0x7f65d4150dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:32.945 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.938+0000 7f65d8ff6700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f65d4142290 0x7f65d40aabd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:32.945 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.938+0000 7f65d37fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f65d40a5f40 0x7f65d4151300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:32.945 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.938+0000 7f65d8ff6700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f65d4142290 0x7f65d40aabd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:35596/0 (socket says 172.21.15.59:35596) 2023-09-30T02:08:32.945 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.938+0000 7f65d8ff6700 1 -- 172.21.15.59:0/2533087033 learned_addr learned my addr 172.21.15.59:0/2533087033 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:08:32.945 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.938+0000 7f65d8ff6700 10 monclient(hunting): get_auth_request con 0x7f65d4142290 auth_method 0 2023-09-30T02:08:32.946 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.938+0000 7f65d8ff6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:08:32.946 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.938+0000 7f65d8ff6700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:32.946 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.938+0000 7f65d8ff6700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:32.946 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.938+0000 7f65d3fff700 10 monclient(hunting): get_auth_request con 0x7f65d40a5b70 auth_method 0 2023-09-30T02:08:32.946 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.938+0000 7f65d3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:08:32.946 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.938+0000 7f65d3fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:32.946 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.938+0000 7f65d3fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:32.946 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.939+0000 7f65d37fe700 10 monclient(hunting): get_auth_request con 0x7f65d40a5f40 auth_method 0 2023-09-30T02:08:32.946 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.939+0000 7f65d37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:08:32.946 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.939+0000 7f65d37fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:32.946 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.939+0000 7f65d37fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:32.946 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.939+0000 7f65d8ff6700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:08:32.947 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.939+0000 7f65d8ff6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:08:32.947 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.939+0000 7f65d8ff6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:08:32.947 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.939+0000 7f65d3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:08:32.947 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.939+0000 7f65d3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:08:32.947 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.939+0000 7f65d3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:08:32.947 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.939+0000 7f65d37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:08:32.947 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.940+0000 7f65d37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:08:32.947 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.940+0000 7f65d37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:08:32.949 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.943+0000 7f65d8ff6700 10 monclient(hunting): handle_auth_done global_id 4508 payload 824 2023-09-30T02:08:32.949 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.943+0000 7f65d8ff6700 1 -- 172.21.15.59:0/2533087033 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f65d40a5f40 msgr2=0x7f65d4151300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:32.949 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.943+0000 7f65d8ff6700 1 --2- 172.21.15.59:0/2533087033 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f65d40a5f40 0x7f65d4151300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:32.949 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.943+0000 7f65d8ff6700 1 -- 172.21.15.59:0/2533087033 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65d40a5b70 msgr2=0x7f65d4150dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:32.949 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.943+0000 7f65d8ff6700 1 --2- 172.21.15.59:0/2533087033 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65d40a5b70 0x7f65d4150dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:32.949 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.943+0000 7f65d8ff6700 10 monclient: _finish_hunting 0 2023-09-30T02:08:32.950 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.943+0000 7f65d8ff6700 1 monclient: found mon.noname-c 2023-09-30T02:08:32.950 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.943+0000 7f65d8ff6700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:08:32.950 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.943+0000 7f65d8ff6700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.59:3301/0 2023-09-30T02:08:32.950 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.943+0000 7f65d8ff6700 1 -- 172.21.15.59:0/2533087033 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65cc009a20 con 0x7f65d4142290 2023-09-30T02:08:32.950 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.945+0000 7f65d8ff6700 1 --2- 172.21.15.59:0/2533087033 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f65d4142290 0x7f65d40aabd0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f65cc00b610 tx=0x7f65cc019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:08:32.950 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.945+0000 7f65bbfff700 10 client.?.objecter ms_handle_connect 0x7f65d4142290 2023-09-30T02:08:32.950 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.945+0000 7f65bbfff700 10 client.?.objecter resend_mon_ops 2023-09-30T02:08:32.950 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.947+0000 7f65bbfff700 1 -- 172.21.15.59:0/2533087033 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65cc012e10 con 0x7f65d4142290 2023-09-30T02:08:32.950 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.947+0000 7f65bbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:08:32.950 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.947+0000 7f65bbfff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:08:32.950 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.947+0000 7f65bbfff700 10 monclient: dump: 2023-09-30T02:08:32.951 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:08:32.951 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:32.951 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:32.951 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:32.951 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:08:32.951 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:32.951 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:08:32.951 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:08:32.951 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:08:32.951 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:32.951 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.947+0000 7f65bbfff700 10 monclient: _finish_auth 0 2023-09-30T02:08:32.951 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.947+0000 7f65bbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:08:32.952 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.947+0000 7f65daa59700 5 monclient: authenticate success, global_id 4508 2023-09-30T02:08:32.952 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.947+0000 7f65daa59700 10 monclient: _renew_subs 2023-09-30T02:08:32.952 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.947+0000 7f65daa59700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:08:32.952 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.948+0000 7f65daa59700 1 -- 172.21.15.59:0/2533087033 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65d40a96c0 con 0x7f65d4142290 2023-09-30T02:08:32.952 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.948+0000 7f65daa59700 10 client.4508.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:08:32.952 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.948+0000 7f65daa59700 10 monclient: _renew_subs 2023-09-30T02:08:32.952 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.948+0000 7f65daa59700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:08:32.952 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.948+0000 7f65daa59700 1 -- 172.21.15.59:0/2533087033 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65d40ab400 con 0x7f65d4142290 2023-09-30T02:08:32.952 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.948+0000 7f65daa59700 1 librados: init done 2023-09-30T02:08:32.952 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.948+0000 7f65daa59700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:08:32.952 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.948+0000 7f65daa59700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:08:32.953 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.948+0000 7f65daa59700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:08:32.953 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.948+0000 7f65daa59700 1 -- 172.21.15.59:0/2533087033 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65d4064a00 con 0x7f65d4142290 2023-09-30T02:08:32.953 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.949+0000 7f65bbfff700 1 -- 172.21.15.59:0/2533087033 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65cc027680 con 0x7f65d4142290 2023-09-30T02:08:32.953 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.949+0000 7f65bbfff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:08:32.953 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.949+0000 7f65bbfff700 1 -- 172.21.15.59:0/2533087033 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65cc027810 con 0x7f65d4142290 2023-09-30T02:08:32.953 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.949+0000 7f65bbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:08:32.953 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.949+0000 7f65bbfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:08:32.954 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.949+0000 7f65bbfff700 10 monclient: dump: 2023-09-30T02:08:32.954 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:08:32.954 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:32.954 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:32.954 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:32.954 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:08:32.954 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:32.954 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:08:32.954 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:08:32.954 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:08:32.954 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:32.954 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:32.949+0000 7f65d27fc700 4 set_mon_vals no callback set 2023-09-30T02:08:33.163 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:33.163+0000 7f65bbfff700 1 -- 172.21.15.59:0/2533087033 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f65cc024070 con 0x7f65d4142290 2023-09-30T02:08:33.164 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:33.163+0000 7f65bbfff700 1 --2- 172.21.15.59:0/2533087033 >> v2:172.21.15.110:6800/115870 conn(0x7f65c0068890 0x7f65c006ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:33.164 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:33.163+0000 7f65bbfff700 1 -- 172.21.15.59:0/2533087033 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f65cc0865d0 con 0x7f65d4142290 2023-09-30T02:08:33.164 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:33.163+0000 7f65bbfff700 10 client.4508.objecter ms_dispatch 0x7f65d4001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:08:33.164 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:33.163+0000 7f65bbfff700 3 client.4508.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:08:33.164 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:33.163+0000 7f65bbfff700 3 client.4508.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:08:33.164 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:33.163+0000 7f65bbfff700 20 client.4508.objecter dump_active .. 0 homeless 2023-09-30T02:08:33.165 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:33.164+0000 7f65d3fff700 1 --2- 172.21.15.59:0/2533087033 >> v2:172.21.15.110:6800/115870 conn(0x7f65c0068890 0x7f65c006ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:33.165 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:33.164+0000 7f65d3fff700 10 monclient: get_auth_request con 0x7f65c0068890 auth_method 0 2023-09-30T02:08:33.165 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:33.164+0000 7f65d3fff700 1 --2- 172.21.15.59:0/2533087033 >> v2:172.21.15.110:6800/115870 conn(0x7f65c0068890 0x7f65c006ad40 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f65c4009db0 tx=0x7f65c40091a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:08:33.165 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:33.164+0000 7f65bbfff700 10 client.4508.objecter ms_handle_connect 0x7f65c0068890 2023-09-30T02:08:33.285 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:33.285+0000 7f65bbfff700 1 -- 172.21.15.59:0/2533087033 <== mon.2 v2:172.21.15.59:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f65cc053df0 con 0x7f65d4142290 2023-09-30T02:08:33.285 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:33.285+0000 7f65bbfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:08:33.285 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:33.285+0000 7f65bbfff700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:08:33.569 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:33.569+0000 7f65daa59700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:08:33.569 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:33.569+0000 7f65daa59700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:08:33.570 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:33.569+0000 7f65daa59700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:08:33.570 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:33.569+0000 7f65daa59700 1 -- 172.21.15.59:0/2533087033 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f65d406a720 con 0x7f65d4142290 2023-09-30T02:08:33.577 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:33.576+0000 7f65bbfff700 1 -- 172.21.15.59:0/2533087033 <== mon.2 v2:172.21.15.59:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f65cc057410 con 0x7f65d4142290 2023-09-30T02:08:33.577 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:33.576+0000 7f65bbfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:08:33.577 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:33.576+0000 7f65bbfff700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:08:33.577 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:08:33.577 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:08:33.580 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:33.579+0000 7f65daa59700 10 librados: watch_flush enter 2023-09-30T02:08:33.580 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:33.579+0000 7f65daa59700 10 librados: watch_flush exit 2023-09-30T02:08:33.580 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:33.579+0000 7f65daa59700 20 client.4508.objecter shutdown clearing up homeless session... 2023-09-30T02:08:33.580 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:33.579+0000 7f65daa59700 10 client.4508.objecter successfully canceled tick 2023-09-30T02:08:33.580 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:33.579+0000 7f65daa59700 1 -- 172.21.15.59:0/2533087033 >> v2:172.21.15.110:6800/115870 conn(0x7f65c0068890 msgr2=0x7f65c006ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:33.580 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:33.579+0000 7f65daa59700 1 --2- 172.21.15.59:0/2533087033 >> v2:172.21.15.110:6800/115870 conn(0x7f65c0068890 0x7f65c006ad40 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f65c4009db0 tx=0x7f65c40091a0 comp rx=0 tx=0).stop 2023-09-30T02:08:33.580 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:33.579+0000 7f65daa59700 10 monclient: shutdown 2023-09-30T02:08:33.580 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:33.579+0000 7f65daa59700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:08:33.580 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:33.579+0000 7f65daa59700 1 -- 172.21.15.59:0/2533087033 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f65d4142290 msgr2=0x7f65d40aabd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:33.581 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:33.579+0000 7f65daa59700 1 --2- 172.21.15.59:0/2533087033 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f65d4142290 0x7f65d40aabd0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f65cc00b610 tx=0x7f65cc019000 comp rx=0 tx=0).stop 2023-09-30T02:08:33.581 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:33.579+0000 7f65daa59700 1 -- 172.21.15.59:0/2533087033 shutdown_connections 2023-09-30T02:08:33.581 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:33.579+0000 7f65daa59700 1 --2- 172.21.15.59:0/2533087033 >> v2:172.21.15.110:6800/115870 conn(0x7f65c0068890 0x7f65c006ad40 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:33.581 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:33.579+0000 7f65daa59700 1 --2- 172.21.15.59:0/2533087033 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f65d4142290 0x7f65d40aabd0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:33.581 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:33.579+0000 7f65daa59700 1 --2- 172.21.15.59:0/2533087033 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65d40a5b70 0x7f65d4150dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:33.581 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:33.579+0000 7f65daa59700 1 --2- 172.21.15.59:0/2533087033 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f65d40a5f40 0x7f65d4151300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:33.581 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:33.579+0000 7f65daa59700 1 -- 172.21.15.59:0/2533087033 >> 172.21.15.59:0/2533087033 conn(0x7f65d40631f0 msgr2=0x7f65d40635c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:08:33.581 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:33.579+0000 7f65daa59700 1 -- 172.21.15.59:0/2533087033 shutdown_connections 2023-09-30T02:08:33.581 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:33.579+0000 7f65daa59700 1 -- 172.21.15.59:0/2533087033 wait complete. 2023-09-30T02:08:33.581 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:33.579+0000 7f65daa59700 1 librados: shutdown 2023-09-30T02:08:33.817 INFO:tasks.ceph.mon.a.smithi059.stderr:==00:00:03:43.682 116256== Warning: unimplemented fcntl command: 1036 2023-09-30T02:08:33.819 INFO:tasks.ceph.mon.a.smithi059.stderr:==00:00:03:43.685 116256== Warning: unimplemented fcntl command: 1036 2023-09-30T02:08:33.845 INFO:tasks.ceph.mon.a.smithi059.stderr:==00:00:03:43.711 116256== Warning: unimplemented fcntl command: 1036 2023-09-30T02:08:33.871 INFO:tasks.ceph.mon.b.smithi110.stderr:==00:00:03:43.718 115869== Warning: unimplemented fcntl command: 1036 2023-09-30T02:08:33.874 INFO:tasks.ceph.mon.b.smithi110.stderr:==00:00:03:43.721 115869== Warning: unimplemented fcntl command: 1036 2023-09-30T02:08:33.874 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:03:43.740 116257== Warning: unimplemented fcntl command: 1036 2023-09-30T02:08:33.890 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:03:43.755 116257== Warning: unimplemented fcntl command: 1036 2023-09-30T02:08:33.897 INFO:tasks.ceph.mon.b.smithi110.stderr:==00:00:03:43.745 115869== Warning: unimplemented fcntl command: 1036 2023-09-30T02:08:33.915 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:03:43.780 116257== Warning: unimplemented fcntl command: 1036 2023-09-30T02:08:36.594 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:08:36.727 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.726+0000 7f8ee3fe5700 10 monclient: get_monmap_and_config 2023-09-30T02:08:36.727 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.726+0000 7f8ee3fe5700 10 monclient: build_initial_monmap 2023-09-30T02:08:36.727 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.726+0000 7f8ee3fe5700 10 monclient: monmap: 2023-09-30T02:08:36.727 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:08:36.727 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:36.727 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:08:36.727849+0000 2023-09-30T02:08:36.727 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:08:36.727849+0000 2023-09-30T02:08:36.727 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:08:36.728 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:36.728 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:08:36.728 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:08:36.728 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:08:36.728 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:08:36.728 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:08:36.728 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:36.728 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.727+0000 7f8ee3fe5700 1 Processor -- start 2023-09-30T02:08:36.730 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.727+0000 7f8ee3fe5700 1 -- start start 2023-09-30T02:08:36.730 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.727+0000 7f8ee3fe5700 10 monclient: init 2023-09-30T02:08:36.730 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.727+0000 7f8ee3fe5700 10 monclient: _reopen_session rank -1 2023-09-30T02:08:36.730 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.727+0000 7f8ee3fe5700 10 monclient: _add_conns ranks=[2,4,3,0,1] 2023-09-30T02:08:36.731 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.727+0000 7f8ee3fe5700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8edc0a5b70 0x7f8edc0b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:36.731 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.727+0000 7f8ee3fe5700 10 monclient(hunting): picked mon.noname-b con 0x7f8edc0a5b70 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:08:36.731 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.727+0000 7f8ee3fe5700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f8edc0a5f40 0x7f8edc141b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:36.731 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.727+0000 7f8ee3fe5700 10 monclient(hunting): picked mon.noname-c con 0x7f8edc0a5f40 addr v2:172.21.15.110:3300/0 2023-09-30T02:08:36.731 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.727+0000 7f8ee3fe5700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f8edc1420c0 0x7f8edc13e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:36.731 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.727+0000 7f8ee3fe5700 10 monclient(hunting): picked mon.noname-a con 0x7f8edc1420c0 addr v2:172.21.15.59:3300/0 2023-09-30T02:08:36.731 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.727+0000 7f8ee3fe5700 10 monclient(hunting): start opening mon connection 2023-09-30T02:08:36.731 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.727+0000 7f8ee3fe5700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8edc0a63b0 con 0x7f8edc1420c0 2023-09-30T02:08:36.731 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.728+0000 7f8ee3fe5700 10 monclient(hunting): start opening mon connection 2023-09-30T02:08:36.731 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.728+0000 7f8ee3fe5700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8edc0a75a0 con 0x7f8edc0a5f40 2023-09-30T02:08:36.731 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.728+0000 7f8ee3fe5700 10 monclient(hunting): start opening mon connection 2023-09-30T02:08:36.732 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.728+0000 7f8ee3fe5700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8edc0af3f0 con 0x7f8edc0a5b70 2023-09-30T02:08:36.732 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.728+0000 7f8ee3fe5700 10 monclient(hunting): _renew_subs 2023-09-30T02:08:36.732 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.728+0000 7f8ee3fe5700 10 monclient(hunting): authenticate will time out at 1962.031023s 2023-09-30T02:08:36.732 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.728+0000 7f8ee1d81700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8edc0a5b70 0x7f8edc0b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:36.732 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.728+0000 7f8ee2582700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f8edc1420c0 0x7f8edc13e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:36.732 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.729+0000 7f8ee1580700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f8edc0a5f40 0x7f8edc141b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:36.732 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.729+0000 7f8ee1d81700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8edc0a5b70 0x7f8edc0b1500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:35600/0 (socket says 172.21.15.59:35600) 2023-09-30T02:08:36.732 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.729+0000 7f8ee1d81700 1 -- 172.21.15.59:0/4142981237 learned_addr learned my addr 172.21.15.59:0/4142981237 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:08:36.732 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.729+0000 7f8ee1d81700 10 monclient(hunting): get_auth_request con 0x7f8edc0a5b70 auth_method 0 2023-09-30T02:08:36.732 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.729+0000 7f8ee1d81700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:08:36.733 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.729+0000 7f8ee1d81700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:36.733 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.729+0000 7f8ee1d81700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:36.733 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.729+0000 7f8ee2582700 10 monclient(hunting): get_auth_request con 0x7f8edc1420c0 auth_method 0 2023-09-30T02:08:36.733 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.729+0000 7f8ee2582700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:08:36.733 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.729+0000 7f8ee2582700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:36.733 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.729+0000 7f8ee2582700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:36.733 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.729+0000 7f8ee1580700 10 monclient(hunting): get_auth_request con 0x7f8edc0a5f40 auth_method 0 2023-09-30T02:08:36.733 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.729+0000 7f8ee1580700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:08:36.733 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.729+0000 7f8ee1580700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:36.733 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.729+0000 7f8ee1580700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:36.733 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.730+0000 7f8ee1d81700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:08:36.734 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.730+0000 7f8ee1d81700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:08:36.734 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.730+0000 7f8ee1d81700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:08:36.734 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.730+0000 7f8ee1580700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:08:36.734 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.730+0000 7f8ee1580700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:08:36.734 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.730+0000 7f8ee1580700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:08:36.734 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.730+0000 7f8ee2582700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:08:36.734 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.730+0000 7f8ee2582700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:08:36.734 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.730+0000 7f8ee2582700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:08:36.735 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.733+0000 7f8ee2582700 10 monclient(hunting): handle_auth_done global_id 4572 payload 274 2023-09-30T02:08:36.735 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.733+0000 7f8ee2582700 1 -- 172.21.15.59:0/4142981237 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8edc0a5b70 msgr2=0x7f8edc0b1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:08:36.735 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.733+0000 7f8ee2582700 1 --2- 172.21.15.59:0/4142981237 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8edc0a5b70 0x7f8edc0b1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:36.735 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.733+0000 7f8ee2582700 1 -- 172.21.15.59:0/4142981237 >> v2:172.21.15.110:3300/0 conn(0x7f8edc0a5f40 msgr2=0x7f8edc141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:08:36.735 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.733+0000 7f8ee2582700 1 --2- 172.21.15.59:0/4142981237 >> v2:172.21.15.110:3300/0 conn(0x7f8edc0a5f40 0x7f8edc141b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:36.735 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.733+0000 7f8ee2582700 10 monclient: _finish_hunting 0 2023-09-30T02:08:36.735 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.733+0000 7f8ee2582700 1 monclient: found mon.noname-a 2023-09-30T02:08:36.735 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.733+0000 7f8ee2582700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:08:36.735 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.733+0000 7f8ee2582700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.59:3300/0 2023-09-30T02:08:36.736 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.733+0000 7f8ee2582700 1 -- 172.21.15.59:0/4142981237 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8edc142500 con 0x7f8edc1420c0 2023-09-30T02:08:36.736 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.736+0000 7f8ee2582700 1 --2- 172.21.15.59:0/4142981237 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8edc1420c0 0x7f8edc13e230 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f8edc069af0 tx=0x7f8ed4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2efe4869348f5cda server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:08:36.741 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.740+0000 7f8ee0d7f700 1 -- 172.21.15.59:0/4142981237 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ed400e070 con 0x7f8edc1420c0 2023-09-30T02:08:36.741 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.740+0000 7f8ee0d7f700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:08:36.741 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.740+0000 7f8ee0d7f700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T02:08:36.741 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.740+0000 7f8ee0d7f700 10 monclient: dump: 2023-09-30T02:08:36.741 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:08:36.741 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:36.741 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:36.742 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:36.742 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:08:36.742 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:36.742 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:08:36.742 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:08:36.742 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:08:36.742 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:36.742 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.740+0000 7f8ee0d7f700 10 monclient: _finish_auth 0 2023-09-30T02:08:36.742 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.740+0000 7f8ee0d7f700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:08:36.742 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.740+0000 7f8ee0d7f700 1 -- 172.21.15.59:0/4142981237 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ed400bab0 con 0x7f8edc1420c0 2023-09-30T02:08:36.742 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.740+0000 7f8ee0d7f700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:08:36.743 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.740+0000 7f8ee0d7f700 1 -- 172.21.15.59:0/4142981237 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ed400be70 con 0x7f8edc1420c0 2023-09-30T02:08:36.743 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.740+0000 7f8ee0d7f700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:08:36.743 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.740+0000 7f8ee0d7f700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T02:08:36.743 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.740+0000 7f8ee0d7f700 10 monclient: dump: 2023-09-30T02:08:36.743 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:08:36.743 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:36.743 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:36.743 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:36.743 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:08:36.743 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:36.743 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:08:36.744 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:08:36.744 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:08:36.744 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:36.744 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.740+0000 7f8ee3fe5700 5 monclient: authenticate success, global_id 4572 2023-09-30T02:08:36.744 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.740+0000 7f8ee3fe5700 10 monclient: get_monmap_and_config success 2023-09-30T02:08:36.744 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.740+0000 7f8ee3fe5700 4 set_mon_vals no callback set 2023-09-30T02:08:36.747 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.741+0000 7f8ee3fe5700 10 monclient: shutdown 2023-09-30T02:08:36.747 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.741+0000 7f8ee3fe5700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:08:36.747 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.741+0000 7f8ee3fe5700 1 -- 172.21.15.59:0/4142981237 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8edc1420c0 msgr2=0x7f8edc13e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:36.747 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.741+0000 7f8ee3fe5700 1 --2- 172.21.15.59:0/4142981237 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8edc1420c0 0x7f8edc13e230 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f8edc069af0 tx=0x7f8ed4009780 comp rx=0 tx=0).stop 2023-09-30T02:08:36.748 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.741+0000 7f8ee3fe5700 1 -- 172.21.15.59:0/4142981237 shutdown_connections 2023-09-30T02:08:36.748 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.741+0000 7f8ee3fe5700 1 --2- 172.21.15.59:0/4142981237 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8edc0a5b70 0x7f8edc0b1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:36.748 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.741+0000 7f8ee3fe5700 1 --2- 172.21.15.59:0/4142981237 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8edc1420c0 0x7f8edc13e230 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:36.748 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.741+0000 7f8ee3fe5700 1 --2- 172.21.15.59:0/4142981237 >> v2:172.21.15.110:3300/0 conn(0x7f8edc0a5f40 0x7f8edc141b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:36.748 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.741+0000 7f8ee3fe5700 1 -- 172.21.15.59:0/4142981237 >> 172.21.15.59:0/4142981237 conn(0x7f8edc0631f0 msgr2=0x7f8edc0635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:08:36.748 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.741+0000 7f8ee3fe5700 1 -- 172.21.15.59:0/4142981237 shutdown_connections 2023-09-30T02:08:36.748 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.741+0000 7f8ee3fe5700 1 -- 172.21.15.59:0/4142981237 wait complete. 2023-09-30T02:08:36.748 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.742+0000 7f8ee3fe5700 10 monclient: build_initial_monmap 2023-09-30T02:08:36.748 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.742+0000 7f8ee3fe5700 10 monclient: monmap: 2023-09-30T02:08:36.748 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:08:36.749 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:08:36.749 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:08:36.749 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:08:36.749 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:08:36.749 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:08:36.749 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:08:36.749 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:08:36.749 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:08:36.749 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:36.749 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.742+0000 7f8ee3fe5700 1 librados: starting msgr at 2023-09-30T02:08:36.749 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.742+0000 7f8ee3fe5700 1 librados: starting objecter 2023-09-30T02:08:36.749 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.742+0000 7f8ee3fe5700 1 Processor -- start 2023-09-30T02:08:36.749 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.742+0000 7f8ee3fe5700 1 -- start start 2023-09-30T02:08:36.750 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.742+0000 7f8ee3fe5700 1 librados: setting wanted keys 2023-09-30T02:08:36.750 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.742+0000 7f8ee3fe5700 1 librados: calling monclient init 2023-09-30T02:08:36.750 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.742+0000 7f8ee3fe5700 10 monclient: init 2023-09-30T02:08:36.750 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.743+0000 7f8ee3fe5700 10 monclient: _reopen_session rank -1 2023-09-30T02:08:36.750 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.743+0000 7f8ee3fe5700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T02:08:36.750 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.743+0000 7f8ee3fe5700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8edc0a5b70 0x7f8edc151440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:36.750 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.743+0000 7f8ee3fe5700 10 monclient(hunting): picked mon.noname-a con 0x7f8edc0a5b70 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:08:36.750 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.743+0000 7f8ee3fe5700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8edc0a5f40 0x7f8edc151980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:36.750 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.743+0000 7f8ee3fe5700 10 monclient(hunting): picked mon.noname-c con 0x7f8edc0a5f40 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:08:36.750 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.743+0000 7f8ee3fe5700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8edc1420c0 0x7f8edc1444a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:36.750 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.743+0000 7f8ee3fe5700 10 monclient(hunting): picked mon.noname-b con 0x7f8edc1420c0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:08:36.751 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.743+0000 7f8ee3fe5700 10 monclient(hunting): start opening mon connection 2023-09-30T02:08:36.751 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.743+0000 7f8ee3fe5700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8edc13bb40 con 0x7f8edc0a5b70 2023-09-30T02:08:36.751 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.743+0000 7f8ee3fe5700 10 monclient(hunting): start opening mon connection 2023-09-30T02:08:36.751 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.743+0000 7f8ee3fe5700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8edc143f80 con 0x7f8edc1420c0 2023-09-30T02:08:36.751 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.743+0000 7f8ee3fe5700 10 monclient(hunting): start opening mon connection 2023-09-30T02:08:36.751 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.743+0000 7f8ee3fe5700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8edc151ec0 con 0x7f8edc0a5f40 2023-09-30T02:08:36.751 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.743+0000 7f8ee3fe5700 10 monclient(hunting): _renew_subs 2023-09-30T02:08:36.751 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.743+0000 7f8ee3fe5700 10 monclient(hunting): authenticate will time out at 1962.046546s 2023-09-30T02:08:36.751 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.744+0000 7f8ee1580700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8edc0a5f40 0x7f8edc151980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:36.751 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.744+0000 7f8ee2582700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8edc1420c0 0x7f8edc1444a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:36.751 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.744+0000 7f8ee2582700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8edc1420c0 0x7f8edc1444a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:48072/0 (socket says 172.21.15.59:48072) 2023-09-30T02:08:36.751 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.744+0000 7f8ee2582700 1 -- 172.21.15.59:0/2881229767 learned_addr learned my addr 172.21.15.59:0/2881229767 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:08:36.752 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.744+0000 7f8ee2582700 10 monclient(hunting): get_auth_request con 0x7f8edc1420c0 auth_method 0 2023-09-30T02:08:36.752 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.744+0000 7f8ee2582700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:08:36.752 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.744+0000 7f8ee2582700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:36.752 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.744+0000 7f8ee2582700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:36.752 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.744+0000 7f8ee1580700 10 monclient(hunting): get_auth_request con 0x7f8edc0a5f40 auth_method 0 2023-09-30T02:08:36.752 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.744+0000 7f8ee1580700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:08:36.752 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.744+0000 7f8ee1580700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:36.752 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.744+0000 7f8ee1580700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:36.752 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.745+0000 7f8ee2582700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:08:36.752 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.745+0000 7f8ee2582700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:08:36.752 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.745+0000 7f8ee2582700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:08:36.752 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.745+0000 7f8ee1d81700 1 --2- 172.21.15.59:0/2881229767 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8edc0a5b70 0x7f8edc151440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:36.753 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.745+0000 7f8ee1580700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:08:36.753 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.745+0000 7f8ee1580700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:08:36.753 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.745+0000 7f8ee1580700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:08:36.753 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.745+0000 7f8ee1d81700 10 monclient(hunting): get_auth_request con 0x7f8edc0a5b70 auth_method 0 2023-09-30T02:08:36.753 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.745+0000 7f8ee1d81700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:08:36.753 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.745+0000 7f8ee1d81700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:36.753 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.745+0000 7f8ee1d81700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:36.753 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.746+0000 7f8ee1d81700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:08:36.753 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.746+0000 7f8ee1d81700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:08:36.753 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.746+0000 7f8ee1d81700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:08:36.754 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.748+0000 7f8ee2582700 10 monclient(hunting): handle_auth_done global_id 4585 payload 824 2023-09-30T02:08:36.754 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.748+0000 7f8ee2582700 1 -- 172.21.15.59:0/2881229767 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8edc0a5f40 msgr2=0x7f8edc151980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:36.754 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.748+0000 7f8ee2582700 1 --2- 172.21.15.59:0/2881229767 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8edc0a5f40 0x7f8edc151980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:36.754 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.748+0000 7f8ee2582700 1 -- 172.21.15.59:0/2881229767 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8edc0a5b70 msgr2=0x7f8edc151440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:36.754 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.748+0000 7f8ee2582700 1 --2- 172.21.15.59:0/2881229767 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8edc0a5b70 0x7f8edc151440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:36.755 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.748+0000 7f8ee2582700 10 monclient: _finish_hunting 0 2023-09-30T02:08:36.755 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.748+0000 7f8ee2582700 1 monclient: found mon.noname-b 2023-09-30T02:08:36.755 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.748+0000 7f8ee2582700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:08:36.755 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.748+0000 7f8ee2582700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:08:36.755 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.748+0000 7f8ee2582700 1 -- 172.21.15.59:0/2881229767 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ed4009a20 con 0x7f8edc1420c0 2023-09-30T02:08:36.755 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.751+0000 7f8ee2582700 1 --2- 172.21.15.59:0/2881229767 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8edc1420c0 0x7f8edc1444a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f8ed4009fd0 tx=0x7f8ed4019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:08:36.755 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.751+0000 7f8ed1ffb700 10 client.?.objecter ms_handle_connect 0x7f8edc1420c0 2023-09-30T02:08:36.755 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.751+0000 7f8ed1ffb700 10 client.?.objecter resend_mon_ops 2023-09-30T02:08:36.756 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.755+0000 7f8ed1ffb700 1 -- 172.21.15.59:0/2881229767 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ed4012dd0 con 0x7f8edc1420c0 2023-09-30T02:08:36.756 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.755+0000 7f8ed1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:08:36.756 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.755+0000 7f8ed1ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:08:36.757 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.755+0000 7f8ed1ffb700 10 monclient: dump: 2023-09-30T02:08:36.757 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:08:36.757 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:36.757 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:36.757 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:36.757 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:08:36.757 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:36.757 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:08:36.757 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:08:36.757 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:08:36.757 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:36.757 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.755+0000 7f8ed1ffb700 10 monclient: _finish_auth 0 2023-09-30T02:08:36.758 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.755+0000 7f8ed1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:08:36.758 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.755+0000 7f8ed1ffb700 1 -- 172.21.15.59:0/2881229767 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ed4027470 con 0x7f8edc1420c0 2023-09-30T02:08:36.758 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.755+0000 7f8ed1ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:08:36.758 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.755+0000 7f8ed1ffb700 1 -- 172.21.15.59:0/2881229767 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ed4027830 con 0x7f8edc1420c0 2023-09-30T02:08:36.758 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.755+0000 7f8ed1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:08:36.758 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.755+0000 7f8ed1ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:08:36.758 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.755+0000 7f8ed1ffb700 10 monclient: dump: 2023-09-30T02:08:36.758 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:08:36.758 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:36.758 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:36.758 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:36.758 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:08:36.759 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:36.759 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:08:36.759 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:08:36.759 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:08:36.759 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:36.759 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.755+0000 7f8ee3fe5700 5 monclient: authenticate success, global_id 4585 2023-09-30T02:08:36.759 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.755+0000 7f8ee3fe5700 10 monclient: _renew_subs 2023-09-30T02:08:36.759 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.755+0000 7f8ee3fe5700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:08:36.759 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.755+0000 7f8ed3fff700 4 set_mon_vals no callback set 2023-09-30T02:08:36.759 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.755+0000 7f8ee3fe5700 1 -- 172.21.15.59:0/2881229767 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8edc1449e0 con 0x7f8edc1420c0 2023-09-30T02:08:36.759 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.756+0000 7f8ee3fe5700 10 client.4585.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:08:36.760 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.756+0000 7f8ee3fe5700 10 monclient: _renew_subs 2023-09-30T02:08:36.760 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.756+0000 7f8ee3fe5700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:08:36.760 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.756+0000 7f8ee3fe5700 1 -- 172.21.15.59:0/2881229767 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8edc144e80 con 0x7f8edc1420c0 2023-09-30T02:08:36.760 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.756+0000 7f8ee3fe5700 1 librados: init done 2023-09-30T02:08:36.760 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.756+0000 7f8ee3fe5700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:08:36.760 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.756+0000 7f8ee3fe5700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:08:36.760 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.756+0000 7f8ee3fe5700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:08:36.761 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.756+0000 7f8ee3fe5700 1 -- 172.21.15.59:0/2881229767 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8edc1441c0 con 0x7f8edc1420c0 2023-09-30T02:08:36.999 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.999+0000 7f8ed1ffb700 1 -- 172.21.15.59:0/2881229767 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f8ed4024070 con 0x7f8edc1420c0 2023-09-30T02:08:37.000 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:36.999+0000 7f8ed1ffb700 1 --2- 172.21.15.59:0/2881229767 >> v2:172.21.15.110:6800/115870 conn(0x7f8ec0068960 0x7f8ec006ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:37.000 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:37.000+0000 7f8ee1d81700 1 --2- 172.21.15.59:0/2881229767 >> v2:172.21.15.110:6800/115870 conn(0x7f8ec0068960 0x7f8ec006ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:37.000 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:37.000+0000 7f8ee1d81700 10 monclient: get_auth_request con 0x7f8ec0068960 auth_method 0 2023-09-30T02:08:37.000 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:37.000+0000 7f8ee1d81700 1 --2- 172.21.15.59:0/2881229767 >> v2:172.21.15.110:6800/115870 conn(0x7f8ec0068960 0x7f8ec006ae10 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f8ed8006d40 tx=0x7f8ed8009250 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:08:37.001 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:37.000+0000 7f8ed1ffb700 10 client.4585.objecter ms_handle_connect 0x7f8ec0068960 2023-09-30T02:08:37.004 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:37.004+0000 7f8ed1ffb700 1 -- 172.21.15.59:0/2881229767 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f8ed40263d0 con 0x7f8edc1420c0 2023-09-30T02:08:37.004 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:37.004+0000 7f8ed1ffb700 10 client.4585.objecter ms_dispatch 0x7f8edc001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:08:37.004 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:37.004+0000 7f8ed1ffb700 3 client.4585.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:08:37.005 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:37.004+0000 7f8ed1ffb700 3 client.4585.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:08:37.005 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:37.004+0000 7f8ed1ffb700 20 client.4585.objecter dump_active .. 0 homeless 2023-09-30T02:08:37.014 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:37.014+0000 7f8ed1ffb700 1 -- 172.21.15.59:0/2881229767 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8ed4053dc0 con 0x7f8edc1420c0 2023-09-30T02:08:37.014 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:37.014+0000 7f8ed1ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:08:37.014 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:37.014+0000 7f8ed1ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:08:37.270 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:37.269+0000 7f8ee3fe5700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:08:37.270 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:37.270+0000 7f8ee3fe5700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:08:37.270 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:37.270+0000 7f8ee3fe5700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:08:37.270 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:37.270+0000 7f8ee3fe5700 1 -- 172.21.15.59:0/2881229767 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f8edc1441c0 con 0x7f8edc1420c0 2023-09-30T02:08:37.274 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:37.275+0000 7f8ed1ffb700 1 -- 172.21.15.59:0/2881229767 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f8edc1441c0 con 0x7f8edc1420c0 2023-09-30T02:08:37.274 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:37.275+0000 7f8ed1ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:08:37.274 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:37.275+0000 7f8ed1ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:08:37.275 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:08:37.275 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:08:37.277 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:37.276+0000 7f8ee3fe5700 10 librados: watch_flush enter 2023-09-30T02:08:37.277 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:37.276+0000 7f8ee3fe5700 10 librados: watch_flush exit 2023-09-30T02:08:37.277 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:37.276+0000 7f8ee3fe5700 20 client.4585.objecter shutdown clearing up homeless session... 2023-09-30T02:08:37.277 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:37.276+0000 7f8ee3fe5700 10 client.4585.objecter successfully canceled tick 2023-09-30T02:08:37.277 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:37.276+0000 7f8ee3fe5700 1 -- 172.21.15.59:0/2881229767 >> v2:172.21.15.110:6800/115870 conn(0x7f8ec0068960 msgr2=0x7f8ec006ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:37.277 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:37.276+0000 7f8ee3fe5700 1 --2- 172.21.15.59:0/2881229767 >> v2:172.21.15.110:6800/115870 conn(0x7f8ec0068960 0x7f8ec006ae10 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f8ed8006d40 tx=0x7f8ed8009250 comp rx=0 tx=0).stop 2023-09-30T02:08:37.278 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:37.276+0000 7f8ee3fe5700 10 monclient: shutdown 2023-09-30T02:08:37.278 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:37.276+0000 7f8ee3fe5700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:08:37.278 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:37.276+0000 7f8ee3fe5700 1 -- 172.21.15.59:0/2881229767 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8edc1420c0 msgr2=0x7f8edc1444a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:37.278 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:37.276+0000 7f8ee3fe5700 1 --2- 172.21.15.59:0/2881229767 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8edc1420c0 0x7f8edc1444a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f8ed4009fd0 tx=0x7f8ed4019000 comp rx=0 tx=0).stop 2023-09-30T02:08:37.278 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:37.276+0000 7f8ee3fe5700 1 -- 172.21.15.59:0/2881229767 shutdown_connections 2023-09-30T02:08:37.278 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:37.276+0000 7f8ee3fe5700 1 --2- 172.21.15.59:0/2881229767 >> v2:172.21.15.110:6800/115870 conn(0x7f8ec0068960 0x7f8ec006ae10 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:37.278 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:37.276+0000 7f8ee3fe5700 1 --2- 172.21.15.59:0/2881229767 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8edc1420c0 0x7f8edc1444a0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:37.278 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:37.276+0000 7f8ee3fe5700 1 --2- 172.21.15.59:0/2881229767 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8edc0a5b70 0x7f8edc151440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:37.278 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:37.276+0000 7f8ee3fe5700 1 --2- 172.21.15.59:0/2881229767 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8edc0a5f40 0x7f8edc151980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:37.278 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:37.276+0000 7f8ee3fe5700 1 -- 172.21.15.59:0/2881229767 >> 172.21.15.59:0/2881229767 conn(0x7f8edc0631f0 msgr2=0x7f8edc0b0be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:08:37.278 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:37.276+0000 7f8ee3fe5700 1 -- 172.21.15.59:0/2881229767 shutdown_connections 2023-09-30T02:08:37.279 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:37.276+0000 7f8ee3fe5700 1 -- 172.21.15.59:0/2881229767 wait complete. 2023-09-30T02:08:37.279 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:37.276+0000 7f8ee3fe5700 1 librados: shutdown 2023-09-30T02:08:40.294 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:08:40.433 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.432+0000 7f3572e56700 10 monclient: get_monmap_and_config 2023-09-30T02:08:40.433 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.432+0000 7f3572e56700 10 monclient: build_initial_monmap 2023-09-30T02:08:40.433 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.432+0000 7f3572e56700 10 monclient: monmap: 2023-09-30T02:08:40.433 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:08:40.433 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:40.433 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:08:40.434066+0000 2023-09-30T02:08:40.433 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:08:40.434066+0000 2023-09-30T02:08:40.434 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:08:40.434 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:40.434 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:08:40.434 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:08:40.434 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:08:40.434 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:08:40.434 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:08:40.434 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:40.436 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.433+0000 7f3572e56700 1 Processor -- start 2023-09-30T02:08:40.436 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.433+0000 7f3572e56700 1 -- start start 2023-09-30T02:08:40.436 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.433+0000 7f3572e56700 10 monclient: init 2023-09-30T02:08:40.436 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.434+0000 7f3572e56700 10 monclient: _reopen_session rank -1 2023-09-30T02:08:40.436 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.434+0000 7f3572e56700 10 monclient: _add_conns ranks=[4,1,3,0,2] 2023-09-30T02:08:40.436 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.434+0000 7f3572e56700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f356c0a5b70 0x7f356c0b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:40.436 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.434+0000 7f3572e56700 10 monclient(hunting): picked mon.noname-c con 0x7f356c0a5b70 addr v2:172.21.15.110:3300/0 2023-09-30T02:08:40.437 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.434+0000 7f3572e56700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f356c0a5f40 addr v1:172.21.15.110:6789/0 2023-09-30T02:08:40.437 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.434+0000 7f3572e56700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f356c142290 0x7f356c13af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:40.437 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.434+0000 7f3572e56700 10 monclient(hunting): picked mon.noname-a con 0x7f356c142290 addr v2:172.21.15.59:3300/0 2023-09-30T02:08:40.437 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.434+0000 7f3572e56700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f356c0a75a0 con 0x7f356c0a5f40 2023-09-30T02:08:40.437 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.434+0000 7f3572e56700 10 monclient(hunting): start opening mon connection 2023-09-30T02:08:40.437 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.434+0000 7f3572e56700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f356c0a63e0 con 0x7f356c142290 2023-09-30T02:08:40.437 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.434+0000 7f3572e56700 10 monclient(hunting): start opening mon connection 2023-09-30T02:08:40.437 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.434+0000 7f3572e56700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f356c1426b0 con 0x7f356c0a5b70 2023-09-30T02:08:40.437 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.434+0000 7f3572e56700 10 monclient(hunting): _renew_subs 2023-09-30T02:08:40.437 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.434+0000 7f3572e56700 10 monclient(hunting): authenticate will time out at 1965.737807s 2023-09-30T02:08:40.437 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.435+0000 7f35713f3700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f356c142290 0x7f356c13af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:40.437 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.435+0000 7f3570bf2700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f356c0a5b70 0x7f356c0b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:40.437 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.435+0000 7f35713f3700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f356c142290 0x7f356c13af30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:56876/0 (socket says 172.21.15.59:56876) 2023-09-30T02:08:40.438 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.436+0000 7f35713f3700 1 -- 172.21.15.59:0/3501555914 learned_addr learned my addr 172.21.15.59:0/3501555914 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:08:40.438 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.436+0000 7f35713f3700 10 monclient(hunting): get_auth_request con 0x7f356c142290 auth_method 0 2023-09-30T02:08:40.438 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.436+0000 7f35713f3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:08:40.438 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.436+0000 7f35713f3700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:40.438 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.436+0000 7f35713f3700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:40.438 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.436+0000 7f3570bf2700 10 monclient(hunting): get_auth_request con 0x7f356c0a5b70 auth_method 0 2023-09-30T02:08:40.438 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.436+0000 7f3570bf2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:08:40.438 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.436+0000 7f3570bf2700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:40.438 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.436+0000 7f3570bf2700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:40.439 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.436+0000 7f35713f3700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:08:40.439 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.436+0000 7f35713f3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:08:40.439 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.436+0000 7f35713f3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:08:40.439 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.437+0000 7f3570bf2700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:08:40.439 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.437+0000 7f3570bf2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:08:40.439 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.437+0000 7f3570bf2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:08:40.440 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.439+0000 7f35713f3700 10 monclient(hunting): handle_auth_done global_id 4578 payload 274 2023-09-30T02:08:40.440 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.439+0000 7f35713f3700 1 -- 172.21.15.59:0/3501555914 >> v2:172.21.15.110:3300/0 conn(0x7f356c0a5b70 msgr2=0x7f356c0b1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:08:40.440 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.439+0000 7f35713f3700 1 --2- 172.21.15.59:0/3501555914 >> v2:172.21.15.110:3300/0 conn(0x7f356c0a5b70 0x7f356c0b1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:40.440 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.439+0000 7f35713f3700 1 -- 172.21.15.59:0/3501555914 >> v1:172.21.15.110:6789/0 conn(0x7f356c0a5f40 legacy=0x7f356c141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:40.440 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.439+0000 7f35713f3700 10 monclient: _finish_hunting 0 2023-09-30T02:08:40.440 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.439+0000 7f35713f3700 1 monclient: found mon.noname-a 2023-09-30T02:08:40.440 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.439+0000 7f35713f3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:08:40.441 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.439+0000 7f35713f3700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.59:3300/0 2023-09-30T02:08:40.441 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.439+0000 7f35713f3700 1 -- 172.21.15.59:0/3501555914 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f356c0af3f0 con 0x7f356c142290 2023-09-30T02:08:40.442 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.442+0000 7f35713f3700 1 --2- 172.21.15.59:0/3501555914 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f356c142290 0x7f356c13af30 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f356c0b49e0 tx=0x7f355c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a31776c626b7492 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:08:40.448 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.447+0000 7f356b7fe700 1 -- 172.21.15.59:0/3501555914 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f355c009ea0 con 0x7f356c142290 2023-09-30T02:08:40.449 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.447+0000 7f356b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:08:40.449 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.447+0000 7f356b7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T02:08:40.449 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.447+0000 7f356b7fe700 10 monclient: dump: 2023-09-30T02:08:40.449 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:08:40.449 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:40.449 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:40.449 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:40.449 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:08:40.449 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:40.449 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:08:40.449 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:08:40.450 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:08:40.450 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:40.450 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.447+0000 7f356b7fe700 10 monclient: _finish_auth 0 2023-09-30T02:08:40.450 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.447+0000 7f356b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:08:40.450 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.447+0000 7f356b7fe700 1 -- 172.21.15.59:0/3501555914 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f355c00b820 con 0x7f356c142290 2023-09-30T02:08:40.450 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.447+0000 7f3572e56700 5 monclient: authenticate success, global_id 4578 2023-09-30T02:08:40.450 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.447+0000 7f3572e56700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:08:40.450 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.447+0000 7f356b7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:08:40.450 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.447+0000 7f3572e56700 10 monclient: get_monmap_and_config success 2023-09-30T02:08:40.450 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.447+0000 7f356b7fe700 1 -- 172.21.15.59:0/3501555914 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f355c00bbc0 con 0x7f356c142290 2023-09-30T02:08:40.450 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.447+0000 7f3572e56700 4 set_mon_vals no callback set 2023-09-30T02:08:40.450 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.447+0000 7f356b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:08:40.451 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.447+0000 7f356b7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T02:08:40.451 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.447+0000 7f356b7fe700 10 monclient: dump: 2023-09-30T02:08:40.451 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:08:40.451 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:40.451 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:40.451 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:40.451 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:08:40.451 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:40.451 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:08:40.451 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:08:40.451 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:08:40.451 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:40.452 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.448+0000 7f3572e56700 10 monclient: shutdown 2023-09-30T02:08:40.452 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.448+0000 7f3572e56700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:08:40.452 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.448+0000 7f3572e56700 1 -- 172.21.15.59:0/3501555914 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f356c142290 msgr2=0x7f356c13af30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:40.452 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.448+0000 7f3572e56700 1 --2- 172.21.15.59:0/3501555914 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f356c142290 0x7f356c13af30 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f356c0b49e0 tx=0x7f355c009250 comp rx=0 tx=0).stop 2023-09-30T02:08:40.452 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.448+0000 7f3572e56700 1 -- 172.21.15.59:0/3501555914 shutdown_connections 2023-09-30T02:08:40.452 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.448+0000 7f3572e56700 1 --2- 172.21.15.59:0/3501555914 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f356c142290 0x7f356c13af30 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:40.452 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.448+0000 7f3572e56700 1 --2- 172.21.15.59:0/3501555914 >> v2:172.21.15.110:3300/0 conn(0x7f356c0a5b70 0x7f356c0b1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:40.452 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.448+0000 7f3572e56700 1 -- 172.21.15.59:0/3501555914 >> 172.21.15.59:0/3501555914 conn(0x7f356c0631f0 msgr2=0x7f356c0635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:08:40.452 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.448+0000 7f3572e56700 1 -- 172.21.15.59:0/3501555914 shutdown_connections 2023-09-30T02:08:40.452 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.448+0000 7f3572e56700 1 -- 172.21.15.59:0/3501555914 wait complete. 2023-09-30T02:08:40.454 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.449+0000 7f3572e56700 10 monclient: build_initial_monmap 2023-09-30T02:08:40.455 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.449+0000 7f3572e56700 10 monclient: monmap: 2023-09-30T02:08:40.455 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:08:40.455 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:08:40.455 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:08:40.455 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:08:40.455 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:08:40.455 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:08:40.455 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:08:40.455 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:08:40.455 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:08:40.455 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:40.456 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.449+0000 7f3572e56700 1 librados: starting msgr at 2023-09-30T02:08:40.456 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.449+0000 7f3572e56700 1 librados: starting objecter 2023-09-30T02:08:40.456 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.450+0000 7f3572e56700 1 Processor -- start 2023-09-30T02:08:40.456 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.450+0000 7f3572e56700 1 -- start start 2023-09-30T02:08:40.456 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.450+0000 7f3572e56700 1 librados: setting wanted keys 2023-09-30T02:08:40.456 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.450+0000 7f3572e56700 1 librados: calling monclient init 2023-09-30T02:08:40.456 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.450+0000 7f3572e56700 10 monclient: init 2023-09-30T02:08:40.456 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.450+0000 7f3572e56700 10 monclient: _reopen_session rank -1 2023-09-30T02:08:40.456 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.450+0000 7f3572e56700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T02:08:40.456 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.450+0000 7f3572e56700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f356c0a5b70 0x7f356c0a7c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:40.456 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.450+0000 7f3572e56700 10 monclient(hunting): picked mon.noname-a con 0x7f356c0a5b70 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:08:40.456 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.450+0000 7f3572e56700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f356c0a5f40 0x7f356c13c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:40.457 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.450+0000 7f3572e56700 10 monclient(hunting): picked mon.noname-c con 0x7f356c0a5f40 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:08:40.457 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.450+0000 7f3572e56700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f356c142290 0x7f356c13caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:40.457 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.450+0000 7f3572e56700 10 monclient(hunting): picked mon.noname-b con 0x7f356c142290 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:08:40.457 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.450+0000 7f3572e56700 10 monclient(hunting): start opening mon connection 2023-09-30T02:08:40.457 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.450+0000 7f3572e56700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f356c0a8320 con 0x7f356c0a5b70 2023-09-30T02:08:40.457 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.450+0000 7f3572e56700 10 monclient(hunting): start opening mon connection 2023-09-30T02:08:40.457 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.450+0000 7f3572e56700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f356c0a9d10 con 0x7f356c142290 2023-09-30T02:08:40.457 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.450+0000 7f3572e56700 10 monclient(hunting): start opening mon connection 2023-09-30T02:08:40.457 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.450+0000 7f3572e56700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f356c0a87b0 con 0x7f356c0a5f40 2023-09-30T02:08:40.457 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.450+0000 7f3572e56700 10 monclient(hunting): _renew_subs 2023-09-30T02:08:40.457 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.450+0000 7f3572e56700 10 monclient(hunting): authenticate will time out at 1965.753947s 2023-09-30T02:08:40.457 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.451+0000 7f35713f3700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f356c142290 0x7f356c13caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:40.458 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.451+0000 7f35713f3700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f356c142290 0x7f356c13caf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:45806/0 (socket says 172.21.15.59:45806) 2023-09-30T02:08:40.458 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.451+0000 7f35713f3700 1 -- 172.21.15.59:0/628475704 learned_addr learned my addr 172.21.15.59:0/628475704 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:08:40.458 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.451+0000 7f35713f3700 10 monclient(hunting): get_auth_request con 0x7f356c142290 auth_method 0 2023-09-30T02:08:40.458 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.451+0000 7f35713f3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:08:40.458 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.451+0000 7f35713f3700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:40.458 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.451+0000 7f35713f3700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:40.458 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.452+0000 7f35713f3700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:08:40.458 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.452+0000 7f35713f3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:08:40.458 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.452+0000 7f35713f3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:08:40.458 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.453+0000 7f356bfff700 1 --2- 172.21.15.59:0/628475704 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f356c0a5f40 0x7f356c13c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:40.458 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.453+0000 7f3570bf2700 1 --2- 172.21.15.59:0/628475704 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f356c0a5b70 0x7f356c0a7c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:40.459 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.454+0000 7f356bfff700 10 monclient(hunting): get_auth_request con 0x7f356c0a5f40 auth_method 0 2023-09-30T02:08:40.459 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.454+0000 7f356bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:08:40.459 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.454+0000 7f356bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:40.459 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.454+0000 7f356bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:40.459 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.454+0000 7f3570bf2700 10 monclient(hunting): get_auth_request con 0x7f356c0a5b70 auth_method 0 2023-09-30T02:08:40.459 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.454+0000 7f3570bf2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:08:40.459 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.454+0000 7f3570bf2700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:40.459 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.454+0000 7f3570bf2700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:40.460 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.455+0000 7f3570bf2700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:08:40.460 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.455+0000 7f3570bf2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:08:40.460 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.455+0000 7f3570bf2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:08:40.460 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.455+0000 7f356bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:08:40.461 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.455+0000 7f356bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:08:40.461 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.455+0000 7f356bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:08:40.461 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.456+0000 7f35713f3700 10 monclient(hunting): handle_auth_done global_id 4594 payload 824 2023-09-30T02:08:40.461 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.456+0000 7f35713f3700 1 -- 172.21.15.59:0/628475704 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f356c0a5f40 msgr2=0x7f356c13c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:40.461 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.456+0000 7f35713f3700 1 --2- 172.21.15.59:0/628475704 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f356c0a5f40 0x7f356c13c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:40.461 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.456+0000 7f35713f3700 1 -- 172.21.15.59:0/628475704 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f356c0a5b70 msgr2=0x7f356c0a7c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:40.461 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.456+0000 7f35713f3700 1 --2- 172.21.15.59:0/628475704 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f356c0a5b70 0x7f356c0a7c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:40.461 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.456+0000 7f35713f3700 10 monclient: _finish_hunting 0 2023-09-30T02:08:40.461 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.456+0000 7f35713f3700 1 monclient: found mon.noname-b 2023-09-30T02:08:40.461 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.456+0000 7f35713f3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:08:40.461 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.456+0000 7f35713f3700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:08:40.462 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.456+0000 7f35713f3700 1 -- 172.21.15.59:0/628475704 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f355c0094f0 con 0x7f356c142290 2023-09-30T02:08:40.462 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.459+0000 7f35713f3700 1 --2- 172.21.15.59:0/628475704 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f356c142290 0x7f356c13caf0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f355c009290 tx=0x7f355c017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:08:40.462 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.459+0000 7f3568ff9700 10 client.?.objecter ms_handle_connect 0x7f356c142290 2023-09-30T02:08:40.462 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.459+0000 7f3568ff9700 10 client.?.objecter resend_mon_ops 2023-09-30T02:08:40.466 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.465+0000 7f3568ff9700 1 -- 172.21.15.59:0/628475704 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f355c011420 con 0x7f356c142290 2023-09-30T02:08:40.466 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.465+0000 7f3568ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:08:40.466 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.465+0000 7f3568ff9700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:08:40.466 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.465+0000 7f3568ff9700 10 monclient: dump: 2023-09-30T02:08:40.466 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:08:40.466 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:40.466 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:40.466 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:40.466 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:08:40.466 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:40.467 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:08:40.467 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:08:40.467 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:08:40.467 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:40.467 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.465+0000 7f3568ff9700 10 monclient: _finish_auth 0 2023-09-30T02:08:40.467 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.465+0000 7f3568ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:08:40.467 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.465+0000 7f3568ff9700 1 -- 172.21.15.59:0/628475704 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f355c011580 con 0x7f356c142290 2023-09-30T02:08:40.467 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.465+0000 7f3572e56700 5 monclient: authenticate success, global_id 4594 2023-09-30T02:08:40.467 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.465+0000 7f3572e56700 10 monclient: _renew_subs 2023-09-30T02:08:40.468 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.465+0000 7f3572e56700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:08:40.468 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.465+0000 7f3572e56700 1 -- 172.21.15.59:0/628475704 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f356c13e2c0 con 0x7f356c142290 2023-09-30T02:08:40.468 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.465+0000 7f3568ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:08:40.468 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.465+0000 7f3568ff9700 1 -- 172.21.15.59:0/628475704 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f355c011940 con 0x7f356c142290 2023-09-30T02:08:40.468 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.465+0000 7f3568ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:08:40.468 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.465+0000 7f3568ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:08:40.468 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.465+0000 7f3568ff9700 10 monclient: dump: 2023-09-30T02:08:40.469 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:08:40.469 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:40.469 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:40.469 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:40.469 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:08:40.469 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:40.469 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:08:40.470 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:08:40.470 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:08:40.470 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:40.470 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.465+0000 7f356affd700 4 set_mon_vals no callback set 2023-09-30T02:08:40.470 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.465+0000 7f3572e56700 10 client.4594.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:08:40.471 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.465+0000 7f3572e56700 10 monclient: _renew_subs 2023-09-30T02:08:40.471 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.465+0000 7f3572e56700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:08:40.471 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.465+0000 7f3572e56700 1 -- 172.21.15.59:0/628475704 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f356c13f0c0 con 0x7f356c142290 2023-09-30T02:08:40.471 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.465+0000 7f3572e56700 1 librados: init done 2023-09-30T02:08:40.472 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.466+0000 7f3572e56700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:08:40.472 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.466+0000 7f3572e56700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:08:40.472 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.466+0000 7f3572e56700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:08:40.472 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.466+0000 7f3572e56700 1 -- 172.21.15.59:0/628475704 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f356c0a84d0 con 0x7f356c142290 2023-09-30T02:08:40.716 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.715+0000 7f3568ff9700 1 -- 172.21.15.59:0/628475704 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f355c011cb0 con 0x7f356c142290 2023-09-30T02:08:40.716 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.715+0000 7f3568ff9700 1 --2- 172.21.15.59:0/628475704 >> v2:172.21.15.110:6800/115870 conn(0x7f354c0689e0 0x7f354c06ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:40.716 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.715+0000 7f3568ff9700 1 -- 172.21.15.59:0/628475704 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f355c085960 con 0x7f356c142290 2023-09-30T02:08:40.716 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.715+0000 7f3568ff9700 10 client.4594.objecter ms_dispatch 0x7f356c001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:08:40.716 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.715+0000 7f3568ff9700 3 client.4594.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:08:40.716 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.715+0000 7f3568ff9700 3 client.4594.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:08:40.717 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.715+0000 7f3568ff9700 20 client.4594.objecter dump_active .. 0 homeless 2023-09-30T02:08:40.717 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.715+0000 7f3570bf2700 1 --2- 172.21.15.59:0/628475704 >> v2:172.21.15.110:6800/115870 conn(0x7f354c0689e0 0x7f354c06ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:40.717 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.715+0000 7f3570bf2700 10 monclient: get_auth_request con 0x7f354c0689e0 auth_method 0 2023-09-30T02:08:40.717 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.716+0000 7f3570bf2700 1 --2- 172.21.15.59:0/628475704 >> v2:172.21.15.110:6800/115870 conn(0x7f354c0689e0 0x7f354c06ae90 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f356400f570 tx=0x7f356400b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:08:40.717 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.716+0000 7f3568ff9700 10 client.4594.objecter ms_handle_connect 0x7f354c0689e0 2023-09-30T02:08:40.724 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.724+0000 7f3568ff9700 1 -- 172.21.15.59:0/628475704 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f355c053280 con 0x7f356c142290 2023-09-30T02:08:40.724 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.724+0000 7f3568ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:08:40.724 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.724+0000 7f3568ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:08:40.981 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.980+0000 7f3572e56700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:08:40.981 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.980+0000 7f3572e56700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:08:40.981 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.980+0000 7f3572e56700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:08:40.981 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.980+0000 7f3572e56700 1 -- 172.21.15.59:0/628475704 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f356c064a00 con 0x7f356c142290 2023-09-30T02:08:40.987 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.988+0000 7f3568ff9700 1 -- 172.21.15.59:0/628475704 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f355c0568a0 con 0x7f356c142290 2023-09-30T02:08:40.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.988+0000 7f3568ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:08:40.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.988+0000 7f3568ff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:08:40.988 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:08:40.988 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:08:40.991 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.989+0000 7f3572e56700 10 librados: watch_flush enter 2023-09-30T02:08:40.991 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.989+0000 7f3572e56700 10 librados: watch_flush exit 2023-09-30T02:08:40.991 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.989+0000 7f3572e56700 20 client.4594.objecter shutdown clearing up homeless session... 2023-09-30T02:08:40.991 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.989+0000 7f3572e56700 10 client.4594.objecter successfully canceled tick 2023-09-30T02:08:40.991 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.990+0000 7f3572e56700 1 -- 172.21.15.59:0/628475704 >> v2:172.21.15.110:6800/115870 conn(0x7f354c0689e0 msgr2=0x7f354c06ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:40.991 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.990+0000 7f3572e56700 1 --2- 172.21.15.59:0/628475704 >> v2:172.21.15.110:6800/115870 conn(0x7f354c0689e0 0x7f354c06ae90 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f356400f570 tx=0x7f356400b040 comp rx=0 tx=0).stop 2023-09-30T02:08:40.991 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.990+0000 7f3572e56700 10 monclient: shutdown 2023-09-30T02:08:40.992 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.990+0000 7f3572e56700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:08:40.992 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.990+0000 7f3572e56700 1 -- 172.21.15.59:0/628475704 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f356c142290 msgr2=0x7f356c13caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:40.992 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.990+0000 7f3572e56700 1 --2- 172.21.15.59:0/628475704 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f356c142290 0x7f356c13caf0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f355c009290 tx=0x7f355c017000 comp rx=0 tx=0).stop 2023-09-30T02:08:40.992 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.990+0000 7f3572e56700 1 -- 172.21.15.59:0/628475704 shutdown_connections 2023-09-30T02:08:40.992 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.990+0000 7f3572e56700 1 --2- 172.21.15.59:0/628475704 >> v2:172.21.15.110:6800/115870 conn(0x7f354c0689e0 0x7f354c06ae90 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:40.992 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.990+0000 7f3572e56700 1 --2- 172.21.15.59:0/628475704 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f356c142290 0x7f356c13caf0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:40.992 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.990+0000 7f3572e56700 1 --2- 172.21.15.59:0/628475704 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f356c0a5b70 0x7f356c0a7c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:40.992 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.990+0000 7f3572e56700 1 --2- 172.21.15.59:0/628475704 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f356c0a5f40 0x7f356c13c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:40.992 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.990+0000 7f3572e56700 1 -- 172.21.15.59:0/628475704 >> 172.21.15.59:0/628475704 conn(0x7f356c0631f0 msgr2=0x7f356c0b3220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:08:40.993 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.990+0000 7f3572e56700 1 -- 172.21.15.59:0/628475704 shutdown_connections 2023-09-30T02:08:40.993 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.990+0000 7f3572e56700 1 -- 172.21.15.59:0/628475704 wait complete. 2023-09-30T02:08:40.993 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:40.990+0000 7f3572e56700 1 librados: shutdown 2023-09-30T02:08:44.005 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:08:44.135 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.135+0000 7f331770e700 10 monclient: get_monmap_and_config 2023-09-30T02:08:44.135 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.135+0000 7f331770e700 10 monclient: build_initial_monmap 2023-09-30T02:08:44.135 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.135+0000 7f331770e700 10 monclient: monmap: 2023-09-30T02:08:44.135 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:08:44.135 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:44.135 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:08:44.136248+0000 2023-09-30T02:08:44.136 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:08:44.136248+0000 2023-09-30T02:08:44.136 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:08:44.136 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:44.136 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:08:44.136 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:08:44.136 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:08:44.136 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:08:44.136 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:08:44.136 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:44.138 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.136+0000 7f331770e700 1 Processor -- start 2023-09-30T02:08:44.138 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.136+0000 7f331770e700 1 -- start start 2023-09-30T02:08:44.138 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.136+0000 7f331770e700 10 monclient: init 2023-09-30T02:08:44.138 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.137+0000 7f331770e700 10 monclient: _reopen_session rank -1 2023-09-30T02:08:44.138 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.137+0000 7f331770e700 10 monclient: _add_conns ranks=[1,2,3,0,4] 2023-09-30T02:08:44.138 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.137+0000 7f331770e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f33100a5b70 addr v1:172.21.15.110:6789/0 2023-09-30T02:08:44.139 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.137+0000 7f331770e700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3310142290 0x7f33100b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:44.139 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.137+0000 7f331770e700 10 monclient(hunting): picked mon.noname-b con 0x7f3310142290 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:08:44.139 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.137+0000 7f331770e700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f33100a5f40 0x7f331013af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:44.139 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.137+0000 7f331770e700 10 monclient(hunting): picked mon.noname-a con 0x7f33100a5f40 addr v2:172.21.15.59:3300/0 2023-09-30T02:08:44.139 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.137+0000 7f331770e700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33100a6380 con 0x7f33100a5b70 2023-09-30T02:08:44.139 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.137+0000 7f331770e700 10 monclient(hunting): start opening mon connection 2023-09-30T02:08:44.139 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.137+0000 7f331770e700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33100a75a0 con 0x7f33100a5f40 2023-09-30T02:08:44.139 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.137+0000 7f331770e700 10 monclient(hunting): start opening mon connection 2023-09-30T02:08:44.139 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.137+0000 7f331770e700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33100af3f0 con 0x7f3310142290 2023-09-30T02:08:44.139 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.137+0000 7f331770e700 10 monclient(hunting): _renew_subs 2023-09-30T02:08:44.139 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.137+0000 7f331770e700 10 monclient(hunting): authenticate will time out at 1969.439529s 2023-09-30T02:08:44.139 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.138+0000 7f33154aa700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f33100a5b70 0x7f3310141b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.59:43834/0 (socket says 172.21.15.59:43834) 2023-09-30T02:08:44.140 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.138+0000 7f33154aa700 1 -- 172.21.15.59:0/2986024682 learned_addr learned my addr 172.21.15.59:0/2986024682 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:08:44.140 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.138+0000 7f3315cab700 1 --2- 172.21.15.59:0/2986024682 >> v2:172.21.15.59:3300/0 conn(0x7f33100a5f40 0x7f331013af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:44.140 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.138+0000 7f3314ca9700 1 --2- 172.21.15.59:0/2986024682 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3310142290 0x7f33100b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:44.140 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.138+0000 7f3314ca9700 10 monclient(hunting): get_auth_request con 0x7f3310142290 auth_method 0 2023-09-30T02:08:44.140 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.138+0000 7f3314ca9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:08:44.140 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.138+0000 7f3314ca9700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:44.140 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.138+0000 7f3314ca9700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:44.140 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.138+0000 7f3315cab700 10 monclient(hunting): get_auth_request con 0x7f33100a5f40 auth_method 0 2023-09-30T02:08:44.140 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.138+0000 7f3315cab700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:08:44.140 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.138+0000 7f3315cab700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:44.140 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.138+0000 7f3315cab700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:44.141 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.139+0000 7f3314ca9700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:08:44.142 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.139+0000 7f3314ca9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:08:44.142 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.139+0000 7f3314ca9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:08:44.142 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.139+0000 7f3315cab700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:08:44.142 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.139+0000 7f3315cab700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:08:44.142 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.139+0000 7f3315cab700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:08:44.142 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.140+0000 7f3303fff700 1 -- 172.21.15.59:0/2986024682 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3296642795 0 0) 0x7f33100a6380 con 0x7f33100a5b70 2023-09-30T02:08:44.142 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.141+0000 7f3303fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:44.142 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.141+0000 7f3303fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:44.142 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.141+0000 7f3303fff700 10 monclient(hunting): my global_id is 4597 2023-09-30T02:08:44.142 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.141+0000 7f3303fff700 1 -- 172.21.15.59:0/2986024682 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32fc003570 con 0x7f33100a5b70 2023-09-30T02:08:44.142 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.142+0000 7f3314ca9700 10 monclient(hunting): handle_auth_done global_id 4520 payload 274 2023-09-30T02:08:44.143 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.142+0000 7f3314ca9700 1 -- 172.21.15.59:0/2986024682 >> v2:172.21.15.59:3300/0 conn(0x7f33100a5f40 msgr2=0x7f331013af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:08:44.143 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.142+0000 7f3314ca9700 1 --2- 172.21.15.59:0/2986024682 >> v2:172.21.15.59:3300/0 conn(0x7f33100a5f40 0x7f331013af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:44.143 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.142+0000 7f3314ca9700 1 -- 172.21.15.59:0/2986024682 >> v1:172.21.15.110:6789/0 conn(0x7f33100a5b70 legacy=0x7f3310141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:44.143 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.142+0000 7f3314ca9700 10 monclient: _finish_hunting 0 2023-09-30T02:08:44.143 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.142+0000 7f3314ca9700 1 monclient: found mon.noname-b 2023-09-30T02:08:44.143 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.142+0000 7f3314ca9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:08:44.143 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.142+0000 7f3314ca9700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.59:3301/0 2023-09-30T02:08:44.143 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.142+0000 7f3314ca9700 1 -- 172.21.15.59:0/2986024682 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f331013b470 con 0x7f3310142290 2023-09-30T02:08:44.143 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.144+0000 7f3314ca9700 1 --2- 172.21.15.59:0/2986024682 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3310142290 0x7f33100b1500 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f33040098e0 tx=0x7f33040091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=57e8813563555c7 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:08:44.149 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.149+0000 7f3303fff700 1 -- 172.21.15.59:0/2986024682 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3304011630 con 0x7f3310142290 2023-09-30T02:08:44.149 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.149+0000 7f3303fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:08:44.149 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.149+0000 7f3303fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:08:44.149 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.149+0000 7f3303fff700 10 monclient: dump: 2023-09-30T02:08:44.149 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:08:44.149 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:44.150 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:44.150 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:44.150 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:08:44.150 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:44.150 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:08:44.150 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:08:44.150 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:08:44.150 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:44.150 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.149+0000 7f3303fff700 10 monclient: _finish_auth 0 2023-09-30T02:08:44.150 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.149+0000 7f3303fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:08:44.150 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.149+0000 7f331770e700 5 monclient: authenticate success, global_id 4520 2023-09-30T02:08:44.150 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.149+0000 7f3303fff700 1 -- 172.21.15.59:0/2986024682 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3304011790 con 0x7f3310142290 2023-09-30T02:08:44.151 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.149+0000 7f331770e700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:08:44.151 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.149+0000 7f3303fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:08:44.151 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.149+0000 7f331770e700 10 monclient: get_monmap_and_config success 2023-09-30T02:08:44.151 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.149+0000 7f3303fff700 1 -- 172.21.15.59:0/2986024682 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3304011b30 con 0x7f3310142290 2023-09-30T02:08:44.151 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.149+0000 7f331770e700 4 set_mon_vals no callback set 2023-09-30T02:08:44.151 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.149+0000 7f3303fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:08:44.151 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.149+0000 7f3303fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:08:44.151 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.149+0000 7f3303fff700 10 monclient: dump: 2023-09-30T02:08:44.151 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:08:44.151 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:44.151 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:44.151 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:44.152 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:08:44.152 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:44.152 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:08:44.152 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:08:44.152 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:08:44.152 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:44.152 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.149+0000 7f331770e700 10 monclient: shutdown 2023-09-30T02:08:44.152 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.150+0000 7f331770e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:08:44.152 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.150+0000 7f331770e700 1 -- 172.21.15.59:0/2986024682 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3310142290 msgr2=0x7f33100b1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:44.152 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.150+0000 7f331770e700 1 --2- 172.21.15.59:0/2986024682 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3310142290 0x7f33100b1500 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f33040098e0 tx=0x7f33040091a0 comp rx=0 tx=0).stop 2023-09-30T02:08:44.152 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.150+0000 7f331770e700 1 -- 172.21.15.59:0/2986024682 shutdown_connections 2023-09-30T02:08:44.152 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.150+0000 7f331770e700 1 --2- 172.21.15.59:0/2986024682 >> v2:172.21.15.59:3300/0 conn(0x7f33100a5f40 0x7f331013af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:44.152 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.150+0000 7f331770e700 1 --2- 172.21.15.59:0/2986024682 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3310142290 0x7f33100b1500 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:44.153 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.150+0000 7f331770e700 1 -- 172.21.15.59:0/2986024682 >> 172.21.15.59:0/2986024682 conn(0x7f33100631f0 msgr2=0x7f33100635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:08:44.153 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.150+0000 7f331770e700 1 -- 172.21.15.59:0/2986024682 shutdown_connections 2023-09-30T02:08:44.153 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.150+0000 7f331770e700 1 -- 172.21.15.59:0/2986024682 wait complete. 2023-09-30T02:08:44.155 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.151+0000 7f331770e700 10 monclient: build_initial_monmap 2023-09-30T02:08:44.155 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.151+0000 7f331770e700 10 monclient: monmap: 2023-09-30T02:08:44.155 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:08:44.155 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:08:44.156 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:08:44.156 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:08:44.156 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:08:44.156 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:08:44.156 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:08:44.156 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:08:44.156 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:08:44.156 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:44.156 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.151+0000 7f331770e700 1 librados: starting msgr at 2023-09-30T02:08:44.156 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.151+0000 7f331770e700 1 librados: starting objecter 2023-09-30T02:08:44.156 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.151+0000 7f331770e700 1 Processor -- start 2023-09-30T02:08:44.157 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.151+0000 7f331770e700 1 -- start start 2023-09-30T02:08:44.157 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.151+0000 7f331770e700 1 librados: setting wanted keys 2023-09-30T02:08:44.157 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.151+0000 7f331770e700 1 librados: calling monclient init 2023-09-30T02:08:44.157 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.151+0000 7f331770e700 10 monclient: init 2023-09-30T02:08:44.157 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.152+0000 7f331770e700 10 monclient: _reopen_session rank -1 2023-09-30T02:08:44.157 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.152+0000 7f331770e700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T02:08:44.157 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.152+0000 7f331770e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33100a5b70 0x7f331013eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:44.157 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.152+0000 7f331770e700 10 monclient(hunting): picked mon.noname-a con 0x7f33100a5b70 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:08:44.157 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.152+0000 7f331770e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f33100a5f40 0x7f33100a7d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:44.157 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.152+0000 7f331770e700 10 monclient(hunting): picked mon.noname-b con 0x7f33100a5f40 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:08:44.157 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.152+0000 7f331770e700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3310142290 0x7f331013c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:44.158 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.152+0000 7f331770e700 10 monclient(hunting): picked mon.noname-c con 0x7f3310142290 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:08:44.158 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.152+0000 7f331770e700 10 monclient(hunting): start opening mon connection 2023-09-30T02:08:44.158 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.152+0000 7f331770e700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f331013c1f0 con 0x7f33100a5b70 2023-09-30T02:08:44.158 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.152+0000 7f331770e700 10 monclient(hunting): start opening mon connection 2023-09-30T02:08:44.158 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.152+0000 7f331770e700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f331013dcc0 con 0x7f33100a5f40 2023-09-30T02:08:44.158 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.152+0000 7f331770e700 10 monclient(hunting): start opening mon connection 2023-09-30T02:08:44.158 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.152+0000 7f331770e700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f331013de30 con 0x7f3310142290 2023-09-30T02:08:44.158 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.152+0000 7f331770e700 10 monclient(hunting): _renew_subs 2023-09-30T02:08:44.158 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.152+0000 7f331770e700 10 monclient(hunting): authenticate will time out at 1969.454848s 2023-09-30T02:08:44.158 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.153+0000 7f33154aa700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33100a5b70 0x7f331013eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:44.158 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.153+0000 7f3314ca9700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f33100a5f40 0x7f33100a7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:44.159 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.153+0000 7f3315cab700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3310142290 0x7f331013c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:44.159 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.153+0000 7f3314ca9700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f33100a5f40 0x7f33100a7d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:45814/0 (socket says 172.21.15.59:45814) 2023-09-30T02:08:44.159 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.153+0000 7f3314ca9700 1 -- 172.21.15.59:0/3416328238 learned_addr learned my addr 172.21.15.59:0/3416328238 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:08:44.159 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.153+0000 7f3314ca9700 10 monclient(hunting): get_auth_request con 0x7f33100a5f40 auth_method 0 2023-09-30T02:08:44.159 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.153+0000 7f3314ca9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:08:44.159 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.153+0000 7f3314ca9700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:44.159 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.153+0000 7f3314ca9700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:44.159 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.153+0000 7f3315cab700 10 monclient(hunting): get_auth_request con 0x7f3310142290 auth_method 0 2023-09-30T02:08:44.159 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.153+0000 7f3315cab700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:08:44.159 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.153+0000 7f3315cab700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:44.159 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.153+0000 7f3315cab700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:44.159 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.154+0000 7f33154aa700 10 monclient(hunting): get_auth_request con 0x7f33100a5b70 auth_method 0 2023-09-30T02:08:44.160 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.154+0000 7f33154aa700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:08:44.160 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.154+0000 7f33154aa700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:44.160 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.154+0000 7f33154aa700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:44.160 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.154+0000 7f3314ca9700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:08:44.160 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.154+0000 7f3314ca9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:08:44.160 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.154+0000 7f3314ca9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:08:44.160 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.154+0000 7f3315cab700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:08:44.160 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.154+0000 7f3315cab700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:08:44.160 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.154+0000 7f3315cab700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:08:44.160 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.154+0000 7f33154aa700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:08:44.160 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.154+0000 7f33154aa700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:08:44.161 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.154+0000 7f33154aa700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:08:44.161 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.157+0000 7f3314ca9700 10 monclient(hunting): handle_auth_done global_id 4600 payload 824 2023-09-30T02:08:44.161 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.157+0000 7f3314ca9700 1 -- 172.21.15.59:0/3416328238 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3310142290 msgr2=0x7f331013c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:44.162 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.157+0000 7f3314ca9700 1 --2- 172.21.15.59:0/3416328238 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3310142290 0x7f331013c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:44.162 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.157+0000 7f3314ca9700 1 -- 172.21.15.59:0/3416328238 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33100a5b70 msgr2=0x7f331013eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:44.162 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.157+0000 7f3314ca9700 1 --2- 172.21.15.59:0/3416328238 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33100a5b70 0x7f331013eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:44.162 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.157+0000 7f3314ca9700 10 monclient: _finish_hunting 0 2023-09-30T02:08:44.162 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.157+0000 7f3314ca9700 1 monclient: found mon.noname-b 2023-09-30T02:08:44.162 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.157+0000 7f3314ca9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:08:44.162 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.157+0000 7f3314ca9700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:08:44.162 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.157+0000 7f3314ca9700 1 -- 172.21.15.59:0/3416328238 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3304009420 con 0x7f33100a5f40 2023-09-30T02:08:44.162 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.160+0000 7f3314ca9700 1 --2- 172.21.15.59:0/3416328238 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f33100a5f40 0x7f33100a7d70 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f3304009ed0 tx=0x7f3304017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:08:44.162 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.160+0000 7f33017fa700 10 client.?.objecter ms_handle_connect 0x7f33100a5f40 2023-09-30T02:08:44.162 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.160+0000 7f33017fa700 10 client.?.objecter resend_mon_ops 2023-09-30T02:08:44.164 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.164+0000 7f33017fa700 1 -- 172.21.15.59:0/3416328238 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3304011900 con 0x7f33100a5f40 2023-09-30T02:08:44.164 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.164+0000 7f33017fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:08:44.164 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.164+0000 7f33017fa700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:08:44.164 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.164+0000 7f33017fa700 10 monclient: dump: 2023-09-30T02:08:44.164 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:08:44.164 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:44.164 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:44.165 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:44.165 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:08:44.165 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:44.165 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:08:44.165 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:08:44.165 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:08:44.165 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:44.165 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.164+0000 7f33017fa700 10 monclient: _finish_auth 0 2023-09-30T02:08:44.165 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.164+0000 7f33017fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:08:44.165 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.164+0000 7f331770e700 5 monclient: authenticate success, global_id 4600 2023-09-30T02:08:44.165 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.164+0000 7f33017fa700 1 -- 172.21.15.59:0/3416328238 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3304011a80 con 0x7f33100a5f40 2023-09-30T02:08:44.165 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.164+0000 7f331770e700 10 monclient: _renew_subs 2023-09-30T02:08:44.166 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.164+0000 7f331770e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:08:44.166 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.164+0000 7f331770e700 1 -- 172.21.15.59:0/3416328238 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33100a82b0 con 0x7f33100a5f40 2023-09-30T02:08:44.166 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.164+0000 7f33017fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:08:44.166 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.164+0000 7f33017fa700 1 -- 172.21.15.59:0/3416328238 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3304025640 con 0x7f33100a5f40 2023-09-30T02:08:44.166 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.164+0000 7f33017fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:08:44.166 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.164+0000 7f33017fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:08:44.166 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.164+0000 7f33017fa700 10 monclient: dump: 2023-09-30T02:08:44.166 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:08:44.166 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:44.166 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:44.166 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:44.166 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:08:44.167 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:44.167 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:08:44.167 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:08:44.167 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:08:44.167 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:44.167 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.164+0000 7f33037fe700 4 set_mon_vals no callback set 2023-09-30T02:08:44.167 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.164+0000 7f331770e700 10 client.4600.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:08:44.167 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.164+0000 7f331770e700 10 monclient: _renew_subs 2023-09-30T02:08:44.167 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.164+0000 7f331770e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:08:44.167 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.164+0000 7f331770e700 1 -- 172.21.15.59:0/3416328238 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f331013cd80 con 0x7f33100a5f40 2023-09-30T02:08:44.167 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.164+0000 7f331770e700 1 librados: init done 2023-09-30T02:08:44.167 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.165+0000 7f331770e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:08:44.167 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.165+0000 7f331770e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:08:44.168 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.165+0000 7f331770e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:08:44.168 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.165+0000 7f331770e700 1 -- 172.21.15.59:0/3416328238 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f331013c3a0 con 0x7f33100a5f40 2023-09-30T02:08:44.412 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.412+0000 7f33017fa700 1 -- 172.21.15.59:0/3416328238 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f33040259f0 con 0x7f33100a5f40 2023-09-30T02:08:44.413 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.413+0000 7f33017fa700 1 --2- 172.21.15.59:0/3416328238 >> v2:172.21.15.110:6800/115870 conn(0x7f32f4068940 0x7f32f406adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:44.413 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.413+0000 7f33017fa700 1 -- 172.21.15.59:0/3416328238 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f3304085b80 con 0x7f33100a5f40 2023-09-30T02:08:44.413 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.413+0000 7f33017fa700 10 client.4600.objecter ms_dispatch 0x7f3310001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:08:44.413 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.413+0000 7f33017fa700 3 client.4600.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:08:44.413 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.413+0000 7f33017fa700 3 client.4600.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:08:44.413 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.413+0000 7f33017fa700 20 client.4600.objecter dump_active .. 0 homeless 2023-09-30T02:08:44.413 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.413+0000 7f33154aa700 1 --2- 172.21.15.59:0/3416328238 >> v2:172.21.15.110:6800/115870 conn(0x7f32f4068940 0x7f32f406adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:44.413 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.413+0000 7f33154aa700 10 monclient: get_auth_request con 0x7f32f4068940 auth_method 0 2023-09-30T02:08:44.414 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.414+0000 7f33154aa700 1 --2- 172.21.15.59:0/3416328238 >> v2:172.21.15.110:6800/115870 conn(0x7f32f4068940 0x7f32f406adf0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f330c007fd0 tx=0x7f330c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:08:44.414 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.414+0000 7f33017fa700 10 client.4600.objecter ms_handle_connect 0x7f32f4068940 2023-09-30T02:08:44.421 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.421+0000 7f33017fa700 1 -- 172.21.15.59:0/3416328238 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3304053420 con 0x7f33100a5f40 2023-09-30T02:08:44.421 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.421+0000 7f33017fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:08:44.421 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.421+0000 7f33017fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:08:44.676 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.677+0000 7f331770e700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:08:44.676 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.677+0000 7f331770e700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:08:44.676 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.677+0000 7f331770e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:08:44.677 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.677+0000 7f331770e700 1 -- 172.21.15.59:0/3416328238 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f331013c3a0 con 0x7f33100a5f40 2023-09-30T02:08:44.680 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.681+0000 7f33017fa700 1 -- 172.21.15.59:0/3416328238 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f3304015100 con 0x7f33100a5f40 2023-09-30T02:08:44.681 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.681+0000 7f33017fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:08:44.681 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.681+0000 7f33017fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:08:44.681 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:08:44.681 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:08:44.684 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.684+0000 7f331770e700 10 librados: watch_flush enter 2023-09-30T02:08:44.684 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.684+0000 7f331770e700 10 librados: watch_flush exit 2023-09-30T02:08:44.684 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.684+0000 7f331770e700 20 client.4600.objecter shutdown clearing up homeless session... 2023-09-30T02:08:44.684 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.684+0000 7f331770e700 10 client.4600.objecter successfully canceled tick 2023-09-30T02:08:44.684 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.684+0000 7f331770e700 1 -- 172.21.15.59:0/3416328238 >> v2:172.21.15.110:6800/115870 conn(0x7f32f4068940 msgr2=0x7f32f406adf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:44.684 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.684+0000 7f331770e700 1 --2- 172.21.15.59:0/3416328238 >> v2:172.21.15.110:6800/115870 conn(0x7f32f4068940 0x7f32f406adf0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f330c007fd0 tx=0x7f330c009040 comp rx=0 tx=0).stop 2023-09-30T02:08:44.684 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.684+0000 7f331770e700 10 monclient: shutdown 2023-09-30T02:08:44.684 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.684+0000 7f331770e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:08:44.684 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.684+0000 7f331770e700 1 -- 172.21.15.59:0/3416328238 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f33100a5f40 msgr2=0x7f33100a7d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:44.685 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.684+0000 7f331770e700 1 --2- 172.21.15.59:0/3416328238 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f33100a5f40 0x7f33100a7d70 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f3304009ed0 tx=0x7f3304017000 comp rx=0 tx=0).stop 2023-09-30T02:08:44.685 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.684+0000 7f331770e700 1 -- 172.21.15.59:0/3416328238 shutdown_connections 2023-09-30T02:08:44.685 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.684+0000 7f331770e700 1 --2- 172.21.15.59:0/3416328238 >> v2:172.21.15.110:6800/115870 conn(0x7f32f4068940 0x7f32f406adf0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:44.685 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.684+0000 7f331770e700 1 --2- 172.21.15.59:0/3416328238 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3310142290 0x7f331013c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:44.685 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.684+0000 7f331770e700 1 --2- 172.21.15.59:0/3416328238 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f33100a5b70 0x7f331013eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:44.685 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.684+0000 7f331770e700 1 --2- 172.21.15.59:0/3416328238 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f33100a5f40 0x7f33100a7d70 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:44.685 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.684+0000 7f331770e700 1 -- 172.21.15.59:0/3416328238 >> 172.21.15.59:0/3416328238 conn(0x7f33100631f0 msgr2=0x7f3310141b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:08:44.685 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.684+0000 7f331770e700 1 -- 172.21.15.59:0/3416328238 shutdown_connections 2023-09-30T02:08:44.685 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.684+0000 7f331770e700 1 -- 172.21.15.59:0/3416328238 wait complete. 2023-09-30T02:08:44.685 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:44.684+0000 7f331770e700 1 librados: shutdown 2023-09-30T02:08:47.699 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:08:47.839 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.840+0000 7f82776b6700 10 monclient: get_monmap_and_config 2023-09-30T02:08:47.839 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.840+0000 7f82776b6700 10 monclient: build_initial_monmap 2023-09-30T02:08:47.840 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.840+0000 7f82776b6700 10 monclient: monmap: 2023-09-30T02:08:47.840 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:08:47.840 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:47.840 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:08:47.840543+0000 2023-09-30T02:08:47.840 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:08:47.840543+0000 2023-09-30T02:08:47.840 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:08:47.840 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:47.840 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:08:47.840 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:08:47.840 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:08:47.840 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:08:47.840 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:08:47.841 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:47.842 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.840+0000 7f82776b6700 1 Processor -- start 2023-09-30T02:08:47.842 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.841+0000 7f82776b6700 1 -- start start 2023-09-30T02:08:47.842 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.841+0000 7f82776b6700 10 monclient: init 2023-09-30T02:08:47.842 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.841+0000 7f82776b6700 10 monclient: _reopen_session rank -1 2023-09-30T02:08:47.843 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.841+0000 7f82776b6700 10 monclient: _add_conns ranks=[3,2,1,4,0] 2023-09-30T02:08:47.843 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.841+0000 7f82776b6700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f82700a5b70 0x7f82700b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:47.843 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.841+0000 7f82776b6700 10 monclient(hunting): picked mon.noname-a con 0x7f82700a5b70 addr v2:172.21.15.59:3300/0 2023-09-30T02:08:47.843 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.841+0000 7f82776b6700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f82700a5f40 0x7f8270141b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:47.843 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.841+0000 7f82776b6700 10 monclient(hunting): picked mon.noname-b con 0x7f82700a5f40 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:08:47.843 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.841+0000 7f82776b6700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f82701420c0 addr v1:172.21.15.110:6789/0 2023-09-30T02:08:47.843 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.841+0000 7f82776b6700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82700a63b0 con 0x7f82701420c0 2023-09-30T02:08:47.843 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.841+0000 7f82776b6700 10 monclient(hunting): start opening mon connection 2023-09-30T02:08:47.843 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.841+0000 7f82776b6700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82700a75a0 con 0x7f82700a5b70 2023-09-30T02:08:47.843 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.841+0000 7f82776b6700 10 monclient(hunting): start opening mon connection 2023-09-30T02:08:47.843 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.841+0000 7f82776b6700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82700af3f0 con 0x7f82700a5f40 2023-09-30T02:08:47.843 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.841+0000 7f82776b6700 10 monclient(hunting): _renew_subs 2023-09-30T02:08:47.844 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.841+0000 7f82776b6700 10 monclient(hunting): authenticate will time out at 1973.143815s 2023-09-30T02:08:47.844 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.842+0000 7f8275452700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f82700a5b70 0x7f82700b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:47.844 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.842+0000 7f8274c51700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f82700a5f40 0x7f8270141b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:47.844 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.842+0000 7f8275c53700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f82701420c0 0x7f827013e230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.59:51314/0 (socket says 172.21.15.59:51314) 2023-09-30T02:08:47.844 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.842+0000 7f8275c53700 1 -- 172.21.15.59:0/2310319352 learned_addr learned my addr 172.21.15.59:0/2310319352 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:08:47.844 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.842+0000 7f8275452700 10 monclient(hunting): get_auth_request con 0x7f82700a5b70 auth_method 0 2023-09-30T02:08:47.844 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.842+0000 7f8275452700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:08:47.844 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.842+0000 7f8275452700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:47.845 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.842+0000 7f8275452700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:47.845 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.842+0000 7f8274c51700 10 monclient(hunting): get_auth_request con 0x7f82700a5f40 auth_method 0 2023-09-30T02:08:47.845 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.842+0000 7f8274c51700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:08:47.845 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.842+0000 7f8274c51700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:47.845 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.842+0000 7f8274c51700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:47.846 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.843+0000 7f8275452700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:08:47.846 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.843+0000 7f8275452700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:08:47.846 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.843+0000 7f8275452700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:08:47.846 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.843+0000 7f8274c51700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:08:47.846 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.843+0000 7f8274c51700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:08:47.847 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.843+0000 7f8274c51700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:08:47.847 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.845+0000 7f8263fff700 1 -- 172.21.15.59:0/2310319352 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 627869574 0 0) 0x7f82700a63b0 con 0x7f82701420c0 2023-09-30T02:08:47.847 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.845+0000 7f8263fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:47.847 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.845+0000 7f8263fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:47.847 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.845+0000 7f8263fff700 10 monclient(hunting): my global_id is 4603 2023-09-30T02:08:47.847 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.845+0000 7f8263fff700 1 -- 172.21.15.59:0/2310319352 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f825c003570 con 0x7f82701420c0 2023-09-30T02:08:47.847 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.847+0000 7f8274c51700 10 monclient(hunting): handle_auth_done global_id 4526 payload 274 2023-09-30T02:08:47.847 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.847+0000 7f8274c51700 1 -- 172.21.15.59:0/2310319352 >> v2:172.21.15.59:3300/0 conn(0x7f82700a5b70 msgr2=0x7f82700b1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:08:47.847 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.847+0000 7f8274c51700 1 --2- 172.21.15.59:0/2310319352 >> v2:172.21.15.59:3300/0 conn(0x7f82700a5b70 0x7f82700b1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:47.847 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.847+0000 7f8274c51700 1 -- 172.21.15.59:0/2310319352 >> v1:172.21.15.110:6789/0 conn(0x7f82701420c0 legacy=0x7f827013e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:47.847 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.847+0000 7f8274c51700 10 monclient: _finish_hunting 0 2023-09-30T02:08:47.847 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.847+0000 7f8274c51700 1 monclient: found mon.noname-b 2023-09-30T02:08:47.848 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.847+0000 7f8274c51700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:08:47.848 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.847+0000 7f8274c51700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.59:3301/0 2023-09-30T02:08:47.848 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.847+0000 7f8274c51700 1 -- 172.21.15.59:0/2310319352 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8270142500 con 0x7f82700a5f40 2023-09-30T02:08:47.848 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.849+0000 7f8274c51700 1 --2- 172.21.15.59:0/2310319352 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f82700a5f40 0x7f8270141b80 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f82640098e0 tx=0x7f82640091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc94c1aa221d096a server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:08:47.853 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.854+0000 7f8263fff700 1 -- 172.21.15.59:0/2310319352 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8264011630 con 0x7f82700a5f40 2023-09-30T02:08:47.854 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.854+0000 7f8263fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:08:47.854 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.854+0000 7f8263fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:08:47.854 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.854+0000 7f8263fff700 10 monclient: dump: 2023-09-30T02:08:47.854 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:08:47.854 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:47.854 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:47.854 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:47.854 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:08:47.854 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:47.854 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:08:47.854 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:08:47.855 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:08:47.855 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:47.855 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.854+0000 7f8263fff700 10 monclient: _finish_auth 0 2023-09-30T02:08:47.855 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.854+0000 7f8263fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:08:47.855 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.854+0000 7f8263fff700 1 -- 172.21.15.59:0/2310319352 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8264011790 con 0x7f82700a5f40 2023-09-30T02:08:47.855 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.854+0000 7f82776b6700 5 monclient: authenticate success, global_id 4526 2023-09-30T02:08:47.855 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.854+0000 7f82776b6700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:08:47.855 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.854+0000 7f8263fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:08:47.855 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.854+0000 7f82776b6700 10 monclient: get_monmap_and_config success 2023-09-30T02:08:47.855 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.854+0000 7f8263fff700 1 -- 172.21.15.59:0/2310319352 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8264011b30 con 0x7f82700a5f40 2023-09-30T02:08:47.855 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.854+0000 7f82776b6700 4 set_mon_vals no callback set 2023-09-30T02:08:47.855 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.854+0000 7f8263fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:08:47.856 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.854+0000 7f8263fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:08:47.856 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.854+0000 7f8263fff700 10 monclient: dump: 2023-09-30T02:08:47.856 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:08:47.856 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:47.856 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:47.856 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:47.856 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:08:47.856 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:47.856 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:08:47.856 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:08:47.856 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:08:47.856 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:47.859 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.855+0000 7f82776b6700 10 monclient: shutdown 2023-09-30T02:08:47.859 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.855+0000 7f82776b6700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:08:47.860 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.855+0000 7f82776b6700 1 -- 172.21.15.59:0/2310319352 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f82700a5f40 msgr2=0x7f8270141b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:47.860 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.855+0000 7f82776b6700 1 --2- 172.21.15.59:0/2310319352 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f82700a5f40 0x7f8270141b80 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f82640098e0 tx=0x7f82640091a0 comp rx=0 tx=0).stop 2023-09-30T02:08:47.860 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.855+0000 7f82776b6700 1 -- 172.21.15.59:0/2310319352 shutdown_connections 2023-09-30T02:08:47.860 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.855+0000 7f82776b6700 1 --2- 172.21.15.59:0/2310319352 >> v2:172.21.15.59:3300/0 conn(0x7f82700a5b70 0x7f82700b1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:47.860 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.855+0000 7f82776b6700 1 --2- 172.21.15.59:0/2310319352 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f82700a5f40 0x7f8270141b80 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:47.860 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.855+0000 7f82776b6700 1 -- 172.21.15.59:0/2310319352 >> 172.21.15.59:0/2310319352 conn(0x7f82700631f0 msgr2=0x7f82700635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:08:47.860 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.855+0000 7f82776b6700 1 -- 172.21.15.59:0/2310319352 shutdown_connections 2023-09-30T02:08:47.860 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.855+0000 7f82776b6700 1 -- 172.21.15.59:0/2310319352 wait complete. 2023-09-30T02:08:47.860 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.856+0000 7f82776b6700 10 monclient: build_initial_monmap 2023-09-30T02:08:47.860 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.856+0000 7f82776b6700 10 monclient: monmap: 2023-09-30T02:08:47.860 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:08:47.860 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:08:47.861 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:08:47.861 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:08:47.861 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:08:47.861 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:08:47.861 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:08:47.861 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:08:47.861 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:08:47.861 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:47.861 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.856+0000 7f82776b6700 1 librados: starting msgr at 2023-09-30T02:08:47.861 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.856+0000 7f82776b6700 1 librados: starting objecter 2023-09-30T02:08:47.861 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.856+0000 7f82776b6700 1 Processor -- start 2023-09-30T02:08:47.862 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.856+0000 7f82776b6700 1 -- start start 2023-09-30T02:08:47.862 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.856+0000 7f82776b6700 1 librados: setting wanted keys 2023-09-30T02:08:47.862 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.856+0000 7f82776b6700 1 librados: calling monclient init 2023-09-30T02:08:47.862 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.856+0000 7f82776b6700 10 monclient: init 2023-09-30T02:08:47.862 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.857+0000 7f82776b6700 10 monclient: _reopen_session rank -1 2023-09-30T02:08:47.862 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.857+0000 7f82776b6700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T02:08:47.862 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.857+0000 7f82776b6700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f82700a5b70 0x7f8270140070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:47.862 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.857+0000 7f82776b6700 10 monclient(hunting): picked mon.noname-b con 0x7f82700a5b70 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:08:47.862 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.857+0000 7f82776b6700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f82700a5f40 0x7f827013c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:47.862 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.857+0000 7f82776b6700 10 monclient(hunting): picked mon.noname-c con 0x7f82700a5f40 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:08:47.862 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.857+0000 7f82776b6700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82701420c0 0x7f8270151840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:47.862 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.857+0000 7f82776b6700 10 monclient(hunting): picked mon.noname-a con 0x7f82701420c0 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:08:47.863 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.857+0000 7f82776b6700 10 monclient(hunting): start opening mon connection 2023-09-30T02:08:47.863 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.857+0000 7f82776b6700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8270143b30 con 0x7f82701420c0 2023-09-30T02:08:47.863 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.857+0000 7f82776b6700 10 monclient(hunting): start opening mon connection 2023-09-30T02:08:47.863 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.857+0000 7f82776b6700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f827013c030 con 0x7f82700a5b70 2023-09-30T02:08:47.863 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.857+0000 7f82776b6700 10 monclient(hunting): start opening mon connection 2023-09-30T02:08:47.863 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.857+0000 7f82776b6700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8270143d30 con 0x7f82700a5f40 2023-09-30T02:08:47.863 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.857+0000 7f82776b6700 10 monclient(hunting): _renew_subs 2023-09-30T02:08:47.863 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.857+0000 7f82776b6700 10 monclient(hunting): authenticate will time out at 1973.159411s 2023-09-30T02:08:47.863 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.858+0000 7f8275452700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f82700a5b70 0x7f8270140070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:47.863 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.858+0000 7f8275c53700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82701420c0 0x7f8270151840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:47.863 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.858+0000 7f8275452700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f82700a5b70 0x7f8270140070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:53030/0 (socket says 172.21.15.59:53030) 2023-09-30T02:08:47.863 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.858+0000 7f8275452700 1 -- 172.21.15.59:0/1638327888 learned_addr learned my addr 172.21.15.59:0/1638327888 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:08:47.864 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.858+0000 7f8275452700 10 monclient(hunting): get_auth_request con 0x7f82700a5b70 auth_method 0 2023-09-30T02:08:47.864 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.858+0000 7f8275452700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:08:47.864 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.858+0000 7f8275452700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:47.864 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.858+0000 7f8275452700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:47.864 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.858+0000 7f8275c53700 10 monclient(hunting): get_auth_request con 0x7f82701420c0 auth_method 0 2023-09-30T02:08:47.864 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.858+0000 7f8275c53700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:08:47.864 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.858+0000 7f8275c53700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:47.864 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.858+0000 7f8275c53700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:47.864 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.859+0000 7f8274c51700 1 --2- 172.21.15.59:0/1638327888 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f82700a5f40 0x7f827013c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:47.864 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.859+0000 7f8275452700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:08:47.864 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.859+0000 7f8275452700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:08:47.864 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.859+0000 7f8275452700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:08:47.864 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.859+0000 7f8275c53700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:08:47.865 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.859+0000 7f8275c53700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:08:47.865 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.859+0000 7f8275c53700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:08:47.865 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.859+0000 7f8274c51700 10 monclient(hunting): get_auth_request con 0x7f82700a5f40 auth_method 0 2023-09-30T02:08:47.865 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.859+0000 7f8274c51700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:08:47.865 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.859+0000 7f8274c51700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:47.865 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.859+0000 7f8274c51700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:47.865 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.860+0000 7f8274c51700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:08:47.865 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.860+0000 7f8274c51700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:08:47.865 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.860+0000 7f8274c51700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:08:47.866 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.863+0000 7f8275c53700 10 monclient(hunting): handle_auth_done global_id 4593 payload 824 2023-09-30T02:08:47.866 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.863+0000 7f8275c53700 1 -- 172.21.15.59:0/1638327888 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f82700a5f40 msgr2=0x7f827013c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:47.866 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.863+0000 7f8275c53700 1 --2- 172.21.15.59:0/1638327888 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f82700a5f40 0x7f827013c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:47.866 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.863+0000 7f8275c53700 1 -- 172.21.15.59:0/1638327888 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f82700a5b70 msgr2=0x7f8270140070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:47.867 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.863+0000 7f8275c53700 1 --2- 172.21.15.59:0/1638327888 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f82700a5b70 0x7f8270140070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:47.867 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.863+0000 7f8275c53700 10 monclient: _finish_hunting 0 2023-09-30T02:08:47.867 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.863+0000 7f8275c53700 1 monclient: found mon.noname-a 2023-09-30T02:08:47.867 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.863+0000 7f8275c53700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:08:47.867 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.863+0000 7f8275c53700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.59:3300/0 2023-09-30T02:08:47.867 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.863+0000 7f8275c53700 1 -- 172.21.15.59:0/1638327888 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8264009420 con 0x7f82701420c0 2023-09-30T02:08:47.867 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.866+0000 7f8275c53700 1 --2- 172.21.15.59:0/1638327888 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82701420c0 0x7f8270151840 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f826800cb90 tx=0x7f8268001980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:08:47.867 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.866+0000 7f82617fa700 10 client.?.objecter ms_handle_connect 0x7f82701420c0 2023-09-30T02:08:47.867 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.866+0000 7f82617fa700 10 client.?.objecter resend_mon_ops 2023-09-30T02:08:47.870 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.870+0000 7f82617fa700 1 -- 172.21.15.59:0/1638327888 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f826800a7e0 con 0x7f82701420c0 2023-09-30T02:08:47.870 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.870+0000 7f82617fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:08:47.870 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.870+0000 7f82617fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T02:08:47.870 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.870+0000 7f82617fa700 10 monclient: dump: 2023-09-30T02:08:47.870 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:08:47.871 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:47.871 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:47.871 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:47.871 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:08:47.871 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:47.871 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:08:47.871 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:08:47.871 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:08:47.871 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:47.871 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.870+0000 7f82617fa700 10 monclient: _finish_auth 0 2023-09-30T02:08:47.871 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.870+0000 7f82617fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:08:47.871 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.870+0000 7f82617fa700 1 -- 172.21.15.59:0/1638327888 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f826800a940 con 0x7f82701420c0 2023-09-30T02:08:47.872 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.870+0000 7f82617fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:08:47.872 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.870+0000 7f82617fa700 1 -- 172.21.15.59:0/1638327888 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f826800acc0 con 0x7f82701420c0 2023-09-30T02:08:47.872 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.870+0000 7f82617fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:08:47.872 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.870+0000 7f82617fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T02:08:47.872 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.870+0000 7f82617fa700 10 monclient: dump: 2023-09-30T02:08:47.872 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:08:47.872 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:47.872 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:47.872 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:47.872 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:08:47.872 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:47.872 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:08:47.872 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:08:47.873 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:08:47.873 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:47.873 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.870+0000 7f82776b6700 5 monclient: authenticate success, global_id 4593 2023-09-30T02:08:47.873 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.870+0000 7f82776b6700 10 monclient: _renew_subs 2023-09-30T02:08:47.873 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.870+0000 7f82776b6700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:08:47.873 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.870+0000 7f82637fe700 4 set_mon_vals no callback set 2023-09-30T02:08:47.873 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.870+0000 7f82776b6700 1 -- 172.21.15.59:0/1638327888 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8270144010 con 0x7f82701420c0 2023-09-30T02:08:47.873 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.870+0000 7f82776b6700 10 client.4593.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:08:47.873 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.870+0000 7f82776b6700 10 monclient: _renew_subs 2023-09-30T02:08:47.873 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.870+0000 7f82776b6700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:08:47.873 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.870+0000 7f82776b6700 1 -- 172.21.15.59:0/1638327888 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82701405b0 con 0x7f82701420c0 2023-09-30T02:08:47.873 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.870+0000 7f82776b6700 1 librados: init done 2023-09-30T02:08:47.874 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.871+0000 7f82776b6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:08:47.874 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.872+0000 7f82776b6700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:08:47.874 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.872+0000 7f82776b6700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:08:47.874 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:47.872+0000 7f82776b6700 1 -- 172.21.15.59:0/1638327888 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8270151560 con 0x7f82701420c0 2023-09-30T02:08:48.007 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:48.007+0000 7f82617fa700 1 -- 172.21.15.59:0/1638327888 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f8268007300 con 0x7f82701420c0 2023-09-30T02:08:48.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:48.008+0000 7f82617fa700 1 --2- 172.21.15.59:0/1638327888 >> v2:172.21.15.110:6800/115870 conn(0x7f82540689e0 0x7f825406ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:48.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:48.008+0000 7f82617fa700 1 -- 172.21.15.59:0/1638327888 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f8268083d70 con 0x7f82701420c0 2023-09-30T02:08:48.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:48.008+0000 7f82617fa700 10 client.4593.objecter ms_dispatch 0x7f8270001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:08:48.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:48.008+0000 7f82617fa700 3 client.4593.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:08:48.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:48.008+0000 7f82617fa700 3 client.4593.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:08:48.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:48.008+0000 7f82617fa700 20 client.4593.objecter dump_active .. 0 homeless 2023-09-30T02:08:48.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:48.008+0000 7f8275452700 1 --2- 172.21.15.59:0/1638327888 >> v2:172.21.15.110:6800/115870 conn(0x7f82540689e0 0x7f825406ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:48.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:48.008+0000 7f8275452700 10 monclient: get_auth_request con 0x7f82540689e0 auth_method 0 2023-09-30T02:08:48.009 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:48.009+0000 7f82617fa700 10 client.4593.objecter ms_handle_connect 0x7f82540689e0 2023-09-30T02:08:48.009 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:48.009+0000 7f8275452700 1 --2- 172.21.15.59:0/1638327888 >> v2:172.21.15.110:6800/115870 conn(0x7f82540689e0 0x7f825406ae90 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f826c009e70 tx=0x7f826c0091a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:08:48.174 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:48.174+0000 7f82617fa700 1 -- 172.21.15.59:0/1638327888 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8268051740 con 0x7f82701420c0 2023-09-30T02:08:48.174 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:48.174+0000 7f82617fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:08:48.174 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:48.174+0000 7f82617fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:08:48.441 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:48.442+0000 7f82776b6700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:08:48.442 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:48.442+0000 7f82776b6700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:08:48.442 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:48.442+0000 7f82776b6700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:08:48.442 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:48.442+0000 7f82776b6700 1 -- 172.21.15.59:0/1638327888 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f8270151560 con 0x7f82701420c0 2023-09-30T02:08:48.450 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:48.450+0000 7f82617fa700 1 -- 172.21.15.59:0/1638327888 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f8268021100 con 0x7f82701420c0 2023-09-30T02:08:48.450 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:48.450+0000 7f82617fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:08:48.450 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:48.450+0000 7f82617fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:08:48.450 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:08:48.450 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:08:48.453 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:48.453+0000 7f82776b6700 10 librados: watch_flush enter 2023-09-30T02:08:48.453 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:48.453+0000 7f82776b6700 10 librados: watch_flush exit 2023-09-30T02:08:48.453 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:48.453+0000 7f82776b6700 20 client.4593.objecter shutdown clearing up homeless session... 2023-09-30T02:08:48.453 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:48.453+0000 7f82776b6700 10 client.4593.objecter successfully canceled tick 2023-09-30T02:08:48.454 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:48.453+0000 7f82776b6700 1 -- 172.21.15.59:0/1638327888 >> v2:172.21.15.110:6800/115870 conn(0x7f82540689e0 msgr2=0x7f825406ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:48.454 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:48.453+0000 7f82776b6700 1 --2- 172.21.15.59:0/1638327888 >> v2:172.21.15.110:6800/115870 conn(0x7f82540689e0 0x7f825406ae90 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f826c009e70 tx=0x7f826c0091a0 comp rx=0 tx=0).stop 2023-09-30T02:08:48.454 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:48.453+0000 7f82776b6700 10 monclient: shutdown 2023-09-30T02:08:48.454 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:48.453+0000 7f82776b6700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:08:48.454 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:48.453+0000 7f82776b6700 1 -- 172.21.15.59:0/1638327888 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82701420c0 msgr2=0x7f8270151840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:48.454 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:48.453+0000 7f82776b6700 1 --2- 172.21.15.59:0/1638327888 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82701420c0 0x7f8270151840 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f826800cb90 tx=0x7f8268001980 comp rx=0 tx=0).stop 2023-09-30T02:08:48.454 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:48.453+0000 7f82776b6700 1 -- 172.21.15.59:0/1638327888 shutdown_connections 2023-09-30T02:08:48.454 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:48.453+0000 7f82776b6700 1 --2- 172.21.15.59:0/1638327888 >> v2:172.21.15.110:6800/115870 conn(0x7f82540689e0 0x7f825406ae90 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:48.454 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:48.453+0000 7f82776b6700 1 --2- 172.21.15.59:0/1638327888 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f82700a5b70 0x7f8270140070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:48.454 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:48.453+0000 7f82776b6700 1 --2- 172.21.15.59:0/1638327888 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82701420c0 0x7f8270151840 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:48.454 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:48.453+0000 7f82776b6700 1 --2- 172.21.15.59:0/1638327888 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f82700a5f40 0x7f827013c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:48.454 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:48.453+0000 7f82776b6700 1 -- 172.21.15.59:0/1638327888 >> 172.21.15.59:0/1638327888 conn(0x7f82700631f0 msgr2=0x7f82700af560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:08:48.455 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:48.453+0000 7f82776b6700 1 -- 172.21.15.59:0/1638327888 shutdown_connections 2023-09-30T02:08:48.455 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:48.454+0000 7f82776b6700 1 -- 172.21.15.59:0/1638327888 wait complete. 2023-09-30T02:08:48.455 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:48.454+0000 7f82776b6700 1 librados: shutdown 2023-09-30T02:08:51.468 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:08:51.607 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.606+0000 7fa59011c700 10 monclient: get_monmap_and_config 2023-09-30T02:08:51.608 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.606+0000 7fa59011c700 10 monclient: build_initial_monmap 2023-09-30T02:08:51.608 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.606+0000 7fa59011c700 10 monclient: monmap: 2023-09-30T02:08:51.608 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:08:51.608 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:51.608 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:08:51.606716+0000 2023-09-30T02:08:51.608 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:08:51.606716+0000 2023-09-30T02:08:51.608 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:08:51.609 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:51.609 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:08:51.609 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:08:51.609 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:08:51.609 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:08:51.609 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:08:51.609 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:51.609 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.606+0000 7fa59011c700 1 Processor -- start 2023-09-30T02:08:51.610 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.607+0000 7fa59011c700 1 -- start start 2023-09-30T02:08:51.610 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.607+0000 7fa59011c700 10 monclient: init 2023-09-30T02:08:51.610 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.607+0000 7fa59011c700 10 monclient: _reopen_session rank -1 2023-09-30T02:08:51.610 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.607+0000 7fa59011c700 10 monclient: _add_conns ranks=[0,3,4,2,1] 2023-09-30T02:08:51.610 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.607+0000 7fa59011c700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa5880a5b70 addr v1:172.21.15.59:6789/0 2023-09-30T02:08:51.610 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.607+0000 7fa59011c700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa588142290 0x7fa5880b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:51.610 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.607+0000 7fa59011c700 10 monclient(hunting): picked mon.noname-a con 0x7fa588142290 addr v2:172.21.15.59:3300/0 2023-09-30T02:08:51.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.607+0000 7fa59011c700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fa5880a5f40 0x7fa58813af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:51.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.607+0000 7fa59011c700 10 monclient(hunting): picked mon.noname-c con 0x7fa5880a5f40 addr v2:172.21.15.110:3300/0 2023-09-30T02:08:51.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.607+0000 7fa59011c700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5880a75a0 con 0x7fa5880a5b70 2023-09-30T02:08:51.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.607+0000 7fa59011c700 10 monclient(hunting): start opening mon connection 2023-09-30T02:08:51.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.607+0000 7fa59011c700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5880a6360 con 0x7fa588142290 2023-09-30T02:08:51.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.607+0000 7fa59011c700 10 monclient(hunting): start opening mon connection 2023-09-30T02:08:51.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.607+0000 7fa59011c700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5880af3f0 con 0x7fa5880a5f40 2023-09-30T02:08:51.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.607+0000 7fa59011c700 10 monclient(hunting): _renew_subs 2023-09-30T02:08:51.612 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.607+0000 7fa59011c700 10 monclient(hunting): authenticate will time out at 1976.909913s 2023-09-30T02:08:51.612 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.608+0000 7fa58d6b7700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa588142290 0x7fa5880b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:51.612 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.608+0000 7fa58e6b9700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fa5880a5f40 0x7fa58813af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:51.613 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.609+0000 7fa58deb8700 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fa5880a5b70 0x7fa588141b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:53408/0 (socket says 172.21.15.59:53408) 2023-09-30T02:08:51.613 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.609+0000 7fa58deb8700 1 -- 172.21.15.59:0/661777210 learned_addr learned my addr 172.21.15.59:0/661777210 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:08:51.613 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.609+0000 7fa58e6b9700 10 monclient(hunting): get_auth_request con 0x7fa5880a5f40 auth_method 0 2023-09-30T02:08:51.613 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.609+0000 7fa58e6b9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:08:51.614 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.609+0000 7fa58e6b9700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:51.614 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.609+0000 7fa58e6b9700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:51.614 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.609+0000 7fa58d6b7700 10 monclient(hunting): get_auth_request con 0x7fa588142290 auth_method 0 2023-09-30T02:08:51.614 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.609+0000 7fa58d6b7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:08:51.614 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.609+0000 7fa58d6b7700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:51.614 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.609+0000 7fa58d6b7700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:51.614 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.609+0000 7fa58e6b9700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:08:51.615 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.609+0000 7fa58e6b9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:08:51.615 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.610+0000 7fa58e6b9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:08:51.615 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.611+0000 7fa58d6b7700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:08:51.615 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.611+0000 7fa58d6b7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:08:51.615 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.611+0000 7fa58d6b7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:08:51.615 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.612+0000 7fa58e6b9700 10 monclient(hunting): handle_auth_done global_id 4609 payload 274 2023-09-30T02:08:51.615 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.612+0000 7fa58e6b9700 1 -- 172.21.15.59:0/661777210 >> v2:172.21.15.59:3300/0 conn(0x7fa588142290 msgr2=0x7fa5880b1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:08:51.616 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.612+0000 7fa58e6b9700 1 --2- 172.21.15.59:0/661777210 >> v2:172.21.15.59:3300/0 conn(0x7fa588142290 0x7fa5880b1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:51.616 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.612+0000 7fa58e6b9700 1 -- 172.21.15.59:0/661777210 >> v1:172.21.15.59:6789/0 conn(0x7fa5880a5b70 legacy=0x7fa588141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:51.616 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.612+0000 7fa58e6b9700 10 monclient: _finish_hunting 0 2023-09-30T02:08:51.616 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.612+0000 7fa58e6b9700 1 monclient: found mon.noname-c 2023-09-30T02:08:51.616 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.612+0000 7fa58e6b9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:08:51.616 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.612+0000 7fa58e6b9700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3300/0 2023-09-30T02:08:51.617 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.612+0000 7fa58e6b9700 1 -- 172.21.15.59:0/661777210 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa58813b470 con 0x7fa5880a5f40 2023-09-30T02:08:51.617 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.614+0000 7fa58e6b9700 1 --2- 172.21.15.59:0/661777210 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5880a5f40 0x7fa58813af30 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fa5880b0640 tx=0x7fa580009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4b79fc25bd4b675b server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:08:51.622 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.621+0000 7fa58ceb6700 1 -- 172.21.15.59:0/661777210 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa58000e070 con 0x7fa5880a5f40 2023-09-30T02:08:51.622 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.621+0000 7fa58ceb6700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:08:51.622 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.621+0000 7fa58ceb6700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:08:51.622 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.621+0000 7fa58ceb6700 10 monclient: dump: 2023-09-30T02:08:51.622 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:08:51.622 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:51.622 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:51.623 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:51.623 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:08:51.623 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:51.623 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:08:51.623 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:08:51.623 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:08:51.623 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:51.623 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.621+0000 7fa58ceb6700 10 monclient: _finish_auth 0 2023-09-30T02:08:51.623 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.621+0000 7fa58ceb6700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:08:51.623 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.621+0000 7fa58ceb6700 1 -- 172.21.15.59:0/661777210 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa58000bb60 con 0x7fa5880a5f40 2023-09-30T02:08:51.623 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.621+0000 7fa58ceb6700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:08:51.623 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.621+0000 7fa58ceb6700 1 -- 172.21.15.59:0/661777210 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa580012430 con 0x7fa5880a5f40 2023-09-30T02:08:51.624 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.621+0000 7fa58ceb6700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:08:51.624 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.621+0000 7fa58ceb6700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:08:51.624 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.621+0000 7fa58ceb6700 10 monclient: dump: 2023-09-30T02:08:51.624 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:08:51.624 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:51.624 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:51.624 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:51.624 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:08:51.624 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:51.624 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:08:51.624 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:08:51.625 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:08:51.625 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:51.625 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.622+0000 7fa59011c700 5 monclient: authenticate success, global_id 4609 2023-09-30T02:08:51.625 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.622+0000 7fa59011c700 10 monclient: get_monmap_and_config success 2023-09-30T02:08:51.625 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.622+0000 7fa59011c700 4 set_mon_vals no callback set 2023-09-30T02:08:51.625 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.622+0000 7fa59011c700 10 monclient: shutdown 2023-09-30T02:08:51.625 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.622+0000 7fa59011c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:08:51.625 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.622+0000 7fa59011c700 1 -- 172.21.15.59:0/661777210 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5880a5f40 msgr2=0x7fa58813af30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:51.625 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.622+0000 7fa59011c700 1 --2- 172.21.15.59:0/661777210 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5880a5f40 0x7fa58813af30 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fa5880b0640 tx=0x7fa580009780 comp rx=0 tx=0).stop 2023-09-30T02:08:51.625 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.622+0000 7fa59011c700 1 -- 172.21.15.59:0/661777210 shutdown_connections 2023-09-30T02:08:51.625 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.622+0000 7fa59011c700 1 --2- 172.21.15.59:0/661777210 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5880a5f40 0x7fa58813af30 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:51.625 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.622+0000 7fa59011c700 1 --2- 172.21.15.59:0/661777210 >> v2:172.21.15.59:3300/0 conn(0x7fa588142290 0x7fa5880b1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:51.626 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.622+0000 7fa59011c700 1 -- 172.21.15.59:0/661777210 >> 172.21.15.59:0/661777210 conn(0x7fa5880631f0 msgr2=0x7fa5880635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:08:51.626 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.622+0000 7fa59011c700 1 -- 172.21.15.59:0/661777210 shutdown_connections 2023-09-30T02:08:51.626 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.622+0000 7fa59011c700 1 -- 172.21.15.59:0/661777210 wait complete. 2023-09-30T02:08:51.628 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.623+0000 7fa59011c700 10 monclient: build_initial_monmap 2023-09-30T02:08:51.628 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.623+0000 7fa59011c700 10 monclient: monmap: 2023-09-30T02:08:51.628 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:08:51.628 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:08:51.628 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:08:51.628 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:08:51.628 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:08:51.628 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:08:51.628 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:08:51.629 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:08:51.629 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:08:51.629 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:51.629 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.624+0000 7fa59011c700 1 librados: starting msgr at 2023-09-30T02:08:51.629 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.624+0000 7fa59011c700 1 librados: starting objecter 2023-09-30T02:08:51.629 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.624+0000 7fa59011c700 1 Processor -- start 2023-09-30T02:08:51.629 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.624+0000 7fa59011c700 1 -- start start 2023-09-30T02:08:51.629 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.624+0000 7fa59011c700 1 librados: setting wanted keys 2023-09-30T02:08:51.629 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.624+0000 7fa59011c700 1 librados: calling monclient init 2023-09-30T02:08:51.629 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.624+0000 7fa59011c700 10 monclient: init 2023-09-30T02:08:51.629 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.625+0000 7fa59011c700 10 monclient: _reopen_session rank -1 2023-09-30T02:08:51.629 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.625+0000 7fa59011c700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T02:08:51.629 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.625+0000 7fa59011c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5880a5b70 0x7fa58813df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:51.630 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.625+0000 7fa59011c700 10 monclient(hunting): picked mon.noname-b con 0x7fa5880a5b70 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:08:51.630 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.625+0000 7fa59011c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5880a5f40 0x7fa58814df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:51.630 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.625+0000 7fa59011c700 10 monclient(hunting): picked mon.noname-a con 0x7fa5880a5f40 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:08:51.630 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.625+0000 7fa59011c700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa588142290 0x7fa58814e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:51.630 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.625+0000 7fa59011c700 10 monclient(hunting): picked mon.noname-c con 0x7fa588142290 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:08:51.630 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.625+0000 7fa59011c700 10 monclient(hunting): start opening mon connection 2023-09-30T02:08:51.630 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.625+0000 7fa59011c700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa58813d1c0 con 0x7fa5880a5f40 2023-09-30T02:08:51.630 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.625+0000 7fa59011c700 10 monclient(hunting): start opening mon connection 2023-09-30T02:08:51.630 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.625+0000 7fa59011c700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5880a7dd0 con 0x7fa5880a5b70 2023-09-30T02:08:51.630 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.625+0000 7fa59011c700 10 monclient(hunting): start opening mon connection 2023-09-30T02:08:51.630 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.625+0000 7fa59011c700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa58814da00 con 0x7fa588142290 2023-09-30T02:08:51.630 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.625+0000 7fa59011c700 10 monclient(hunting): _renew_subs 2023-09-30T02:08:51.631 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.625+0000 7fa59011c700 10 monclient(hunting): authenticate will time out at 1976.927343s 2023-09-30T02:08:51.631 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.625+0000 7fa58deb8700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5880a5b70 0x7fa58813df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:51.631 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.626+0000 7fa58deb8700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5880a5b70 0x7fa58813df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:53038/0 (socket says 172.21.15.59:53038) 2023-09-30T02:08:51.631 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.626+0000 7fa58deb8700 1 -- 172.21.15.59:0/2770676494 learned_addr learned my addr 172.21.15.59:0/2770676494 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:08:51.631 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.626+0000 7fa58deb8700 10 monclient(hunting): get_auth_request con 0x7fa5880a5b70 auth_method 0 2023-09-30T02:08:51.631 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.626+0000 7fa58deb8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:08:51.631 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.626+0000 7fa58deb8700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:51.631 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.626+0000 7fa58deb8700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:51.631 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.626+0000 7fa58e6b9700 1 --2- 172.21.15.59:0/2770676494 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa588142290 0x7fa58814e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:51.631 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.626+0000 7fa58e6b9700 10 monclient(hunting): get_auth_request con 0x7fa588142290 auth_method 0 2023-09-30T02:08:51.631 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.626+0000 7fa58e6b9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:08:51.631 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.626+0000 7fa58e6b9700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:51.632 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.626+0000 7fa58e6b9700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:51.632 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.626+0000 7fa58deb8700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:08:51.632 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.626+0000 7fa58deb8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:08:51.632 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.626+0000 7fa58deb8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:08:51.632 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.627+0000 7fa58e6b9700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:08:51.632 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.627+0000 7fa58e6b9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:08:51.632 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.627+0000 7fa58e6b9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:08:51.633 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.630+0000 7fa58deb8700 10 monclient(hunting): handle_auth_done global_id 4612 payload 824 2023-09-30T02:08:51.633 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.630+0000 7fa58deb8700 1 -- 172.21.15.59:0/2770676494 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa588142290 msgr2=0x7fa58814e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:51.633 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.630+0000 7fa58deb8700 1 --2- 172.21.15.59:0/2770676494 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa588142290 0x7fa58814e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:51.633 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.630+0000 7fa58deb8700 1 -- 172.21.15.59:0/2770676494 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5880a5f40 msgr2=0x7fa58814df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:51.633 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.630+0000 7fa58deb8700 1 --2- 172.21.15.59:0/2770676494 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5880a5f40 0x7fa58814df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:51.633 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.630+0000 7fa58deb8700 10 monclient: _finish_hunting 0 2023-09-30T02:08:51.633 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.630+0000 7fa58deb8700 1 monclient: found mon.noname-b 2023-09-30T02:08:51.633 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.630+0000 7fa58deb8700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:08:51.634 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.630+0000 7fa58deb8700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:08:51.634 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.630+0000 7fa58deb8700 1 -- 172.21.15.59:0/2770676494 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa580009a20 con 0x7fa5880a5b70 2023-09-30T02:08:51.634 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.632+0000 7fa58deb8700 1 --2- 172.21.15.59:0/2770676494 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5880a5b70 0x7fa58813df70 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa58400c7c0 tx=0x7fa584007f70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:08:51.634 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.632+0000 7fa57dffb700 10 client.?.objecter ms_handle_connect 0x7fa5880a5b70 2023-09-30T02:08:51.634 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.632+0000 7fa57dffb700 10 client.?.objecter resend_mon_ops 2023-09-30T02:08:51.635 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.634+0000 7fa57dffb700 1 -- 172.21.15.59:0/2770676494 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa584005b30 con 0x7fa5880a5b70 2023-09-30T02:08:51.635 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.634+0000 7fa57dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:08:51.635 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.634+0000 7fa57dffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:08:51.635 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.634+0000 7fa57dffb700 10 monclient: dump: 2023-09-30T02:08:51.635 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:08:51.635 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:51.635 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:51.640 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:51.640 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:08:51.640 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:51.640 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:08:51.640 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:08:51.640 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:08:51.641 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:51.641 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.634+0000 7fa57dffb700 10 monclient: _finish_auth 0 2023-09-30T02:08:51.641 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.634+0000 7fa57dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:08:51.641 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.634+0000 7fa59011c700 5 monclient: authenticate success, global_id 4612 2023-09-30T02:08:51.641 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.634+0000 7fa59011c700 10 monclient: _renew_subs 2023-09-30T02:08:51.641 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.634+0000 7fa59011c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:08:51.641 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.634+0000 7fa59011c700 1 -- 172.21.15.59:0/2770676494 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa58813c5a0 con 0x7fa5880a5b70 2023-09-30T02:08:51.641 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.634+0000 7fa59011c700 10 client.4612.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:08:51.641 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.634+0000 7fa59011c700 10 monclient: _renew_subs 2023-09-30T02:08:51.641 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.634+0000 7fa59011c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:08:51.641 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.634+0000 7fa59011c700 1 -- 172.21.15.59:0/2770676494 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa588139830 con 0x7fa5880a5b70 2023-09-30T02:08:51.641 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.634+0000 7fa59011c700 1 librados: init done 2023-09-30T02:08:51.642 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.635+0000 7fa59011c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:08:51.642 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.635+0000 7fa59011c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:08:51.642 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.635+0000 7fa59011c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:08:51.642 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.635+0000 7fa59011c700 1 -- 172.21.15.59:0/2770676494 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa588064a00 con 0x7fa5880a5b70 2023-09-30T02:08:51.642 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.636+0000 7fa57dffb700 1 -- 172.21.15.59:0/2770676494 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa584004830 con 0x7fa5880a5b70 2023-09-30T02:08:51.643 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.636+0000 7fa57dffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:08:51.643 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.636+0000 7fa57dffb700 1 -- 172.21.15.59:0/2770676494 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa584005ea0 con 0x7fa5880a5b70 2023-09-30T02:08:51.643 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.636+0000 7fa57dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:08:51.643 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.636+0000 7fa57dffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:08:51.643 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.636+0000 7fa57dffb700 10 monclient: dump: 2023-09-30T02:08:51.643 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:08:51.643 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:51.643 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:51.643 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:51.643 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:08:51.643 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:51.643 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:08:51.643 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:08:51.644 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:08:51.644 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:51.644 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.636+0000 7fa57ffff700 4 set_mon_vals no callback set 2023-09-30T02:08:51.886 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.887+0000 7fa57dffb700 1 -- 172.21.15.59:0/2770676494 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fa5840049e0 con 0x7fa5880a5b70 2023-09-30T02:08:51.887 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.887+0000 7fa57dffb700 1 --2- 172.21.15.59:0/2770676494 >> v2:172.21.15.110:6800/115870 conn(0x7fa56c068890 0x7fa56c06ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:51.887 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.887+0000 7fa58d6b7700 1 --2- 172.21.15.59:0/2770676494 >> v2:172.21.15.110:6800/115870 conn(0x7fa56c068890 0x7fa56c06ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:51.887 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.887+0000 7fa58d6b7700 10 monclient: get_auth_request con 0x7fa56c068890 auth_method 0 2023-09-30T02:08:51.887 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.888+0000 7fa58d6b7700 1 --2- 172.21.15.59:0/2770676494 >> v2:172.21.15.110:6800/115870 conn(0x7fa56c068890 0x7fa56c06ad40 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa588141cf0 tx=0x7fa5780091a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:08:51.887 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.888+0000 7fa57dffb700 10 client.4612.objecter ms_handle_connect 0x7fa56c068890 2023-09-30T02:08:51.891 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.891+0000 7fa57dffb700 1 -- 172.21.15.59:0/2770676494 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fa584055950 con 0x7fa5880a5b70 2023-09-30T02:08:51.891 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.891+0000 7fa57dffb700 10 client.4612.objecter ms_dispatch 0x7fa588001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:08:51.891 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.891+0000 7fa57dffb700 3 client.4612.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:08:51.891 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.891+0000 7fa57dffb700 3 client.4612.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:08:51.891 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.892+0000 7fa57dffb700 20 client.4612.objecter dump_active .. 0 homeless 2023-09-30T02:08:51.900 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.901+0000 7fa57dffb700 1 -- 172.21.15.59:0/2770676494 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa584051590 con 0x7fa5880a5b70 2023-09-30T02:08:51.900 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.901+0000 7fa57dffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:08:51.900 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:51.901+0000 7fa57dffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:08:52.152 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:52.152+0000 7fa59011c700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:08:52.152 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:52.152+0000 7fa59011c700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:08:52.152 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:52.152+0000 7fa59011c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:08:52.152 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:52.152+0000 7fa59011c700 1 -- 172.21.15.59:0/2770676494 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fa58806a720 con 0x7fa5880a5b70 2023-09-30T02:08:52.157 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:52.158+0000 7fa57dffb700 1 -- 172.21.15.59:0/2770676494 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fa584021020 con 0x7fa5880a5b70 2023-09-30T02:08:52.157 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:52.158+0000 7fa57dffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:08:52.157 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:52.158+0000 7fa57dffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:08:52.158 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:08:52.158 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:08:52.160 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:52.160+0000 7fa59011c700 10 librados: watch_flush enter 2023-09-30T02:08:52.160 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:52.160+0000 7fa59011c700 10 librados: watch_flush exit 2023-09-30T02:08:52.160 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:52.160+0000 7fa59011c700 20 client.4612.objecter shutdown clearing up homeless session... 2023-09-30T02:08:52.160 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:52.160+0000 7fa59011c700 10 client.4612.objecter successfully canceled tick 2023-09-30T02:08:52.161 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:52.160+0000 7fa59011c700 1 -- 172.21.15.59:0/2770676494 >> v2:172.21.15.110:6800/115870 conn(0x7fa56c068890 msgr2=0x7fa56c06ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:52.161 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:52.160+0000 7fa59011c700 1 --2- 172.21.15.59:0/2770676494 >> v2:172.21.15.110:6800/115870 conn(0x7fa56c068890 0x7fa56c06ad40 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa588141cf0 tx=0x7fa5780091a0 comp rx=0 tx=0).stop 2023-09-30T02:08:52.161 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:52.160+0000 7fa59011c700 10 monclient: shutdown 2023-09-30T02:08:52.161 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:52.160+0000 7fa59011c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:08:52.161 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:52.160+0000 7fa59011c700 1 -- 172.21.15.59:0/2770676494 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5880a5b70 msgr2=0x7fa58813df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:52.161 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:52.160+0000 7fa59011c700 1 --2- 172.21.15.59:0/2770676494 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5880a5b70 0x7fa58813df70 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa58400c7c0 tx=0x7fa584007f70 comp rx=0 tx=0).stop 2023-09-30T02:08:52.161 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:52.160+0000 7fa59011c700 1 -- 172.21.15.59:0/2770676494 shutdown_connections 2023-09-30T02:08:52.161 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:52.160+0000 7fa59011c700 1 --2- 172.21.15.59:0/2770676494 >> v2:172.21.15.110:6800/115870 conn(0x7fa56c068890 0x7fa56c06ad40 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:52.161 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:52.160+0000 7fa59011c700 1 --2- 172.21.15.59:0/2770676494 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa5880a5b70 0x7fa58813df70 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:52.161 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:52.160+0000 7fa59011c700 1 --2- 172.21.15.59:0/2770676494 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa588142290 0x7fa58814e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:52.161 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:52.160+0000 7fa59011c700 1 --2- 172.21.15.59:0/2770676494 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa5880a5f40 0x7fa58814df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:52.162 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:52.160+0000 7fa59011c700 1 -- 172.21.15.59:0/2770676494 >> 172.21.15.59:0/2770676494 conn(0x7fa5880631f0 msgr2=0x7fa588144fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:08:52.162 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:52.161+0000 7fa59011c700 1 -- 172.21.15.59:0/2770676494 shutdown_connections 2023-09-30T02:08:52.162 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:52.161+0000 7fa59011c700 1 -- 172.21.15.59:0/2770676494 wait complete. 2023-09-30T02:08:52.162 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:52.161+0000 7fa59011c700 1 librados: shutdown 2023-09-30T02:08:53.880 INFO:tasks.ceph.mon.b.smithi110.stderr:==00:00:04:03.727 115869== Warning: unimplemented fcntl command: 1036 2023-09-30T02:08:53.882 INFO:tasks.ceph.mon.b.smithi110.stderr:==00:00:04:03.729 115869== Warning: unimplemented fcntl command: 1036 2023-09-30T02:08:53.889 INFO:tasks.ceph.mon.a.smithi059.stderr:==00:00:04:03.754 116256== Warning: unimplemented fcntl command: 1036 2023-09-30T02:08:53.892 INFO:tasks.ceph.mon.a.smithi059.stderr:==00:00:04:03.758 116256== Warning: unimplemented fcntl command: 1036 2023-09-30T02:08:53.902 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:04:03.767 116257== Warning: unimplemented fcntl command: 1036 2023-09-30T02:08:53.914 INFO:tasks.ceph.mon.b.smithi110.stderr:==00:00:04:03.761 115869== Warning: unimplemented fcntl command: 1036 2023-09-30T02:08:53.927 INFO:tasks.ceph.mon.a.smithi059.stderr:==00:00:04:03.793 116256== Warning: unimplemented fcntl command: 1036 2023-09-30T02:08:53.953 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:04:03.819 116257== Warning: unimplemented fcntl command: 1036 2023-09-30T02:08:53.979 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:04:03.844 116257== Warning: unimplemented fcntl command: 1036 2023-09-30T02:08:55.175 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:08:55.316 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.316+0000 7fd78dbf4700 10 monclient: get_monmap_and_config 2023-09-30T02:08:55.316 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.316+0000 7fd78dbf4700 10 monclient: build_initial_monmap 2023-09-30T02:08:55.316 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.316+0000 7fd78dbf4700 10 monclient: monmap: 2023-09-30T02:08:55.316 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:08:55.316 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:55.317 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:08:55.316606+0000 2023-09-30T02:08:55.317 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:08:55.316606+0000 2023-09-30T02:08:55.317 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:08:55.317 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:55.317 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:08:55.317 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:08:55.317 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:08:55.317 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:08:55.317 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:08:55.317 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:55.317 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.317+0000 7fd78dbf4700 1 Processor -- start 2023-09-30T02:08:55.317 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.317+0000 7fd78dbf4700 1 -- start start 2023-09-30T02:08:55.317 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.317+0000 7fd78dbf4700 10 monclient: init 2023-09-30T02:08:55.319 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.317+0000 7fd78dbf4700 10 monclient: _reopen_session rank -1 2023-09-30T02:08:55.320 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.317+0000 7fd78dbf4700 10 monclient: _add_conns ranks=[2,4,0,3,1] 2023-09-30T02:08:55.320 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.317+0000 7fd78dbf4700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd7880a5b70 0x7fd7880b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:55.320 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.317+0000 7fd78dbf4700 10 monclient(hunting): picked mon.noname-b con 0x7fd7880a5b70 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:08:55.320 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.317+0000 7fd78dbf4700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd7880a5f40 0x7fd788141b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:55.320 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.317+0000 7fd78dbf4700 10 monclient(hunting): picked mon.noname-c con 0x7fd7880a5f40 addr v2:172.21.15.110:3300/0 2023-09-30T02:08:55.320 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.317+0000 7fd78dbf4700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd7881420c0 addr v1:172.21.15.59:6789/0 2023-09-30T02:08:55.320 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.317+0000 7fd78dbf4700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7880a63b0 con 0x7fd7881420c0 2023-09-30T02:08:55.320 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.317+0000 7fd78dbf4700 10 monclient(hunting): start opening mon connection 2023-09-30T02:08:55.320 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.317+0000 7fd78dbf4700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7880a75a0 con 0x7fd7880a5f40 2023-09-30T02:08:55.320 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.317+0000 7fd78dbf4700 10 monclient(hunting): start opening mon connection 2023-09-30T02:08:55.320 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.317+0000 7fd78dbf4700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7880af3f0 con 0x7fd7880a5b70 2023-09-30T02:08:55.320 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.317+0000 7fd78dbf4700 10 monclient(hunting): _renew_subs 2023-09-30T02:08:55.321 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.317+0000 7fd78dbf4700 10 monclient(hunting): authenticate will time out at 1980.619997s 2023-09-30T02:08:55.321 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.318+0000 7fd786ffd700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fd7880a5f40 0x7fd788141b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:55.321 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.318+0000 7fd787fff700 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fd7881420c0 0x7fd78813e230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:53414/0 (socket says 172.21.15.59:53414) 2023-09-30T02:08:55.321 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.318+0000 7fd787fff700 1 -- 172.21.15.59:0/2033898007 learned_addr learned my addr 172.21.15.59:0/2033898007 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:08:55.321 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.318+0000 7fd7877fe700 1 --2- 172.21.15.59:0/2033898007 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd7880a5b70 0x7fd7880b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:55.321 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.318+0000 7fd786ffd700 10 monclient(hunting): get_auth_request con 0x7fd7880a5f40 auth_method 0 2023-09-30T02:08:55.321 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.318+0000 7fd786ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:08:55.321 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.318+0000 7fd786ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:55.321 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.318+0000 7fd786ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:55.321 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.319+0000 7fd7877fe700 10 monclient(hunting): get_auth_request con 0x7fd7880a5b70 auth_method 0 2023-09-30T02:08:55.321 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.319+0000 7fd7877fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:08:55.321 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.319+0000 7fd7877fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:55.322 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.319+0000 7fd7877fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:55.322 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.319+0000 7fd786ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:08:55.322 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.319+0000 7fd786ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:08:55.322 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.319+0000 7fd786ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:08:55.322 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.319+0000 7fd7877fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:08:55.322 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.319+0000 7fd7877fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:08:55.322 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.319+0000 7fd7877fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:08:55.323 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.322+0000 7fd7867fc700 1 -- 172.21.15.59:0/2033898007 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 289306640 0 0) 0x7fd7880a63b0 con 0x7fd7881420c0 2023-09-30T02:08:55.323 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.322+0000 7fd7867fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:55.323 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.322+0000 7fd7867fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:55.323 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.322+0000 7fd7867fc700 10 monclient(hunting): my global_id is 4602 2023-09-30T02:08:55.323 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.322+0000 7fd7867fc700 1 -- 172.21.15.59:0/2033898007 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd774003570 con 0x7fd7881420c0 2023-09-30T02:08:55.323 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.322+0000 7fd786ffd700 10 monclient(hunting): handle_auth_done global_id 4615 payload 274 2023-09-30T02:08:55.323 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.322+0000 7fd786ffd700 1 -- 172.21.15.59:0/2033898007 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd7880a5b70 msgr2=0x7fd7880b1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:08:55.324 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.322+0000 7fd786ffd700 1 --2- 172.21.15.59:0/2033898007 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd7880a5b70 0x7fd7880b1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:55.324 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.322+0000 7fd786ffd700 1 -- 172.21.15.59:0/2033898007 >> v1:172.21.15.59:6789/0 conn(0x7fd7881420c0 legacy=0x7fd78813e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:55.324 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.322+0000 7fd786ffd700 10 monclient: _finish_hunting 0 2023-09-30T02:08:55.324 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.322+0000 7fd786ffd700 1 monclient: found mon.noname-c 2023-09-30T02:08:55.324 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.322+0000 7fd786ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:08:55.324 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.322+0000 7fd786ffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3300/0 2023-09-30T02:08:55.324 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.322+0000 7fd786ffd700 1 -- 172.21.15.59:0/2033898007 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd788142500 con 0x7fd7880a5f40 2023-09-30T02:08:55.324 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.324+0000 7fd786ffd700 1 --2- 172.21.15.59:0/2033898007 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd7880a5f40 0x7fd788141b80 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd7700098e0 tx=0x7fd7700091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9b9e9700231f52d1 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:08:55.328 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.328+0000 7fd7867fc700 1 -- 172.21.15.59:0/2033898007 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd770011630 con 0x7fd7880a5f40 2023-09-30T02:08:55.328 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.328+0000 7fd7867fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:08:55.328 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.328+0000 7fd7867fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:08:55.328 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.328+0000 7fd7867fc700 10 monclient: dump: 2023-09-30T02:08:55.328 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:08:55.328 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:55.328 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:55.329 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:55.329 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:08:55.329 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:55.329 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:08:55.329 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:08:55.329 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:08:55.329 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:55.329 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.328+0000 7fd7867fc700 10 monclient: _finish_auth 0 2023-09-30T02:08:55.329 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.328+0000 7fd7867fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:08:55.329 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.328+0000 7fd7867fc700 1 -- 172.21.15.59:0/2033898007 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd770011790 con 0x7fd7880a5f40 2023-09-30T02:08:55.329 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.328+0000 7fd7867fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:08:55.329 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.328+0000 7fd7867fc700 1 -- 172.21.15.59:0/2033898007 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd770011b30 con 0x7fd7880a5f40 2023-09-30T02:08:55.329 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.328+0000 7fd7867fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:08:55.330 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.328+0000 7fd7867fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:08:55.330 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.328+0000 7fd7867fc700 10 monclient: dump: 2023-09-30T02:08:55.330 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:08:55.330 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:55.330 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:55.330 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:55.330 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:08:55.330 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:55.330 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:08:55.330 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:08:55.330 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:08:55.330 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:55.331 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.329+0000 7fd78dbf4700 5 monclient: authenticate success, global_id 4615 2023-09-30T02:08:55.331 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.329+0000 7fd78dbf4700 10 monclient: get_monmap_and_config success 2023-09-30T02:08:55.331 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.329+0000 7fd78dbf4700 4 set_mon_vals no callback set 2023-09-30T02:08:55.334 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.329+0000 7fd78dbf4700 10 monclient: shutdown 2023-09-30T02:08:55.334 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.329+0000 7fd78dbf4700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:08:55.334 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.329+0000 7fd78dbf4700 1 -- 172.21.15.59:0/2033898007 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd7880a5f40 msgr2=0x7fd788141b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:55.334 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.329+0000 7fd78dbf4700 1 --2- 172.21.15.59:0/2033898007 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd7880a5f40 0x7fd788141b80 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd7700098e0 tx=0x7fd7700091a0 comp rx=0 tx=0).stop 2023-09-30T02:08:55.334 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.329+0000 7fd78dbf4700 1 -- 172.21.15.59:0/2033898007 shutdown_connections 2023-09-30T02:08:55.334 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.329+0000 7fd78dbf4700 1 --2- 172.21.15.59:0/2033898007 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd7880a5b70 0x7fd7880b1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:55.334 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.329+0000 7fd78dbf4700 1 --2- 172.21.15.59:0/2033898007 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd7880a5f40 0x7fd788141b80 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:55.334 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.329+0000 7fd78dbf4700 1 -- 172.21.15.59:0/2033898007 >> 172.21.15.59:0/2033898007 conn(0x7fd7880631f0 msgr2=0x7fd7880635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:08:55.334 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.329+0000 7fd78dbf4700 1 -- 172.21.15.59:0/2033898007 shutdown_connections 2023-09-30T02:08:55.334 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.329+0000 7fd78dbf4700 1 -- 172.21.15.59:0/2033898007 wait complete. 2023-09-30T02:08:55.335 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.330+0000 7fd78dbf4700 10 monclient: build_initial_monmap 2023-09-30T02:08:55.335 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.330+0000 7fd78dbf4700 10 monclient: monmap: 2023-09-30T02:08:55.335 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:08:55.335 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:08:55.335 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:08:55.335 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:08:55.335 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:08:55.335 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:08:55.335 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:08:55.335 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:08:55.335 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:08:55.335 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:55.335 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.330+0000 7fd78dbf4700 1 librados: starting msgr at 2023-09-30T02:08:55.336 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.330+0000 7fd78dbf4700 1 librados: starting objecter 2023-09-30T02:08:55.336 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.331+0000 7fd78dbf4700 1 Processor -- start 2023-09-30T02:08:55.336 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.331+0000 7fd78dbf4700 1 -- start start 2023-09-30T02:08:55.336 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.331+0000 7fd78dbf4700 1 librados: setting wanted keys 2023-09-30T02:08:55.336 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.331+0000 7fd78dbf4700 1 librados: calling monclient init 2023-09-30T02:08:55.336 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.331+0000 7fd78dbf4700 10 monclient: init 2023-09-30T02:08:55.336 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.331+0000 7fd78dbf4700 10 monclient: _reopen_session rank -1 2023-09-30T02:08:55.336 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.331+0000 7fd78dbf4700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T02:08:55.336 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.331+0000 7fd78dbf4700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd7880a5b70 0x7fd788151430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:55.336 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.331+0000 7fd78dbf4700 10 monclient(hunting): picked mon.noname-b con 0x7fd7880a5b70 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:08:55.336 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.331+0000 7fd78dbf4700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd7880a5f40 0x7fd788151970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:55.336 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.331+0000 7fd78dbf4700 10 monclient(hunting): picked mon.noname-c con 0x7fd7880a5f40 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:08:55.337 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.331+0000 7fd78dbf4700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd7881420c0 0x7fd788137880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:55.337 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.331+0000 7fd78dbf4700 10 monclient(hunting): picked mon.noname-a con 0x7fd7881420c0 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:08:55.337 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.331+0000 7fd78dbf4700 10 monclient(hunting): start opening mon connection 2023-09-30T02:08:55.337 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.331+0000 7fd78dbf4700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7880b56c0 con 0x7fd7881420c0 2023-09-30T02:08:55.337 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.331+0000 7fd78dbf4700 10 monclient(hunting): start opening mon connection 2023-09-30T02:08:55.337 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.331+0000 7fd78dbf4700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7880a9540 con 0x7fd7880a5b70 2023-09-30T02:08:55.337 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.331+0000 7fd78dbf4700 10 monclient(hunting): start opening mon connection 2023-09-30T02:08:55.337 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.331+0000 7fd78dbf4700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7880a9820 con 0x7fd7880a5f40 2023-09-30T02:08:55.338 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.331+0000 7fd78dbf4700 10 monclient(hunting): _renew_subs 2023-09-30T02:08:55.338 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.331+0000 7fd78dbf4700 10 monclient(hunting): authenticate will time out at 1980.633986s 2023-09-30T02:08:55.338 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.332+0000 7fd786ffd700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd7880a5f40 0x7fd788151970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:55.338 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.332+0000 7fd7877fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd7880a5b70 0x7fd788151430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:55.338 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.332+0000 7fd787fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd7881420c0 0x7fd788137880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:55.338 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.332+0000 7fd7877fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd7880a5b70 0x7fd788151430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:53052/0 (socket says 172.21.15.59:53052) 2023-09-30T02:08:55.338 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.332+0000 7fd7877fe700 1 -- 172.21.15.59:0/2963227435 learned_addr learned my addr 172.21.15.59:0/2963227435 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:08:55.338 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.332+0000 7fd7877fe700 10 monclient(hunting): get_auth_request con 0x7fd7880a5b70 auth_method 0 2023-09-30T02:08:55.338 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.332+0000 7fd7877fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:08:55.338 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.332+0000 7fd7877fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:55.338 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.332+0000 7fd7877fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:55.339 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.333+0000 7fd786ffd700 10 monclient(hunting): get_auth_request con 0x7fd7880a5f40 auth_method 0 2023-09-30T02:08:55.339 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.333+0000 7fd786ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:08:55.339 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.333+0000 7fd786ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:55.339 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.333+0000 7fd786ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:55.339 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.333+0000 7fd787fff700 10 monclient(hunting): get_auth_request con 0x7fd7881420c0 auth_method 0 2023-09-30T02:08:55.339 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.333+0000 7fd787fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:08:55.339 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.333+0000 7fd787fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:55.339 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.333+0000 7fd787fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:55.339 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.333+0000 7fd7877fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:08:55.339 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.333+0000 7fd7877fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:08:55.340 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.333+0000 7fd7877fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:08:55.340 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.333+0000 7fd786ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:08:55.340 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.333+0000 7fd786ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:08:55.340 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.333+0000 7fd786ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:08:55.340 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.334+0000 7fd787fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:08:55.340 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.334+0000 7fd787fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:08:55.340 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.334+0000 7fd787fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:08:55.341 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.336+0000 7fd7877fe700 10 monclient(hunting): handle_auth_done global_id 4618 payload 824 2023-09-30T02:08:55.341 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.336+0000 7fd7877fe700 1 -- 172.21.15.59:0/2963227435 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd7880a5f40 msgr2=0x7fd788151970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:55.341 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.336+0000 7fd7877fe700 1 --2- 172.21.15.59:0/2963227435 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd7880a5f40 0x7fd788151970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:55.341 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.336+0000 7fd7877fe700 1 -- 172.21.15.59:0/2963227435 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd7881420c0 msgr2=0x7fd788137880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:55.341 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.336+0000 7fd7877fe700 1 --2- 172.21.15.59:0/2963227435 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd7881420c0 0x7fd788137880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:55.341 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.336+0000 7fd7877fe700 10 monclient: _finish_hunting 0 2023-09-30T02:08:55.341 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.336+0000 7fd7877fe700 1 monclient: found mon.noname-b 2023-09-30T02:08:55.341 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.336+0000 7fd7877fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:08:55.342 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.336+0000 7fd7877fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:08:55.342 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.336+0000 7fd7877fe700 1 -- 172.21.15.59:0/2963227435 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd770009420 con 0x7fd7880a5b70 2023-09-30T02:08:55.342 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.339+0000 7fd7877fe700 1 --2- 172.21.15.59:0/2963227435 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd7880a5b70 0x7fd788151430 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd778009ef0 tx=0x7fd77800b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:08:55.342 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.339+0000 7fd77f7fe700 10 client.?.objecter ms_handle_connect 0x7fd7880a5b70 2023-09-30T02:08:55.342 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.339+0000 7fd77f7fe700 10 client.?.objecter resend_mon_ops 2023-09-30T02:08:55.343 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.342+0000 7fd77f7fe700 1 -- 172.21.15.59:0/2963227435 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd77800cb70 con 0x7fd7880a5b70 2023-09-30T02:08:55.343 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.342+0000 7fd77f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:08:55.343 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.342+0000 7fd77f7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:08:55.343 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.342+0000 7fd77f7fe700 10 monclient: dump: 2023-09-30T02:08:55.343 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:08:55.344 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:55.344 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:55.344 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:55.344 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:08:55.344 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:55.344 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:08:55.344 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:08:55.344 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:08:55.344 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:55.344 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.342+0000 7fd77f7fe700 10 monclient: _finish_auth 0 2023-09-30T02:08:55.344 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.342+0000 7fd77f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:08:55.344 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.342+0000 7fd77f7fe700 1 -- 172.21.15.59:0/2963227435 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd77800f040 con 0x7fd7880a5b70 2023-09-30T02:08:55.345 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.342+0000 7fd78dbf4700 5 monclient: authenticate success, global_id 4618 2023-09-30T02:08:55.345 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.342+0000 7fd78dbf4700 10 monclient: _renew_subs 2023-09-30T02:08:55.345 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.342+0000 7fd78dbf4700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:08:55.345 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.342+0000 7fd78dbf4700 1 -- 172.21.15.59:0/2963227435 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7880b48c0 con 0x7fd7880a5b70 2023-09-30T02:08:55.345 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.342+0000 7fd77f7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:08:55.345 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.342+0000 7fd77f7fe700 1 -- 172.21.15.59:0/2963227435 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd778014480 con 0x7fd7880a5b70 2023-09-30T02:08:55.345 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.342+0000 7fd77f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:08:55.345 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.342+0000 7fd77f7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:08:55.345 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.342+0000 7fd77f7fe700 10 monclient: dump: 2023-09-30T02:08:55.345 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:08:55.345 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:55.345 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:55.346 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:55.346 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:08:55.346 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:55.346 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:08:55.346 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:08:55.346 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:08:55.346 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:55.346 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.342+0000 7fd785ffb700 4 set_mon_vals no callback set 2023-09-30T02:08:55.346 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.342+0000 7fd78dbf4700 10 client.4618.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:08:55.346 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.342+0000 7fd78dbf4700 10 monclient: _renew_subs 2023-09-30T02:08:55.346 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.342+0000 7fd78dbf4700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:08:55.346 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.342+0000 7fd78dbf4700 1 -- 172.21.15.59:0/2963227435 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd788152070 con 0x7fd7880a5b70 2023-09-30T02:08:55.347 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.342+0000 7fd78dbf4700 1 librados: init done 2023-09-30T02:08:55.347 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.343+0000 7fd78dbf4700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:08:55.347 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.343+0000 7fd78dbf4700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:08:55.347 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.343+0000 7fd78dbf4700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:08:55.347 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.343+0000 7fd78dbf4700 1 -- 172.21.15.59:0/2963227435 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7880b59a0 con 0x7fd7880a5b70 2023-09-30T02:08:55.584 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.584+0000 7fd77f7fe700 1 -- 172.21.15.59:0/2963227435 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fd7780147f0 con 0x7fd7880a5b70 2023-09-30T02:08:55.584 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.584+0000 7fd77f7fe700 1 --2- 172.21.15.59:0/2963227435 >> v2:172.21.15.110:6800/115870 conn(0x7fd7680689e0 0x7fd76806ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:55.584 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.585+0000 7fd77f7fe700 1 -- 172.21.15.59:0/2963227435 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fd778084d90 con 0x7fd7880a5b70 2023-09-30T02:08:55.585 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.585+0000 7fd77f7fe700 10 client.4618.objecter ms_dispatch 0x7fd788001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:08:55.585 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.585+0000 7fd77f7fe700 3 client.4618.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:08:55.585 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.585+0000 7fd77f7fe700 3 client.4618.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:08:55.585 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.585+0000 7fd786ffd700 1 --2- 172.21.15.59:0/2963227435 >> v2:172.21.15.110:6800/115870 conn(0x7fd7680689e0 0x7fd76806ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:55.585 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.585+0000 7fd77f7fe700 20 client.4618.objecter dump_active .. 0 homeless 2023-09-30T02:08:55.585 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.585+0000 7fd786ffd700 10 monclient: get_auth_request con 0x7fd7680689e0 auth_method 0 2023-09-30T02:08:55.585 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.585+0000 7fd786ffd700 1 --2- 172.21.15.59:0/2963227435 >> v2:172.21.15.110:6800/115870 conn(0x7fd7680689e0 0x7fd76806ae90 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd7700091e0 tx=0x7fd770017000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:08:55.585 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.585+0000 7fd77f7fe700 10 client.4618.objecter ms_handle_connect 0x7fd7680689e0 2023-09-30T02:08:55.609 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.610+0000 7fd77f7fe700 1 -- 172.21.15.59:0/2963227435 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd7780526b0 con 0x7fd7880a5b70 2023-09-30T02:08:55.609 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.610+0000 7fd77f7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:08:55.610 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.610+0000 7fd77f7fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:08:55.871 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.871+0000 7fd78dbf4700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:08:55.871 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.871+0000 7fd78dbf4700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:08:55.871 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.871+0000 7fd78dbf4700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:08:55.871 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.871+0000 7fd78dbf4700 1 -- 172.21.15.59:0/2963227435 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd788152070 con 0x7fd7880a5b70 2023-09-30T02:08:55.877 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.878+0000 7fd77f7fe700 1 -- 172.21.15.59:0/2963227435 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fd788152070 con 0x7fd7880a5b70 2023-09-30T02:08:55.877 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.878+0000 7fd77f7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:08:55.877 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.878+0000 7fd77f7fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:08:55.878 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:08:55.878 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:08:55.880 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.880+0000 7fd78dbf4700 10 librados: watch_flush enter 2023-09-30T02:08:55.880 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.880+0000 7fd78dbf4700 10 librados: watch_flush exit 2023-09-30T02:08:55.880 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.880+0000 7fd78dbf4700 20 client.4618.objecter shutdown clearing up homeless session... 2023-09-30T02:08:55.881 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.880+0000 7fd78dbf4700 10 client.4618.objecter successfully canceled tick 2023-09-30T02:08:55.881 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.880+0000 7fd78dbf4700 1 -- 172.21.15.59:0/2963227435 >> v2:172.21.15.110:6800/115870 conn(0x7fd7680689e0 msgr2=0x7fd76806ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:55.881 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.880+0000 7fd78dbf4700 1 --2- 172.21.15.59:0/2963227435 >> v2:172.21.15.110:6800/115870 conn(0x7fd7680689e0 0x7fd76806ae90 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd7700091e0 tx=0x7fd770017000 comp rx=0 tx=0).stop 2023-09-30T02:08:55.881 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.880+0000 7fd78dbf4700 10 monclient: shutdown 2023-09-30T02:08:55.881 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.880+0000 7fd78dbf4700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:08:55.881 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.880+0000 7fd78dbf4700 1 -- 172.21.15.59:0/2963227435 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd7880a5b70 msgr2=0x7fd788151430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:55.881 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.880+0000 7fd78dbf4700 1 --2- 172.21.15.59:0/2963227435 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd7880a5b70 0x7fd788151430 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fd778009ef0 tx=0x7fd77800b040 comp rx=0 tx=0).stop 2023-09-30T02:08:55.881 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.880+0000 7fd78dbf4700 1 -- 172.21.15.59:0/2963227435 shutdown_connections 2023-09-30T02:08:55.881 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.880+0000 7fd78dbf4700 1 --2- 172.21.15.59:0/2963227435 >> v2:172.21.15.110:6800/115870 conn(0x7fd7680689e0 0x7fd76806ae90 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:55.881 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.880+0000 7fd78dbf4700 1 --2- 172.21.15.59:0/2963227435 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fd7880a5b70 0x7fd788151430 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:55.881 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.880+0000 7fd78dbf4700 1 --2- 172.21.15.59:0/2963227435 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd7881420c0 0x7fd788137880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:55.882 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.880+0000 7fd78dbf4700 1 --2- 172.21.15.59:0/2963227435 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fd7880a5f40 0x7fd788151970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:55.882 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.880+0000 7fd78dbf4700 1 -- 172.21.15.59:0/2963227435 >> 172.21.15.59:0/2963227435 conn(0x7fd7880631f0 msgr2=0x7fd7880af530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:08:55.882 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.880+0000 7fd78dbf4700 1 -- 172.21.15.59:0/2963227435 shutdown_connections 2023-09-30T02:08:55.882 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.880+0000 7fd78dbf4700 1 -- 172.21.15.59:0/2963227435 wait complete. 2023-09-30T02:08:55.882 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:55.881+0000 7fd78dbf4700 1 librados: shutdown 2023-09-30T02:08:58.894 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:08:59.027 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.028+0000 7f77ac60d700 10 monclient: get_monmap_and_config 2023-09-30T02:08:59.027 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.028+0000 7f77ac60d700 10 monclient: build_initial_monmap 2023-09-30T02:08:59.027 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.028+0000 7f77ac60d700 10 monclient: monmap: 2023-09-30T02:08:59.028 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:08:59.028 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:59.028 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:08:59.028544+0000 2023-09-30T02:08:59.028 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:08:59.028544+0000 2023-09-30T02:08:59.028 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:08:59.028 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:59.029 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:08:59.029 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:08:59.029 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:08:59.029 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:08:59.029 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:08:59.029 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:59.031 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.028+0000 7f77ac60d700 1 Processor -- start 2023-09-30T02:08:59.032 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.028+0000 7f77ac60d700 1 -- start start 2023-09-30T02:08:59.032 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.028+0000 7f77ac60d700 10 monclient: init 2023-09-30T02:08:59.032 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.029+0000 7f77ac60d700 10 monclient: _reopen_session rank -1 2023-09-30T02:08:59.032 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.029+0000 7f77ac60d700 10 monclient: _add_conns ranks=[0,2,4,1,3] 2023-09-30T02:08:59.032 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.029+0000 7f77ac60d700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f77a40a5b70 addr v1:172.21.15.59:6789/0 2023-09-30T02:08:59.032 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.029+0000 7f77ac60d700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f77a4142290 0x7f77a40b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:59.032 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.029+0000 7f77ac60d700 10 monclient(hunting): picked mon.noname-b con 0x7f77a4142290 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:08:59.033 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.029+0000 7f77ac60d700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f77a40a5f40 0x7f77a413af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:59.033 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.029+0000 7f77ac60d700 10 monclient(hunting): picked mon.noname-c con 0x7f77a40a5f40 addr v2:172.21.15.110:3300/0 2023-09-30T02:08:59.033 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.029+0000 7f77ac60d700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77a40a6380 con 0x7f77a40a5b70 2023-09-30T02:08:59.033 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.029+0000 7f77ac60d700 10 monclient(hunting): start opening mon connection 2023-09-30T02:08:59.033 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.029+0000 7f77ac60d700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77a40a75a0 con 0x7f77a40a5f40 2023-09-30T02:08:59.033 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.029+0000 7f77ac60d700 10 monclient(hunting): start opening mon connection 2023-09-30T02:08:59.033 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.029+0000 7f77ac60d700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77a40af3f0 con 0x7f77a4142290 2023-09-30T02:08:59.033 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.029+0000 7f77ac60d700 10 monclient(hunting): _renew_subs 2023-09-30T02:08:59.034 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.029+0000 7f77ac60d700 10 monclient(hunting): authenticate will time out at 1984.331780s 2023-09-30T02:08:59.034 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.030+0000 7f77a9ba8700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f77a4142290 0x7f77a40b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:59.034 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.030+0000 7f77aa3a9700 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f77a40a5b70 0x7f77a4141b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:58678/0 (socket says 172.21.15.59:58678) 2023-09-30T02:08:59.034 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.030+0000 7f77aa3a9700 1 -- 172.21.15.59:0/2498082442 learned_addr learned my addr 172.21.15.59:0/2498082442 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:08:59.034 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.030+0000 7f77aabaa700 1 --2- 172.21.15.59:0/2498082442 >> v2:172.21.15.110:3300/0 conn(0x7f77a40a5f40 0x7f77a413af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:59.034 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.030+0000 7f77aabaa700 10 monclient(hunting): get_auth_request con 0x7f77a40a5f40 auth_method 0 2023-09-30T02:08:59.035 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.030+0000 7f77aabaa700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:08:59.035 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.030+0000 7f77aabaa700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:59.035 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.030+0000 7f77aabaa700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:59.035 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.030+0000 7f77a9ba8700 10 monclient(hunting): get_auth_request con 0x7f77a4142290 auth_method 0 2023-09-30T02:08:59.035 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.030+0000 7f77a9ba8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:08:59.035 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.030+0000 7f77a9ba8700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:59.036 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.030+0000 7f77a9ba8700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:59.036 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.031+0000 7f77aabaa700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:08:59.036 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.031+0000 7f77aabaa700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:08:59.036 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.031+0000 7f77aabaa700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:08:59.036 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.031+0000 7f77a9ba8700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:08:59.036 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.031+0000 7f77a9ba8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:08:59.037 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.031+0000 7f77a9ba8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:08:59.037 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.033+0000 7f77a93a7700 1 -- 172.21.15.59:0/2498082442 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2603377605 0 0) 0x7f77a40a6380 con 0x7f77a40a5b70 2023-09-30T02:08:59.038 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.033+0000 7f77a93a7700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:59.038 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.033+0000 7f77a93a7700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:59.038 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.033+0000 7f77a93a7700 10 monclient(hunting): my global_id is 4608 2023-09-30T02:08:59.038 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.033+0000 7f77a93a7700 1 -- 172.21.15.59:0/2498082442 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7790003570 con 0x7f77a40a5b70 2023-09-30T02:08:59.038 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.033+0000 7f77aabaa700 10 monclient(hunting): handle_auth_done global_id 4621 payload 274 2023-09-30T02:08:59.038 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.033+0000 7f77aabaa700 1 -- 172.21.15.59:0/2498082442 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f77a4142290 msgr2=0x7f77a40b1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:08:59.038 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.033+0000 7f77aabaa700 1 --2- 172.21.15.59:0/2498082442 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f77a4142290 0x7f77a40b1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:59.039 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.033+0000 7f77aabaa700 1 -- 172.21.15.59:0/2498082442 >> v1:172.21.15.59:6789/0 conn(0x7f77a40a5b70 legacy=0x7f77a4141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:59.039 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.033+0000 7f77aabaa700 10 monclient: _finish_hunting 0 2023-09-30T02:08:59.039 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.033+0000 7f77aabaa700 1 monclient: found mon.noname-c 2023-09-30T02:08:59.039 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.033+0000 7f77aabaa700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:08:59.039 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.033+0000 7f77aabaa700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3300/0 2023-09-30T02:08:59.039 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.033+0000 7f77aabaa700 1 -- 172.21.15.59:0/2498082442 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77a413b470 con 0x7f77a40a5f40 2023-09-30T02:08:59.039 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.035+0000 7f77aabaa700 1 --2- 172.21.15.59:0/2498082442 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f77a40a5f40 0x7f77a413af30 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f779c0098e0 tx=0x7f779c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fd8f3e65985a11b6 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:08:59.045 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.039+0000 7f77a93a7700 1 -- 172.21.15.59:0/2498082442 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f779c011630 con 0x7f77a40a5f40 2023-09-30T02:08:59.045 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.039+0000 7f77a93a7700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:08:59.045 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.039+0000 7f77a93a7700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:08:59.045 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.039+0000 7f77a93a7700 10 monclient: dump: 2023-09-30T02:08:59.046 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:08:59.046 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:59.046 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:59.046 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:59.046 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:08:59.046 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:59.047 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:08:59.047 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:08:59.047 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:08:59.047 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:59.047 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.039+0000 7f77a93a7700 10 monclient: _finish_auth 0 2023-09-30T02:08:59.047 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.039+0000 7f77a93a7700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:08:59.048 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.039+0000 7f77a93a7700 1 -- 172.21.15.59:0/2498082442 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f779c011790 con 0x7f77a40a5f40 2023-09-30T02:08:59.048 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.039+0000 7f77a93a7700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:08:59.048 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.039+0000 7f77a93a7700 1 -- 172.21.15.59:0/2498082442 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f779c011b30 con 0x7f77a40a5f40 2023-09-30T02:08:59.048 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.039+0000 7f77ac60d700 5 monclient: authenticate success, global_id 4621 2023-09-30T02:08:59.048 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.039+0000 7f77ac60d700 10 monclient: get_monmap_and_config success 2023-09-30T02:08:59.048 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.039+0000 7f77a93a7700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:08:59.049 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.039+0000 7f77ac60d700 4 set_mon_vals no callback set 2023-09-30T02:08:59.049 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.039+0000 7f77a93a7700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:08:59.049 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.039+0000 7f77a93a7700 10 monclient: dump: 2023-09-30T02:08:59.049 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:08:59.049 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:59.049 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:59.050 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:59.050 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:08:59.050 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:59.050 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:08:59.050 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:08:59.051 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:08:59.051 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:59.051 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.040+0000 7f77ac60d700 10 monclient: shutdown 2023-09-30T02:08:59.051 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.040+0000 7f77ac60d700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:08:59.051 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.040+0000 7f77ac60d700 1 -- 172.21.15.59:0/2498082442 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f77a40a5f40 msgr2=0x7f77a413af30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:59.051 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.040+0000 7f77ac60d700 1 --2- 172.21.15.59:0/2498082442 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f77a40a5f40 0x7f77a413af30 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f779c0098e0 tx=0x7f779c0091a0 comp rx=0 tx=0).stop 2023-09-30T02:08:59.052 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.040+0000 7f77ac60d700 1 -- 172.21.15.59:0/2498082442 shutdown_connections 2023-09-30T02:08:59.052 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.040+0000 7f77ac60d700 1 --2- 172.21.15.59:0/2498082442 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f77a40a5f40 0x7f77a413af30 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:59.052 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.040+0000 7f77ac60d700 1 --2- 172.21.15.59:0/2498082442 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f77a4142290 0x7f77a40b1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:59.052 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.040+0000 7f77ac60d700 1 -- 172.21.15.59:0/2498082442 >> 172.21.15.59:0/2498082442 conn(0x7f77a40631f0 msgr2=0x7f77a40635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:08:59.052 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.040+0000 7f77ac60d700 1 -- 172.21.15.59:0/2498082442 shutdown_connections 2023-09-30T02:08:59.052 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.040+0000 7f77ac60d700 1 -- 172.21.15.59:0/2498082442 wait complete. 2023-09-30T02:08:59.052 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.041+0000 7f77ac60d700 10 monclient: build_initial_monmap 2023-09-30T02:08:59.053 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.041+0000 7f77ac60d700 10 monclient: monmap: 2023-09-30T02:08:59.053 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:08:59.053 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:08:59.053 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:08:59.053 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:08:59.053 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:08:59.053 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:08:59.053 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:08:59.054 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:08:59.054 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:08:59.054 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:59.054 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.042+0000 7f77ac60d700 1 librados: starting msgr at 2023-09-30T02:08:59.054 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.042+0000 7f77ac60d700 1 librados: starting objecter 2023-09-30T02:08:59.054 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.042+0000 7f77ac60d700 1 Processor -- start 2023-09-30T02:08:59.054 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.042+0000 7f77ac60d700 1 -- start start 2023-09-30T02:08:59.054 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.042+0000 7f77ac60d700 1 librados: setting wanted keys 2023-09-30T02:08:59.055 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.042+0000 7f77ac60d700 1 librados: calling monclient init 2023-09-30T02:08:59.055 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.042+0000 7f77ac60d700 10 monclient: init 2023-09-30T02:08:59.055 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.043+0000 7f77ac60d700 10 monclient: _reopen_session rank -1 2023-09-30T02:08:59.055 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.043+0000 7f77ac60d700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T02:08:59.055 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.043+0000 7f77ac60d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77a40a5b70 0x7f77a40a7af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:59.055 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.043+0000 7f77ac60d700 10 monclient(hunting): picked mon.noname-a con 0x7f77a40a5b70 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:08:59.055 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.043+0000 7f77ac60d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f77a40a5f40 0x7f77a414d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:59.055 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.043+0000 7f77ac60d700 10 monclient(hunting): picked mon.noname-b con 0x7f77a40a5f40 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:08:59.056 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.043+0000 7f77ac60d700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f77a4142290 0x7f77a414de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:59.056 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.043+0000 7f77ac60d700 10 monclient(hunting): picked mon.noname-c con 0x7f77a4142290 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:08:59.056 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.043+0000 7f77ac60d700 10 monclient(hunting): start opening mon connection 2023-09-30T02:08:59.056 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.043+0000 7f77ac60d700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77a413f100 con 0x7f77a40a5b70 2023-09-30T02:08:59.056 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.043+0000 7f77ac60d700 10 monclient(hunting): start opening mon connection 2023-09-30T02:08:59.056 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.043+0000 7f77ac60d700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77a40a91e0 con 0x7f77a40a5f40 2023-09-30T02:08:59.056 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.043+0000 7f77ac60d700 10 monclient(hunting): start opening mon connection 2023-09-30T02:08:59.056 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.043+0000 7f77ac60d700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77a413c030 con 0x7f77a4142290 2023-09-30T02:08:59.057 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.043+0000 7f77ac60d700 10 monclient(hunting): _renew_subs 2023-09-30T02:08:59.057 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.043+0000 7f77ac60d700 10 monclient(hunting): authenticate will time out at 1984.345194s 2023-09-30T02:08:59.057 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.043+0000 7f77a9ba8700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f77a40a5f40 0x7f77a414d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:59.057 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.043+0000 7f77aa3a9700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77a40a5b70 0x7f77a40a7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:59.057 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.043+0000 7f77aabaa700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f77a4142290 0x7f77a414de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:59.057 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.044+0000 7f77a9ba8700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f77a40a5f40 0x7f77a414d8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:38462/0 (socket says 172.21.15.59:38462) 2023-09-30T02:08:59.058 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.044+0000 7f77a9ba8700 1 -- 172.21.15.59:0/264324265 learned_addr learned my addr 172.21.15.59:0/264324265 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:08:59.058 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.044+0000 7f77a9ba8700 10 monclient(hunting): get_auth_request con 0x7f77a40a5f40 auth_method 0 2023-09-30T02:08:59.058 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.044+0000 7f77a9ba8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:08:59.058 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.044+0000 7f77a9ba8700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:59.058 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.044+0000 7f77a9ba8700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:59.058 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.044+0000 7f77aabaa700 10 monclient(hunting): get_auth_request con 0x7f77a4142290 auth_method 0 2023-09-30T02:08:59.058 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.044+0000 7f77aabaa700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:08:59.059 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.044+0000 7f77aabaa700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:59.059 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.044+0000 7f77aabaa700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:59.059 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.044+0000 7f77aa3a9700 10 monclient(hunting): get_auth_request con 0x7f77a40a5b70 auth_method 0 2023-09-30T02:08:59.059 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.044+0000 7f77aa3a9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:08:59.059 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.044+0000 7f77aa3a9700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:08:59.059 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.044+0000 7f77aa3a9700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:08:59.060 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.044+0000 7f77a9ba8700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:08:59.060 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.044+0000 7f77a9ba8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:08:59.060 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.044+0000 7f77a9ba8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:08:59.060 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.045+0000 7f77aabaa700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:08:59.060 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.045+0000 7f77aabaa700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:08:59.061 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.045+0000 7f77aabaa700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:08:59.061 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.045+0000 7f77aa3a9700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:08:59.061 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.045+0000 7f77aa3a9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:08:59.061 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.045+0000 7f77aa3a9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:08:59.064 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.047+0000 7f77a9ba8700 10 monclient(hunting): handle_auth_done global_id 4624 payload 824 2023-09-30T02:08:59.064 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.047+0000 7f77a9ba8700 1 -- 172.21.15.59:0/264324265 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f77a4142290 msgr2=0x7f77a414de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:59.064 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.047+0000 7f77a9ba8700 1 --2- 172.21.15.59:0/264324265 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f77a4142290 0x7f77a414de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:59.064 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.047+0000 7f77a9ba8700 1 -- 172.21.15.59:0/264324265 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77a40a5b70 msgr2=0x7f77a40a7af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:59.065 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.047+0000 7f77a9ba8700 1 --2- 172.21.15.59:0/264324265 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77a40a5b70 0x7f77a40a7af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:59.065 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.047+0000 7f77a9ba8700 10 monclient: _finish_hunting 0 2023-09-30T02:08:59.065 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.047+0000 7f77a9ba8700 1 monclient: found mon.noname-b 2023-09-30T02:08:59.065 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.047+0000 7f77a9ba8700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:08:59.065 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.047+0000 7f77a9ba8700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:08:59.065 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.047+0000 7f77a9ba8700 1 -- 172.21.15.59:0/264324265 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f779c009420 con 0x7f77a40a5f40 2023-09-30T02:08:59.065 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.050+0000 7f77a9ba8700 1 --2- 172.21.15.59:0/264324265 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f77a40a5f40 0x7f77a414d8d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f779400e930 tx=0x7f77940091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:08:59.065 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.050+0000 7f779a7fc700 10 client.?.objecter ms_handle_connect 0x7f77a40a5f40 2023-09-30T02:08:59.066 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.050+0000 7f779a7fc700 10 client.?.objecter resend_mon_ops 2023-09-30T02:08:59.066 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.053+0000 7f779a7fc700 1 -- 172.21.15.59:0/264324265 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7794013680 con 0x7f77a40a5f40 2023-09-30T02:08:59.066 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.053+0000 7f779a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:08:59.066 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.053+0000 7f779a7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:08:59.066 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.053+0000 7f779a7fc700 10 monclient: dump: 2023-09-30T02:08:59.066 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:08:59.066 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:59.067 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:59.067 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:59.067 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:08:59.067 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:59.067 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:08:59.067 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:08:59.067 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:08:59.067 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:59.068 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.053+0000 7f779a7fc700 10 monclient: _finish_auth 0 2023-09-30T02:08:59.068 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.053+0000 7f779a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:08:59.068 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.053+0000 7f779a7fc700 1 -- 172.21.15.59:0/264324265 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77940137e0 con 0x7f77a40a5f40 2023-09-30T02:08:59.068 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.053+0000 7f77ac60d700 5 monclient: authenticate success, global_id 4624 2023-09-30T02:08:59.068 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.053+0000 7f77ac60d700 10 monclient: _renew_subs 2023-09-30T02:08:59.068 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.053+0000 7f77ac60d700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:08:59.068 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.053+0000 7f77ac60d700 1 -- 172.21.15.59:0/264324265 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77a413d960 con 0x7f77a40a5f40 2023-09-30T02:08:59.069 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.053+0000 7f779a7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:08:59.069 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.053+0000 7f779a7fc700 1 -- 172.21.15.59:0/264324265 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7794013b80 con 0x7f77a40a5f40 2023-09-30T02:08:59.069 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.053+0000 7f779a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:08:59.069 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.053+0000 7f779a7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:08:59.069 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.053+0000 7f779a7fc700 10 monclient: dump: 2023-09-30T02:08:59.069 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:08:59.069 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:08:59.069 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:59.070 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:08:59.070 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:08:59.070 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:08:59.070 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:08:59.070 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:08:59.070 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:08:59.070 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:08:59.070 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.053+0000 7f77a8ba6700 4 set_mon_vals no callback set 2023-09-30T02:08:59.071 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.054+0000 7f77ac60d700 10 client.4624.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:08:59.071 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.054+0000 7f77ac60d700 10 monclient: _renew_subs 2023-09-30T02:08:59.071 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.054+0000 7f77ac60d700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:08:59.071 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.054+0000 7f77ac60d700 1 -- 172.21.15.59:0/264324265 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77a4137be0 con 0x7f77a40a5f40 2023-09-30T02:08:59.071 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.054+0000 7f77ac60d700 1 librados: init done 2023-09-30T02:08:59.071 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.055+0000 7f77ac60d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:08:59.071 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.055+0000 7f77ac60d700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:08:59.071 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.055+0000 7f77ac60d700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:08:59.072 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.055+0000 7f77ac60d700 1 -- 172.21.15.59:0/264324265 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77a40a9050 con 0x7f77a40a5f40 2023-09-30T02:08:59.304 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.305+0000 7f779a7fc700 1 -- 172.21.15.59:0/264324265 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f779400c3e0 con 0x7f77a40a5f40 2023-09-30T02:08:59.305 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.305+0000 7f779a7fc700 1 --2- 172.21.15.59:0/264324265 >> v2:172.21.15.110:6800/115870 conn(0x7f7788068a10 0x7f778806aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:08:59.305 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.305+0000 7f779a7fc700 1 -- 172.21.15.59:0/264324265 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f77940846d0 con 0x7f77a40a5f40 2023-09-30T02:08:59.306 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.305+0000 7f779a7fc700 10 client.4624.objecter ms_dispatch 0x7f77a4001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:08:59.306 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.305+0000 7f779a7fc700 3 client.4624.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:08:59.306 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.305+0000 7f779a7fc700 3 client.4624.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:08:59.306 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.306+0000 7f779a7fc700 20 client.4624.objecter dump_active .. 0 homeless 2023-09-30T02:08:59.306 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.306+0000 7f77aa3a9700 1 --2- 172.21.15.59:0/264324265 >> v2:172.21.15.110:6800/115870 conn(0x7f7788068a10 0x7f778806aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:08:59.306 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.306+0000 7f77aa3a9700 10 monclient: get_auth_request con 0x7f7788068a10 auth_method 0 2023-09-30T02:08:59.306 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.306+0000 7f77aa3a9700 1 --2- 172.21.15.59:0/264324265 >> v2:172.21.15.110:6800/115870 conn(0x7f7788068a10 0x7f778806aec0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f77a0005c50 tx=0x7f77a0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:08:59.306 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.306+0000 7f779a7fc700 10 client.4624.objecter ms_handle_connect 0x7f7788068a10 2023-09-30T02:08:59.312 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.313+0000 7f779a7fc700 1 -- 172.21.15.59:0/264324265 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7794051ff0 con 0x7f77a40a5f40 2023-09-30T02:08:59.313 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.313+0000 7f779a7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:08:59.313 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.313+0000 7f779a7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:08:59.580 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.581+0000 7f77ac60d700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:08:59.581 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.581+0000 7f77ac60d700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:08:59.581 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.581+0000 7f77ac60d700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:08:59.581 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.581+0000 7f77ac60d700 1 -- 172.21.15.59:0/264324265 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f77a4064a00 con 0x7f77a40a5f40 2023-09-30T02:08:59.587 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.587+0000 7f779a7fc700 1 -- 172.21.15.59:0/264324265 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f7794055610 con 0x7f77a40a5f40 2023-09-30T02:08:59.587 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.587+0000 7f779a7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:08:59.587 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.587+0000 7f779a7fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:08:59.588 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:08:59.588 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:08:59.591 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.590+0000 7f77ac60d700 10 librados: watch_flush enter 2023-09-30T02:08:59.591 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.590+0000 7f77ac60d700 10 librados: watch_flush exit 2023-09-30T02:08:59.591 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.590+0000 7f77ac60d700 20 client.4624.objecter shutdown clearing up homeless session... 2023-09-30T02:08:59.591 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.590+0000 7f77ac60d700 10 client.4624.objecter successfully canceled tick 2023-09-30T02:08:59.591 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.590+0000 7f77ac60d700 1 -- 172.21.15.59:0/264324265 >> v2:172.21.15.110:6800/115870 conn(0x7f7788068a10 msgr2=0x7f778806aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:59.592 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.590+0000 7f77ac60d700 1 --2- 172.21.15.59:0/264324265 >> v2:172.21.15.110:6800/115870 conn(0x7f7788068a10 0x7f778806aec0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f77a0005c50 tx=0x7f77a0009040 comp rx=0 tx=0).stop 2023-09-30T02:08:59.592 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.590+0000 7f77ac60d700 10 monclient: shutdown 2023-09-30T02:08:59.592 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.590+0000 7f77ac60d700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:08:59.592 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.590+0000 7f77ac60d700 1 -- 172.21.15.59:0/264324265 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f77a40a5f40 msgr2=0x7f77a414d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:08:59.592 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.590+0000 7f77ac60d700 1 --2- 172.21.15.59:0/264324265 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f77a40a5f40 0x7f77a414d8d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f779400e930 tx=0x7f77940091a0 comp rx=0 tx=0).stop 2023-09-30T02:08:59.592 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.590+0000 7f77ac60d700 1 -- 172.21.15.59:0/264324265 shutdown_connections 2023-09-30T02:08:59.593 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.590+0000 7f77ac60d700 1 --2- 172.21.15.59:0/264324265 >> v2:172.21.15.110:6800/115870 conn(0x7f7788068a10 0x7f778806aec0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:59.593 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.590+0000 7f77ac60d700 1 --2- 172.21.15.59:0/264324265 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f77a4142290 0x7f77a414de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:59.593 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.590+0000 7f77ac60d700 1 --2- 172.21.15.59:0/264324265 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77a40a5b70 0x7f77a40a7af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:59.593 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.590+0000 7f77ac60d700 1 --2- 172.21.15.59:0/264324265 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f77a40a5f40 0x7f77a414d8d0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:08:59.593 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.590+0000 7f77ac60d700 1 -- 172.21.15.59:0/264324265 >> 172.21.15.59:0/264324265 conn(0x7f77a40631f0 msgr2=0x7f77a40635c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:08:59.594 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.590+0000 7f77ac60d700 1 -- 172.21.15.59:0/264324265 shutdown_connections 2023-09-30T02:08:59.594 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.590+0000 7f77ac60d700 1 -- 172.21.15.59:0/264324265 wait complete. 2023-09-30T02:08:59.594 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:08:59.590+0000 7f77ac60d700 1 librados: shutdown 2023-09-30T02:09:02.604 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:09:02.744 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.745+0000 7f6b31f3c700 10 monclient: get_monmap_and_config 2023-09-30T02:09:02.745 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.745+0000 7f6b31f3c700 10 monclient: build_initial_monmap 2023-09-30T02:09:02.745 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.745+0000 7f6b31f3c700 10 monclient: monmap: 2023-09-30T02:09:02.745 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:09:02.745 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:02.745 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:09:02.745729+0000 2023-09-30T02:09:02.745 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:09:02.745729+0000 2023-09-30T02:09:02.745 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:09:02.745 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:02.745 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:09:02.745 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:09:02.745 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:09:02.746 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:09:02.746 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:09:02.746 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:02.747 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.746+0000 7f6b31f3c700 1 Processor -- start 2023-09-30T02:09:02.747 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.746+0000 7f6b31f3c700 1 -- start start 2023-09-30T02:09:02.747 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.746+0000 7f6b31f3c700 10 monclient: init 2023-09-30T02:09:02.747 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.746+0000 7f6b31f3c700 10 monclient: _reopen_session rank -1 2023-09-30T02:09:02.747 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.746+0000 7f6b31f3c700 10 monclient: _add_conns ranks=[0,1,2,4,3] 2023-09-30T02:09:02.748 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.746+0000 7f6b31f3c700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6b2c0b16e0 addr v1:172.21.15.59:6789/0 2023-09-30T02:09:02.748 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.746+0000 7f6b31f3c700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6b2c0a7650 addr v1:172.21.15.110:6789/0 2023-09-30T02:09:02.748 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.746+0000 7f6b31f3c700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6b2c146d30 0x7f6b2c149110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:02.748 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.746+0000 7f6b31f3c700 10 monclient(hunting): picked mon.noname-b con 0x7f6b2c146d30 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:09:02.748 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.746+0000 7f6b31f3c700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b2c0a5d40 con 0x7f6b2c0b16e0 2023-09-30T02:09:02.748 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.746+0000 7f6b31f3c700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b2c0b1ab0 con 0x7f6b2c0a7650 2023-09-30T02:09:02.748 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.747+0000 7f6b31f3c700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:02.748 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.747+0000 7f6b31f3c700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b2c0a6c70 con 0x7f6b2c146d30 2023-09-30T02:09:02.748 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.747+0000 7f6b31f3c700 10 monclient(hunting): _renew_subs 2023-09-30T02:09:02.748 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.747+0000 7f6b31f3c700 10 monclient(hunting): authenticate will time out at 1988.049024s 2023-09-30T02:09:02.772 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.773+0000 7f6b2affd700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f6b2c0a7650 0x7f6b2c0a7a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.59:38640/0 (socket says 172.21.15.59:38640) 2023-09-30T02:09:02.772 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.773+0000 7f6b2affd700 1 -- 172.21.15.59:0/350090673 learned_addr learned my addr 172.21.15.59:0/350090673 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:09:02.782 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.783+0000 7f6b2bfff700 1 --2- 172.21.15.59:0/350090673 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6b2c146d30 0x7f6b2c149110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:02.783 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.783+0000 7f6b2bfff700 10 monclient(hunting): get_auth_request con 0x7f6b2c146d30 auth_method 0 2023-09-30T02:09:02.783 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.783+0000 7f6b2bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:09:02.783 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.783+0000 7f6b2bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:02.783 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.783+0000 7f6b2bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:09:02.784 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.784+0000 7f6b2a7fc700 1 -- 172.21.15.59:0/350090673 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1448943603 0 0) 0x7f6b2c0a5d40 con 0x7f6b2c0b16e0 2023-09-30T02:09:02.784 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.784+0000 7f6b2a7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:02.784 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.784+0000 7f6b2a7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:09:02.784 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.784+0000 7f6b2a7fc700 10 monclient(hunting): my global_id is 4614 2023-09-30T02:09:02.784 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.784+0000 7f6b2a7fc700 1 -- 172.21.15.59:0/350090673 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b14003570 con 0x7f6b2c0b16e0 2023-09-30T02:09:02.784 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.784+0000 7f6b2a7fc700 1 -- 172.21.15.59:0/350090673 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3499379702 0 0) 0x7f6b2c0b1ab0 con 0x7f6b2c0a7650 2023-09-30T02:09:02.784 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.784+0000 7f6b2a7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:02.784 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.784+0000 7f6b2a7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:09:02.784 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.784+0000 7f6b2a7fc700 10 monclient(hunting): my global_id is 4627 2023-09-30T02:09:02.785 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.784+0000 7f6b2a7fc700 1 -- 172.21.15.59:0/350090673 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b2c0a5d40 con 0x7f6b2c0a7650 2023-09-30T02:09:02.785 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.785+0000 7f6b2bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:09:02.785 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.785+0000 7f6b2bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:09:02.785 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.785+0000 7f6b2bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:09:02.787 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.787+0000 7f6b2bfff700 10 monclient(hunting): handle_auth_done global_id 4547 payload 274 2023-09-30T02:09:02.787 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.787+0000 7f6b2bfff700 1 -- 172.21.15.59:0/350090673 >> v1:172.21.15.110:6789/0 conn(0x7f6b2c0a7650 legacy=0x7f6b2c0a7a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:02.787 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.787+0000 7f6b2bfff700 1 -- 172.21.15.59:0/350090673 >> v1:172.21.15.59:6789/0 conn(0x7f6b2c0b16e0 legacy=0x7f6b2c0a54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:02.787 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.787+0000 7f6b2bfff700 10 monclient: _finish_hunting 0 2023-09-30T02:09:02.787 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.787+0000 7f6b2bfff700 1 monclient: found mon.noname-b 2023-09-30T02:09:02.787 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.787+0000 7f6b2bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:09:02.787 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.787+0000 7f6b2bfff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.59:3301/0 2023-09-30T02:09:02.787 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.787+0000 7f6b2bfff700 1 -- 172.21.15.59:0/350090673 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b2c0a6f50 con 0x7f6b2c146d30 2023-09-30T02:09:02.791 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.792+0000 7f6b2bfff700 1 --2- 172.21.15.59:0/350090673 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6b2c146d30 0x7f6b2c149110 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6b24009e90 tx=0x7f6b24009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4c89391a581874f server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:09:02.796 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.796+0000 7f6b2a7fc700 1 -- 172.21.15.59:0/350090673 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b2400e070 con 0x7f6b2c146d30 2023-09-30T02:09:02.796 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.797+0000 7f6b2a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:09:02.797 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.797+0000 7f6b2a7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:09:02.797 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.797+0000 7f6b2a7fc700 10 monclient: dump: 2023-09-30T02:09:02.797 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:09:02.797 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:02.797 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:02.797 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:02.797 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:09:02.797 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:02.797 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:09:02.797 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:09:02.798 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:09:02.798 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:02.798 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.797+0000 7f6b2a7fc700 10 monclient: _finish_auth 0 2023-09-30T02:09:02.798 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.797+0000 7f6b2a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:09:02.798 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.797+0000 7f6b2a7fc700 1 -- 172.21.15.59:0/350090673 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b2400bb80 con 0x7f6b2c146d30 2023-09-30T02:09:02.798 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.797+0000 7f6b31f3c700 5 monclient: authenticate success, global_id 4547 2023-09-30T02:09:02.798 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.797+0000 7f6b31f3c700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:09:02.798 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.797+0000 7f6b2a7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:09:02.798 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.797+0000 7f6b31f3c700 10 monclient: get_monmap_and_config success 2023-09-30T02:09:02.798 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.797+0000 7f6b31f3c700 4 set_mon_vals no callback set 2023-09-30T02:09:02.799 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.797+0000 7f6b2a7fc700 1 -- 172.21.15.59:0/350090673 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b24012430 con 0x7f6b2c146d30 2023-09-30T02:09:02.799 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.797+0000 7f6b2a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:09:02.799 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.797+0000 7f6b2a7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:09:02.799 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.797+0000 7f6b2a7fc700 10 monclient: dump: 2023-09-30T02:09:02.799 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:09:02.799 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:02.799 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:02.799 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:02.799 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:09:02.799 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:02.799 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:09:02.799 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:09:02.800 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:09:02.800 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:02.803 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.797+0000 7f6b31f3c700 10 monclient: shutdown 2023-09-30T02:09:02.803 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.797+0000 7f6b31f3c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:09:02.803 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.797+0000 7f6b31f3c700 1 -- 172.21.15.59:0/350090673 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6b2c146d30 msgr2=0x7f6b2c149110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:02.803 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.797+0000 7f6b31f3c700 1 --2- 172.21.15.59:0/350090673 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6b2c146d30 0x7f6b2c149110 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6b24009e90 tx=0x7f6b24009780 comp rx=0 tx=0).stop 2023-09-30T02:09:02.803 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.797+0000 7f6b31f3c700 1 -- 172.21.15.59:0/350090673 shutdown_connections 2023-09-30T02:09:02.803 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.797+0000 7f6b31f3c700 1 --2- 172.21.15.59:0/350090673 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6b2c146d30 0x7f6b2c149110 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:02.803 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.798+0000 7f6b31f3c700 1 -- 172.21.15.59:0/350090673 >> 172.21.15.59:0/350090673 conn(0x7f6b2c068fe0 msgr2=0x7f6b2c0631f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:09:02.804 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.798+0000 7f6b31f3c700 1 -- 172.21.15.59:0/350090673 shutdown_connections 2023-09-30T02:09:02.804 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.798+0000 7f6b31f3c700 1 -- 172.21.15.59:0/350090673 wait complete. 2023-09-30T02:09:02.804 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.798+0000 7f6b31f3c700 10 monclient: build_initial_monmap 2023-09-30T02:09:02.804 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.799+0000 7f6b31f3c700 10 monclient: monmap: 2023-09-30T02:09:02.804 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:09:02.804 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:09:02.804 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:09:02.804 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:09:02.804 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:09:02.804 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:09:02.805 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:09:02.805 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:09:02.805 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:09:02.805 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:02.805 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.799+0000 7f6b31f3c700 1 librados: starting msgr at 2023-09-30T02:09:02.805 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.799+0000 7f6b31f3c700 1 librados: starting objecter 2023-09-30T02:09:02.805 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.799+0000 7f6b31f3c700 1 Processor -- start 2023-09-30T02:09:02.805 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.799+0000 7f6b31f3c700 1 -- start start 2023-09-30T02:09:02.805 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.799+0000 7f6b31f3c700 1 librados: setting wanted keys 2023-09-30T02:09:02.805 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.799+0000 7f6b31f3c700 1 librados: calling monclient init 2023-09-30T02:09:02.806 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.799+0000 7f6b31f3c700 10 monclient: init 2023-09-30T02:09:02.806 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.800+0000 7f6b31f3c700 10 monclient: _reopen_session rank -1 2023-09-30T02:09:02.806 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.800+0000 7f6b31f3c700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T02:09:02.806 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.800+0000 7f6b31f3c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6b2c0a7650 0x7f6b2c0abdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:02.806 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.800+0000 7f6b31f3c700 10 monclient(hunting): picked mon.noname-b con 0x7f6b2c0a7650 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:09:02.806 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.800+0000 7f6b31f3c700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6b2c0b16e0 0x7f6b2c0ae150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:02.806 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.800+0000 7f6b31f3c700 10 monclient(hunting): picked mon.noname-c con 0x7f6b2c0b16e0 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:09:02.806 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.800+0000 7f6b31f3c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6b2c146d30 0x7f6b2c0ae690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:02.806 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.800+0000 7f6b31f3c700 10 monclient(hunting): picked mon.noname-a con 0x7f6b2c146d30 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:09:02.806 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.800+0000 7f6b31f3c700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:02.807 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.800+0000 7f6b31f3c700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b2c0ad1e0 con 0x7f6b2c146d30 2023-09-30T02:09:02.807 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.800+0000 7f6b31f3c700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:02.807 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.800+0000 7f6b31f3c700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b2c0adc30 con 0x7f6b2c0a7650 2023-09-30T02:09:02.807 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.800+0000 7f6b31f3c700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:02.807 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.800+0000 7f6b31f3c700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b2c0ad8e0 con 0x7f6b2c0b16e0 2023-09-30T02:09:02.807 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.800+0000 7f6b31f3c700 10 monclient(hunting): _renew_subs 2023-09-30T02:09:02.807 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.800+0000 7f6b31f3c700 10 monclient(hunting): authenticate will time out at 1988.102813s 2023-09-30T02:09:02.807 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.801+0000 7f6b2b7fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6b2c0a7650 0x7f6b2c0abdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:02.807 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.801+0000 7f6b2bfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6b2c146d30 0x7f6b2c0ae690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:02.807 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.801+0000 7f6b2affd700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6b2c0b16e0 0x7f6b2c0ae150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:02.807 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.801+0000 7f6b2b7fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6b2c0a7650 0x7f6b2c0abdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:38478/0 (socket says 172.21.15.59:38478) 2023-09-30T02:09:02.808 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.801+0000 7f6b2b7fe700 1 -- 172.21.15.59:0/4276959385 learned_addr learned my addr 172.21.15.59:0/4276959385 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:09:02.808 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.801+0000 7f6b2b7fe700 10 monclient(hunting): get_auth_request con 0x7f6b2c0a7650 auth_method 0 2023-09-30T02:09:02.808 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.801+0000 7f6b2b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:09:02.808 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.801+0000 7f6b2b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:02.808 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.801+0000 7f6b2b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:09:02.808 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.802+0000 7f6b2affd700 10 monclient(hunting): get_auth_request con 0x7f6b2c0b16e0 auth_method 0 2023-09-30T02:09:02.808 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.802+0000 7f6b2affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:09:02.808 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.802+0000 7f6b2affd700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:02.808 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.802+0000 7f6b2affd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:09:02.809 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.802+0000 7f6b2bfff700 10 monclient(hunting): get_auth_request con 0x7f6b2c146d30 auth_method 0 2023-09-30T02:09:02.809 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.802+0000 7f6b2bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:09:02.809 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.802+0000 7f6b2bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:02.809 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.802+0000 7f6b2bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:09:02.809 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.802+0000 7f6b2b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:09:02.809 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.802+0000 7f6b2b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:09:02.809 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.802+0000 7f6b2b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:09:02.809 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.802+0000 7f6b2affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:09:02.809 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.802+0000 7f6b2affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:09:02.809 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.802+0000 7f6b2affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:09:02.810 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.803+0000 7f6b2bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:09:02.810 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.803+0000 7f6b2bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:09:02.810 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.803+0000 7f6b2bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:09:02.812 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.805+0000 7f6b2b7fe700 10 monclient(hunting): handle_auth_done global_id 4630 payload 824 2023-09-30T02:09:02.812 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.805+0000 7f6b2b7fe700 1 -- 172.21.15.59:0/4276959385 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6b2c0b16e0 msgr2=0x7f6b2c0ae150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:02.812 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.805+0000 7f6b2b7fe700 1 --2- 172.21.15.59:0/4276959385 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6b2c0b16e0 0x7f6b2c0ae150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:02.812 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.805+0000 7f6b2b7fe700 1 -- 172.21.15.59:0/4276959385 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6b2c146d30 msgr2=0x7f6b2c0ae690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:02.812 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.805+0000 7f6b2b7fe700 1 --2- 172.21.15.59:0/4276959385 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6b2c146d30 0x7f6b2c0ae690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:02.812 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.805+0000 7f6b2b7fe700 10 monclient: _finish_hunting 0 2023-09-30T02:09:02.812 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.805+0000 7f6b2b7fe700 1 monclient: found mon.noname-b 2023-09-30T02:09:02.812 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.805+0000 7f6b2b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:09:02.812 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.805+0000 7f6b2b7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:09:02.812 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.805+0000 7f6b2b7fe700 1 -- 172.21.15.59:0/4276959385 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b24009a20 con 0x7f6b2c0a7650 2023-09-30T02:09:02.812 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.807+0000 7f6b2b7fe700 1 --2- 172.21.15.59:0/4276959385 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6b2c0a7650 0x7f6b2c0abdd0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6b1c00c790 tx=0x7f6b1c007e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:09:02.813 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.807+0000 7f6b137fe700 10 client.?.objecter ms_handle_connect 0x7f6b2c0a7650 2023-09-30T02:09:02.813 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.807+0000 7f6b137fe700 10 client.?.objecter resend_mon_ops 2023-09-30T02:09:02.813 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.811+0000 7f6b137fe700 1 -- 172.21.15.59:0/4276959385 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b1c010510 con 0x7f6b2c0a7650 2023-09-30T02:09:02.813 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.811+0000 7f6b137fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:09:02.813 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.811+0000 7f6b137fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:09:02.813 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.811+0000 7f6b137fe700 10 monclient: dump: 2023-09-30T02:09:02.813 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:09:02.813 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:02.813 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:02.813 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:02.813 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:09:02.814 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:02.814 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:09:02.814 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:09:02.814 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:09:02.814 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:02.814 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.811+0000 7f6b137fe700 10 monclient: _finish_auth 0 2023-09-30T02:09:02.814 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.811+0000 7f6b137fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:09:02.814 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.811+0000 7f6b137fe700 1 -- 172.21.15.59:0/4276959385 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b1c010670 con 0x7f6b2c0a7650 2023-09-30T02:09:02.814 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.811+0000 7f6b31f3c700 5 monclient: authenticate success, global_id 4630 2023-09-30T02:09:02.814 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.811+0000 7f6b31f3c700 10 monclient: _renew_subs 2023-09-30T02:09:02.814 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.811+0000 7f6b31f3c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:09:02.815 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.811+0000 7f6b31f3c700 1 -- 172.21.15.59:0/4276959385 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b2c13c000 con 0x7f6b2c0a7650 2023-09-30T02:09:02.815 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.811+0000 7f6b137fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:09:02.815 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.811+0000 7f6b137fe700 1 -- 172.21.15.59:0/4276959385 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b1c010a10 con 0x7f6b2c0a7650 2023-09-30T02:09:02.815 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.811+0000 7f6b137fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:09:02.815 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.811+0000 7f6b137fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:09:02.815 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.811+0000 7f6b137fe700 10 monclient: dump: 2023-09-30T02:09:02.815 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:09:02.815 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:02.815 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:02.815 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:02.817 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:09:02.817 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:02.817 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:09:02.817 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:09:02.817 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:09:02.817 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:02.817 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.811+0000 7f6b29ffb700 4 set_mon_vals no callback set 2023-09-30T02:09:02.818 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.811+0000 7f6b31f3c700 10 client.4630.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:09:02.818 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.811+0000 7f6b31f3c700 10 monclient: _renew_subs 2023-09-30T02:09:02.818 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.811+0000 7f6b31f3c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:09:02.818 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.811+0000 7f6b31f3c700 1 -- 172.21.15.59:0/4276959385 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b2c13e570 con 0x7f6b2c0a7650 2023-09-30T02:09:02.818 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.811+0000 7f6b31f3c700 1 librados: init done 2023-09-30T02:09:02.818 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.812+0000 7f6b31f3c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:09:02.818 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.812+0000 7f6b31f3c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:09:02.818 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.812+0000 7f6b31f3c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:09:02.818 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:02.812+0000 7f6b31f3c700 1 -- 172.21.15.59:0/4276959385 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b2c072b30 con 0x7f6b2c0a7650 2023-09-30T02:09:03.059 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:03.059+0000 7f6b137fe700 1 -- 172.21.15.59:0/4276959385 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f6b1c01b3b0 con 0x7f6b2c0a7650 2023-09-30T02:09:03.060 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:03.060+0000 7f6b137fe700 1 --2- 172.21.15.59:0/4276959385 >> v2:172.21.15.110:6800/115870 conn(0x7f6b18068960 0x7f6b1806ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:03.060 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:03.060+0000 7f6b137fe700 1 -- 172.21.15.59:0/4276959385 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f6b1c083bb0 con 0x7f6b2c0a7650 2023-09-30T02:09:03.060 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:03.060+0000 7f6b137fe700 10 client.4630.objecter ms_dispatch 0x7f6b2c001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:09:03.060 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:03.060+0000 7f6b137fe700 3 client.4630.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:09:03.060 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:03.060+0000 7f6b137fe700 3 client.4630.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:09:03.060 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:03.060+0000 7f6b137fe700 20 client.4630.objecter dump_active .. 0 homeless 2023-09-30T02:09:03.060 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:03.060+0000 7f6b2affd700 1 --2- 172.21.15.59:0/4276959385 >> v2:172.21.15.110:6800/115870 conn(0x7f6b18068960 0x7f6b1806ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:03.060 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:03.060+0000 7f6b2affd700 10 monclient: get_auth_request con 0x7f6b18068960 auth_method 0 2023-09-30T02:09:03.061 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:03.061+0000 7f6b137fe700 10 client.4630.objecter ms_handle_connect 0x7f6b18068960 2023-09-30T02:09:03.061 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:03.061+0000 7f6b2affd700 1 --2- 172.21.15.59:0/4276959385 >> v2:172.21.15.110:6800/115870 conn(0x7f6b18068960 0x7f6b1806ae10 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6b2c13e500 tx=0x7f6b20009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:09:03.070 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:03.071+0000 7f6b137fe700 1 -- 172.21.15.59:0/4276959385 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6b1c0514d0 con 0x7f6b2c0a7650 2023-09-30T02:09:03.070 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:03.071+0000 7f6b137fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:09:03.070 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:03.071+0000 7f6b137fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:09:03.320 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:03.321+0000 7f6b31f3c700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:09:03.321 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:03.321+0000 7f6b31f3c700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:09:03.321 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:03.321+0000 7f6b31f3c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:09:03.321 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:03.321+0000 7f6b31f3c700 1 -- 172.21.15.59:0/4276959385 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f6b2c0a81a0 con 0x7f6b2c0a7650 2023-09-30T02:09:03.338 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:03.338+0000 7f6b137fe700 1 -- 172.21.15.59:0/4276959385 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f6b1c017080 con 0x7f6b2c0a7650 2023-09-30T02:09:03.338 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:03.338+0000 7f6b137fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:09:03.338 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:03.338+0000 7f6b137fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:09:03.338 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:09:03.338 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:09:03.340 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:03.340+0000 7f6b31f3c700 10 librados: watch_flush enter 2023-09-30T02:09:03.341 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:03.340+0000 7f6b31f3c700 10 librados: watch_flush exit 2023-09-30T02:09:03.341 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:03.340+0000 7f6b31f3c700 20 client.4630.objecter shutdown clearing up homeless session... 2023-09-30T02:09:03.341 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:03.340+0000 7f6b31f3c700 10 client.4630.objecter successfully canceled tick 2023-09-30T02:09:03.341 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:03.340+0000 7f6b31f3c700 1 -- 172.21.15.59:0/4276959385 >> v2:172.21.15.110:6800/115870 conn(0x7f6b18068960 msgr2=0x7f6b1806ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:03.341 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:03.340+0000 7f6b31f3c700 1 --2- 172.21.15.59:0/4276959385 >> v2:172.21.15.110:6800/115870 conn(0x7f6b18068960 0x7f6b1806ae10 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6b2c13e500 tx=0x7f6b20009040 comp rx=0 tx=0).stop 2023-09-30T02:09:03.341 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:03.340+0000 7f6b31f3c700 10 monclient: shutdown 2023-09-30T02:09:03.341 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:03.340+0000 7f6b31f3c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:09:03.341 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:03.340+0000 7f6b31f3c700 1 -- 172.21.15.59:0/4276959385 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6b2c0a7650 msgr2=0x7f6b2c0abdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:03.341 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:03.340+0000 7f6b31f3c700 1 --2- 172.21.15.59:0/4276959385 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6b2c0a7650 0x7f6b2c0abdd0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6b1c00c790 tx=0x7f6b1c007e30 comp rx=0 tx=0).stop 2023-09-30T02:09:03.341 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:03.341+0000 7f6b31f3c700 1 -- 172.21.15.59:0/4276959385 shutdown_connections 2023-09-30T02:09:03.341 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:03.341+0000 7f6b31f3c700 1 --2- 172.21.15.59:0/4276959385 >> v2:172.21.15.110:6800/115870 conn(0x7f6b18068960 0x7f6b1806ae10 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:03.342 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:03.341+0000 7f6b31f3c700 1 --2- 172.21.15.59:0/4276959385 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6b2c0a7650 0x7f6b2c0abdd0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:03.342 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:03.341+0000 7f6b31f3c700 1 --2- 172.21.15.59:0/4276959385 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6b2c146d30 0x7f6b2c0ae690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:03.342 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:03.341+0000 7f6b31f3c700 1 --2- 172.21.15.59:0/4276959385 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6b2c0b16e0 0x7f6b2c0ae150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:03.342 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:03.341+0000 7f6b31f3c700 1 -- 172.21.15.59:0/4276959385 >> 172.21.15.59:0/4276959385 conn(0x7f6b2c068fe0 msgr2=0x7f6b2c13a5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:09:03.342 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:03.341+0000 7f6b31f3c700 1 -- 172.21.15.59:0/4276959385 shutdown_connections 2023-09-30T02:09:03.342 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:03.341+0000 7f6b31f3c700 1 -- 172.21.15.59:0/4276959385 wait complete. 2023-09-30T02:09:03.342 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:03.341+0000 7f6b31f3c700 1 librados: shutdown 2023-09-30T02:09:06.354 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:09:06.494 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.495+0000 7f9571df2700 10 monclient: get_monmap_and_config 2023-09-30T02:09:06.494 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.495+0000 7f9571df2700 10 monclient: build_initial_monmap 2023-09-30T02:09:06.495 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.495+0000 7f9571df2700 10 monclient: monmap: 2023-09-30T02:09:06.495 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:09:06.495 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:06.495 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:09:06.495420+0000 2023-09-30T02:09:06.495 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:09:06.495420+0000 2023-09-30T02:09:06.495 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:09:06.495 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:06.495 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:09:06.495 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:09:06.495 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:09:06.496 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:09:06.496 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:09:06.496 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:06.497 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.495+0000 7f9571df2700 1 Processor -- start 2023-09-30T02:09:06.497 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.495+0000 7f9571df2700 1 -- start start 2023-09-30T02:09:06.498 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.495+0000 7f9571df2700 10 monclient: init 2023-09-30T02:09:06.498 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.496+0000 7f9571df2700 10 monclient: _reopen_session rank -1 2023-09-30T02:09:06.498 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.496+0000 7f9571df2700 10 monclient: _add_conns ranks=[1,0,2,4,3] 2023-09-30T02:09:06.498 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.496+0000 7f9571df2700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f956c0a5b70 addr v1:172.21.15.110:6789/0 2023-09-30T02:09:06.498 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.496+0000 7f9571df2700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f956c142290 addr v1:172.21.15.59:6789/0 2023-09-30T02:09:06.498 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.496+0000 7f9571df2700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f956c0b1500 0x7f956c0a5f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:06.498 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.496+0000 7f9571df2700 10 monclient(hunting): picked mon.noname-b con 0x7f956c0b1500 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:09:06.498 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.496+0000 7f9571df2700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f956c0a75a0 con 0x7f956c142290 2023-09-30T02:09:06.498 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.496+0000 7f9571df2700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f956c0af3f0 con 0x7f956c0a5b70 2023-09-30T02:09:06.499 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.496+0000 7f9571df2700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:06.499 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.496+0000 7f9571df2700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f956c0b1940 con 0x7f956c0b1500 2023-09-30T02:09:06.499 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.496+0000 7f9571df2700 10 monclient(hunting): _renew_subs 2023-09-30T02:09:06.499 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.496+0000 7f9571df2700 10 monclient(hunting): authenticate will time out at 1991.798849s 2023-09-30T02:09:06.499 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.497+0000 7f956b7fe700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f956c0a5b70 0x7f956c141b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.59:38642/0 (socket says 172.21.15.59:38642) 2023-09-30T02:09:06.499 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.497+0000 7f956b7fe700 1 -- 172.21.15.59:0/1012128970 learned_addr learned my addr 172.21.15.59:0/1012128970 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:09:06.499 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.498+0000 7f956bfff700 1 --2- 172.21.15.59:0/1012128970 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f956c0b1500 0x7f956c0a5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:06.500 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.500+0000 7f956bfff700 10 monclient(hunting): get_auth_request con 0x7f956c0b1500 auth_method 0 2023-09-30T02:09:06.500 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.500+0000 7f956bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:09:06.500 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.500+0000 7f956bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:06.500 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.500+0000 7f956bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:09:06.500 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.500+0000 7f956a7fc700 1 -- 172.21.15.59:0/1012128970 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1609332174 0 0) 0x7f956c0af3f0 con 0x7f956c0a5b70 2023-09-30T02:09:06.500 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.500+0000 7f956a7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:06.500 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.500+0000 7f956a7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:09:06.501 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.500+0000 7f956a7fc700 10 monclient(hunting): my global_id is 4633 2023-09-30T02:09:06.501 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.500+0000 7f956a7fc700 1 -- 172.21.15.59:0/1012128970 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9554003570 con 0x7f956c0a5b70 2023-09-30T02:09:06.501 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.501+0000 7f956bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:09:06.502 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.501+0000 7f956bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:09:06.502 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.501+0000 7f956bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:09:06.502 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.501+0000 7f956a7fc700 1 -- 172.21.15.59:0/1012128970 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 461381928 0 0) 0x7f956c0a75a0 con 0x7f956c142290 2023-09-30T02:09:06.502 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.501+0000 7f956a7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:06.502 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.501+0000 7f956a7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:09:06.502 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.501+0000 7f956a7fc700 10 monclient(hunting): my global_id is 4620 2023-09-30T02:09:06.502 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.502+0000 7f956a7fc700 1 -- 172.21.15.59:0/1012128970 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f956c0af3f0 con 0x7f956c142290 2023-09-30T02:09:06.503 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.503+0000 7f956bfff700 10 monclient(hunting): handle_auth_done global_id 4553 payload 274 2023-09-30T02:09:06.503 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.503+0000 7f956bfff700 1 -- 172.21.15.59:0/1012128970 >> v1:172.21.15.110:6789/0 conn(0x7f956c0a5b70 legacy=0x7f956c141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:06.503 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.503+0000 7f956bfff700 1 -- 172.21.15.59:0/1012128970 >> v1:172.21.15.59:6789/0 conn(0x7f956c142290 legacy=0x7f956c13af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:06.503 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.503+0000 7f956bfff700 10 monclient: _finish_hunting 0 2023-09-30T02:09:06.503 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.503+0000 7f956bfff700 1 monclient: found mon.noname-b 2023-09-30T02:09:06.503 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.504+0000 7f956bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:09:06.504 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.504+0000 7f956bfff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.59:3301/0 2023-09-30T02:09:06.504 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.504+0000 7f956bfff700 1 -- 172.21.15.59:0/1012128970 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f956c13b640 con 0x7f956c0b1500 2023-09-30T02:09:06.505 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.506+0000 7f956bfff700 1 --2- 172.21.15.59:0/1012128970 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f956c0b1500 0x7f956c0a5f40 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f9564009910 tx=0x7f95640091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a8bb44691a263b9e server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:09:06.509 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.510+0000 7f956a7fc700 1 -- 172.21.15.59:0/1012128970 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9564011630 con 0x7f956c0b1500 2023-09-30T02:09:06.510 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.510+0000 7f956a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:09:06.510 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.510+0000 7f956a7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:09:06.510 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.510+0000 7f956a7fc700 10 monclient: dump: 2023-09-30T02:09:06.510 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:09:06.510 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:06.510 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:06.510 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:06.510 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:09:06.511 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:06.511 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:09:06.511 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:09:06.511 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:09:06.511 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:06.511 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.510+0000 7f956a7fc700 10 monclient: _finish_auth 0 2023-09-30T02:09:06.511 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.510+0000 7f956a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:09:06.511 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.510+0000 7f956a7fc700 1 -- 172.21.15.59:0/1012128970 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9564011790 con 0x7f956c0b1500 2023-09-30T02:09:06.511 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.510+0000 7f9571df2700 5 monclient: authenticate success, global_id 4553 2023-09-30T02:09:06.511 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.510+0000 7f9571df2700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:09:06.512 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.510+0000 7f956a7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:09:06.512 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.510+0000 7f9571df2700 10 monclient: get_monmap_and_config success 2023-09-30T02:09:06.512 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.510+0000 7f9571df2700 4 set_mon_vals no callback set 2023-09-30T02:09:06.512 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.510+0000 7f956a7fc700 1 -- 172.21.15.59:0/1012128970 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9564011b30 con 0x7f956c0b1500 2023-09-30T02:09:06.512 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.510+0000 7f956a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:09:06.512 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.510+0000 7f956a7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:09:06.512 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.510+0000 7f956a7fc700 10 monclient: dump: 2023-09-30T02:09:06.512 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:09:06.512 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:06.512 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:06.513 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:06.513 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:09:06.513 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:06.513 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:09:06.513 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:09:06.513 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:09:06.513 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:06.517 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.511+0000 7f9571df2700 10 monclient: shutdown 2023-09-30T02:09:06.517 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.511+0000 7f9571df2700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:09:06.517 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.511+0000 7f9571df2700 1 -- 172.21.15.59:0/1012128970 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f956c0b1500 msgr2=0x7f956c0a5f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:06.517 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.511+0000 7f9571df2700 1 --2- 172.21.15.59:0/1012128970 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f956c0b1500 0x7f956c0a5f40 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f9564009910 tx=0x7f95640091a0 comp rx=0 tx=0).stop 2023-09-30T02:09:06.517 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.511+0000 7f9571df2700 1 -- 172.21.15.59:0/1012128970 shutdown_connections 2023-09-30T02:09:06.517 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.511+0000 7f9571df2700 1 --2- 172.21.15.59:0/1012128970 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f956c0b1500 0x7f956c0a5f40 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:06.517 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.511+0000 7f9571df2700 1 -- 172.21.15.59:0/1012128970 >> 172.21.15.59:0/1012128970 conn(0x7f956c0631f0 msgr2=0x7f956c0635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:09:06.517 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.511+0000 7f9571df2700 1 -- 172.21.15.59:0/1012128970 shutdown_connections 2023-09-30T02:09:06.517 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.511+0000 7f9571df2700 1 -- 172.21.15.59:0/1012128970 wait complete. 2023-09-30T02:09:06.518 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.511+0000 7f9571df2700 10 monclient: build_initial_monmap 2023-09-30T02:09:06.518 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.512+0000 7f9571df2700 10 monclient: monmap: 2023-09-30T02:09:06.518 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:09:06.518 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:09:06.518 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:09:06.518 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:09:06.518 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:09:06.518 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:09:06.518 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:09:06.518 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:09:06.519 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:09:06.519 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:06.519 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.512+0000 7f9571df2700 1 librados: starting msgr at 2023-09-30T02:09:06.519 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.512+0000 7f9571df2700 1 librados: starting objecter 2023-09-30T02:09:06.519 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.512+0000 7f9571df2700 1 Processor -- start 2023-09-30T02:09:06.519 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.512+0000 7f9571df2700 1 -- start start 2023-09-30T02:09:06.519 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.512+0000 7f9571df2700 1 librados: setting wanted keys 2023-09-30T02:09:06.519 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.512+0000 7f9571df2700 1 librados: calling monclient init 2023-09-30T02:09:06.519 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.512+0000 7f9571df2700 10 monclient: init 2023-09-30T02:09:06.519 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.513+0000 7f9571df2700 10 monclient: _reopen_session rank -1 2023-09-30T02:09:06.520 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.513+0000 7f9571df2700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T02:09:06.520 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.513+0000 7f9571df2700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f956c0a5b70 0x7f956c13dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:06.520 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.513+0000 7f9571df2700 10 monclient(hunting): picked mon.noname-a con 0x7f956c0a5b70 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:09:06.520 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.513+0000 7f9571df2700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f956c0b1500 0x7f956c13c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:06.520 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.513+0000 7f9571df2700 10 monclient(hunting): picked mon.noname-c con 0x7f956c0b1500 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:09:06.520 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.513+0000 7f9571df2700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f956c142290 0x7f956c13e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:06.520 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.513+0000 7f9571df2700 10 monclient(hunting): picked mon.noname-b con 0x7f956c142290 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:09:06.520 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.513+0000 7f9571df2700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:06.520 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.513+0000 7f9571df2700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f956c13e430 con 0x7f956c0a5b70 2023-09-30T02:09:06.520 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.513+0000 7f9571df2700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:06.521 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.513+0000 7f9571df2700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f956c145770 con 0x7f956c142290 2023-09-30T02:09:06.521 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.513+0000 7f9571df2700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:06.521 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.513+0000 7f9571df2700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f956c13ee00 con 0x7f956c0b1500 2023-09-30T02:09:06.521 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.513+0000 7f9571df2700 10 monclient(hunting): _renew_subs 2023-09-30T02:09:06.521 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.513+0000 7f9571df2700 10 monclient(hunting): authenticate will time out at 1991.815259s 2023-09-30T02:09:06.521 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.513+0000 7f956affd700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f956c0b1500 0x7f956c13c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:06.521 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.513+0000 7f956b7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f956c0a5b70 0x7f956c13dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:06.521 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.513+0000 7f956bfff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f956c142290 0x7f956c13e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:06.521 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.514+0000 7f956affd700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f956c0b1500 0x7f956c13c030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:51328/0 (socket says 172.21.15.59:51328) 2023-09-30T02:09:06.521 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.514+0000 7f956affd700 1 -- 172.21.15.59:0/1935532615 learned_addr learned my addr 172.21.15.59:0/1935532615 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:09:06.521 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.514+0000 7f956affd700 10 monclient(hunting): get_auth_request con 0x7f956c0b1500 auth_method 0 2023-09-30T02:09:06.522 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.514+0000 7f956affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:09:06.522 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.514+0000 7f956affd700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:06.522 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.514+0000 7f956affd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:09:06.522 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.514+0000 7f956bfff700 10 monclient(hunting): get_auth_request con 0x7f956c142290 auth_method 0 2023-09-30T02:09:06.522 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.514+0000 7f956bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:09:06.522 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.514+0000 7f956bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:06.522 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.514+0000 7f956bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:09:06.522 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.514+0000 7f956b7fe700 10 monclient(hunting): get_auth_request con 0x7f956c0a5b70 auth_method 0 2023-09-30T02:09:06.522 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.514+0000 7f956b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:09:06.522 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.514+0000 7f956b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:06.523 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.514+0000 7f956b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:09:06.523 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.514+0000 7f956affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:09:06.523 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.514+0000 7f956affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:09:06.523 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.514+0000 7f956affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:09:06.523 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.514+0000 7f956bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:09:06.523 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.514+0000 7f956bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:09:06.523 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.514+0000 7f956bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:09:06.523 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.515+0000 7f956b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:09:06.523 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.515+0000 7f956b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:09:06.523 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.515+0000 7f956b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:09:06.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.518+0000 7f956affd700 10 monclient(hunting): handle_auth_done global_id 4556 payload 824 2023-09-30T02:09:06.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.518+0000 7f956affd700 1 -- 172.21.15.59:0/1935532615 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f956c142290 msgr2=0x7f956c13e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:06.526 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.518+0000 7f956affd700 1 --2- 172.21.15.59:0/1935532615 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f956c142290 0x7f956c13e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:06.526 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.518+0000 7f956affd700 1 -- 172.21.15.59:0/1935532615 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f956c0a5b70 msgr2=0x7f956c13dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:06.526 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.518+0000 7f956affd700 1 --2- 172.21.15.59:0/1935532615 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f956c0a5b70 0x7f956c13dc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:06.526 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.518+0000 7f956affd700 10 monclient: _finish_hunting 0 2023-09-30T02:09:06.526 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.518+0000 7f956affd700 1 monclient: found mon.noname-c 2023-09-30T02:09:06.526 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.518+0000 7f956affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:09:06.526 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.518+0000 7f956affd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.59:3301/0 2023-09-30T02:09:06.526 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.518+0000 7f956affd700 1 -- 172.21.15.59:0/1935532615 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9564009420 con 0x7f956c0b1500 2023-09-30T02:09:06.527 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.521+0000 7f956affd700 1 --2- 172.21.15.59:0/1935532615 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f956c0b1500 0x7f956c13c030 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f956000c590 tx=0x7f9560007dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:09:06.527 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.521+0000 7f95537fe700 10 client.?.objecter ms_handle_connect 0x7f956c0b1500 2023-09-30T02:09:06.527 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.521+0000 7f95537fe700 10 client.?.objecter resend_mon_ops 2023-09-30T02:09:06.527 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.525+0000 7f95537fe700 1 -- 172.21.15.59:0/1935532615 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9560010510 con 0x7f956c0b1500 2023-09-30T02:09:06.527 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.525+0000 7f95537fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:09:06.527 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.525+0000 7f95537fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:09:06.527 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.525+0000 7f95537fe700 10 monclient: dump: 2023-09-30T02:09:06.527 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:09:06.527 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:06.527 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:06.528 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:06.528 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:09:06.528 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:06.528 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:09:06.528 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:09:06.528 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:09:06.528 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:06.528 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.525+0000 7f95537fe700 10 monclient: _finish_auth 0 2023-09-30T02:09:06.528 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.525+0000 7f95537fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:09:06.528 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.525+0000 7f95537fe700 1 -- 172.21.15.59:0/1935532615 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9560010670 con 0x7f956c0b1500 2023-09-30T02:09:06.528 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.525+0000 7f95537fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:09:06.529 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.525+0000 7f95537fe700 1 -- 172.21.15.59:0/1935532615 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9560010a30 con 0x7f956c0b1500 2023-09-30T02:09:06.529 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.525+0000 7f95537fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:09:06.529 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.525+0000 7f95537fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:09:06.529 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.525+0000 7f95537fe700 10 monclient: dump: 2023-09-30T02:09:06.529 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:09:06.529 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:06.529 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:06.529 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:06.529 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:09:06.529 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:06.530 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:09:06.530 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:09:06.530 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:09:06.530 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:06.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.525+0000 7f9571df2700 5 monclient: authenticate success, global_id 4556 2023-09-30T02:09:06.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.525+0000 7f9571df2700 10 monclient: _renew_subs 2023-09-30T02:09:06.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.525+0000 7f9571df2700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:09:06.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.525+0000 7f9571df2700 1 -- 172.21.15.59:0/1935532615 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f956c1448a0 con 0x7f956c0b1500 2023-09-30T02:09:06.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.525+0000 7f9569ffb700 4 set_mon_vals no callback set 2023-09-30T02:09:06.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.525+0000 7f9571df2700 10 client.4556.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:09:06.531 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.525+0000 7f9571df2700 10 monclient: _renew_subs 2023-09-30T02:09:06.531 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.525+0000 7f9571df2700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:09:06.531 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.525+0000 7f9571df2700 1 -- 172.21.15.59:0/1935532615 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f956c144d90 con 0x7f956c0b1500 2023-09-30T02:09:06.531 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.525+0000 7f9571df2700 1 librados: init done 2023-09-30T02:09:06.531 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.526+0000 7f9571df2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:09:06.531 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.526+0000 7f9571df2700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:09:06.531 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.526+0000 7f9571df2700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:09:06.531 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.526+0000 7f9571df2700 1 -- 172.21.15.59:0/1935532615 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f956c13e5a0 con 0x7f956c0b1500 2023-09-30T02:09:06.618 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.618+0000 7f95537fe700 1 -- 172.21.15.59:0/1935532615 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f9560025430 con 0x7f956c0b1500 2023-09-30T02:09:06.619 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.619+0000 7f95537fe700 1 --2- 172.21.15.59:0/1935532615 >> v2:172.21.15.110:6800/115870 conn(0x7f9558068910 0x7f955806adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:06.619 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.619+0000 7f95537fe700 1 -- 172.21.15.59:0/1935532615 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f9560084500 con 0x7f956c0b1500 2023-09-30T02:09:06.619 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.619+0000 7f95537fe700 10 client.4556.objecter ms_dispatch 0x7f956c001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:09:06.619 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.619+0000 7f95537fe700 3 client.4556.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:09:06.619 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.619+0000 7f95537fe700 3 client.4556.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:09:06.619 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.619+0000 7f95537fe700 20 client.4556.objecter dump_active .. 0 homeless 2023-09-30T02:09:06.619 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.619+0000 7f956b7fe700 1 --2- 172.21.15.59:0/1935532615 >> v2:172.21.15.110:6800/115870 conn(0x7f9558068910 0x7f955806adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:06.619 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.619+0000 7f956b7fe700 10 monclient: get_auth_request con 0x7f9558068910 auth_method 0 2023-09-30T02:09:06.620 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.620+0000 7f95537fe700 10 client.4556.objecter ms_handle_connect 0x7f9558068910 2023-09-30T02:09:06.620 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.620+0000 7f956b7fe700 1 --2- 172.21.15.59:0/1935532615 >> v2:172.21.15.110:6800/115870 conn(0x7f9558068910 0x7f955806adc0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f955c005c50 tx=0x7f955c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:09:06.831 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.832+0000 7f95537fe700 1 -- 172.21.15.59:0/1935532615 <== mon.2 v2:172.21.15.59:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9560051da0 con 0x7f956c0b1500 2023-09-30T02:09:06.832 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.832+0000 7f95537fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:09:06.832 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:06.832+0000 7f95537fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:09:07.086 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:07.087+0000 7f9571df2700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:09:07.086 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:07.087+0000 7f9571df2700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:09:07.087 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:07.087+0000 7f9571df2700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:09:07.087 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:07.087+0000 7f9571df2700 1 -- 172.21.15.59:0/1935532615 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f956c064a00 con 0x7f956c0b1500 2023-09-30T02:09:07.092 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:07.093+0000 7f95537fe700 1 -- 172.21.15.59:0/1935532615 <== mon.2 v2:172.21.15.59:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f95600553c0 con 0x7f956c0b1500 2023-09-30T02:09:07.092 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:07.093+0000 7f95537fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:09:07.092 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:07.093+0000 7f95537fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:09:07.093 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:09:07.093 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:09:07.095 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:07.095+0000 7f9571df2700 10 librados: watch_flush enter 2023-09-30T02:09:07.095 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:07.095+0000 7f9571df2700 10 librados: watch_flush exit 2023-09-30T02:09:07.095 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:07.095+0000 7f9571df2700 20 client.4556.objecter shutdown clearing up homeless session... 2023-09-30T02:09:07.095 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:07.095+0000 7f9571df2700 10 client.4556.objecter successfully canceled tick 2023-09-30T02:09:07.095 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:07.095+0000 7f9571df2700 1 -- 172.21.15.59:0/1935532615 >> v2:172.21.15.110:6800/115870 conn(0x7f9558068910 msgr2=0x7f955806adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:07.096 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:07.095+0000 7f9571df2700 1 --2- 172.21.15.59:0/1935532615 >> v2:172.21.15.110:6800/115870 conn(0x7f9558068910 0x7f955806adc0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f955c005c50 tx=0x7f955c009040 comp rx=0 tx=0).stop 2023-09-30T02:09:07.096 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:07.095+0000 7f9571df2700 10 monclient: shutdown 2023-09-30T02:09:07.096 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:07.095+0000 7f9571df2700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:09:07.096 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:07.095+0000 7f9571df2700 1 -- 172.21.15.59:0/1935532615 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f956c0b1500 msgr2=0x7f956c13c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:07.096 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:07.095+0000 7f9571df2700 1 --2- 172.21.15.59:0/1935532615 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f956c0b1500 0x7f956c13c030 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f956000c590 tx=0x7f9560007dc0 comp rx=0 tx=0).stop 2023-09-30T02:09:07.096 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:07.095+0000 7f9571df2700 1 -- 172.21.15.59:0/1935532615 shutdown_connections 2023-09-30T02:09:07.096 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:07.095+0000 7f9571df2700 1 --2- 172.21.15.59:0/1935532615 >> v2:172.21.15.110:6800/115870 conn(0x7f9558068910 0x7f955806adc0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:07.096 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:07.095+0000 7f9571df2700 1 --2- 172.21.15.59:0/1935532615 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f956c142290 0x7f956c13e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:07.096 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:07.095+0000 7f9571df2700 1 --2- 172.21.15.59:0/1935532615 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f956c0a5b70 0x7f956c13dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:07.096 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:07.095+0000 7f9571df2700 1 --2- 172.21.15.59:0/1935532615 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f956c0b1500 0x7f956c13c030 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:07.096 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:07.095+0000 7f9571df2700 1 -- 172.21.15.59:0/1935532615 >> 172.21.15.59:0/1935532615 conn(0x7f956c0631f0 msgr2=0x7f956c0b0be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:09:07.096 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:07.095+0000 7f9571df2700 1 -- 172.21.15.59:0/1935532615 shutdown_connections 2023-09-30T02:09:07.097 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:07.095+0000 7f9571df2700 1 -- 172.21.15.59:0/1935532615 wait complete. 2023-09-30T02:09:07.097 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:07.095+0000 7f9571df2700 1 librados: shutdown 2023-09-30T02:09:10.110 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:09:10.262 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.262+0000 7f1034f1a700 10 monclient: get_monmap_and_config 2023-09-30T02:09:10.262 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.262+0000 7f1034f1a700 10 monclient: build_initial_monmap 2023-09-30T02:09:10.262 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.262+0000 7f1034f1a700 10 monclient: monmap: 2023-09-30T02:09:10.262 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:09:10.262 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:10.262 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:09:10.262831+0000 2023-09-30T02:09:10.262 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:09:10.262831+0000 2023-09-30T02:09:10.263 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:09:10.263 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:10.263 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:09:10.263 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:09:10.263 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:09:10.263 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:09:10.263 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:09:10.263 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:10.263 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.263+0000 7f1034f1a700 1 Processor -- start 2023-09-30T02:09:10.263 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.263+0000 7f1034f1a700 1 -- start start 2023-09-30T02:09:10.263 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.263+0000 7f1034f1a700 10 monclient: init 2023-09-30T02:09:10.265 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.264+0000 7f1034f1a700 10 monclient: _reopen_session rank -1 2023-09-30T02:09:10.265 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.264+0000 7f1034f1a700 10 monclient: _add_conns ranks=[0,4,3,1,2] 2023-09-30T02:09:10.265 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.264+0000 7f1034f1a700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f10300a5b70 addr v1:172.21.15.59:6789/0 2023-09-30T02:09:10.266 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.264+0000 7f1034f1a700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f1030142290 0x7f10300b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:10.266 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.264+0000 7f1034f1a700 10 monclient(hunting): picked mon.noname-c con 0x7f1030142290 addr v2:172.21.15.110:3300/0 2023-09-30T02:09:10.266 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.264+0000 7f1034f1a700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f10300a5f40 0x7f103013af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:10.266 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.264+0000 7f1034f1a700 10 monclient(hunting): picked mon.noname-a con 0x7f10300a5f40 addr v2:172.21.15.59:3300/0 2023-09-30T02:09:10.266 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.264+0000 7f1034f1a700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10300a75a0 con 0x7f10300a5b70 2023-09-30T02:09:10.266 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.264+0000 7f1034f1a700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:10.266 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.264+0000 7f1034f1a700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10300a6360 con 0x7f10300a5f40 2023-09-30T02:09:10.266 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.264+0000 7f1034f1a700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:10.266 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.264+0000 7f1034f1a700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10300af3f0 con 0x7f1030142290 2023-09-30T02:09:10.266 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.264+0000 7f1034f1a700 10 monclient(hunting): _renew_subs 2023-09-30T02:09:10.266 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.264+0000 7f1034f1a700 10 monclient(hunting): authenticate will time out at 1995.566148s 2023-09-30T02:09:10.267 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.264+0000 7f102dd9b700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f1030142290 0x7f10300b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:10.267 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.264+0000 7f102ed9d700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f10300a5f40 0x7f103013af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:10.267 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.265+0000 7f102dd9b700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f1030142290 0x7f10300b1500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:36860/0 (socket says 172.21.15.59:36860) 2023-09-30T02:09:10.267 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.265+0000 7f102dd9b700 1 -- 172.21.15.59:0/2167290644 learned_addr learned my addr 172.21.15.59:0/2167290644 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:09:10.267 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.265+0000 7f102dd9b700 10 monclient(hunting): get_auth_request con 0x7f1030142290 auth_method 0 2023-09-30T02:09:10.267 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.265+0000 7f102dd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:09:10.267 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.265+0000 7f102dd9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:10.267 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.265+0000 7f102dd9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:09:10.267 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.265+0000 7f102ed9d700 10 monclient(hunting): get_auth_request con 0x7f10300a5f40 auth_method 0 2023-09-30T02:09:10.267 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.265+0000 7f102ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:09:10.267 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.265+0000 7f102ed9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:10.267 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.265+0000 7f102ed9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:09:10.268 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.265+0000 7f102dd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:09:10.268 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.265+0000 7f102dd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:09:10.268 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.265+0000 7f102dd9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:09:10.268 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.266+0000 7f102ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:09:10.268 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.266+0000 7f102ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:09:10.268 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.266+0000 7f102ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:09:10.269 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.268+0000 7f102dd9b700 10 monclient(hunting): handle_auth_done global_id 4639 payload 274 2023-09-30T02:09:10.269 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.268+0000 7f102dd9b700 1 -- 172.21.15.59:0/2167290644 >> v2:172.21.15.59:3300/0 conn(0x7f10300a5f40 msgr2=0x7f103013af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:09:10.269 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.268+0000 7f102dd9b700 1 --2- 172.21.15.59:0/2167290644 >> v2:172.21.15.59:3300/0 conn(0x7f10300a5f40 0x7f103013af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:10.269 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.268+0000 7f102dd9b700 1 -- 172.21.15.59:0/2167290644 >> v1:172.21.15.59:6789/0 conn(0x7f10300a5b70 legacy=0x7f1030141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:09:10.269 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.268+0000 7f102dd9b700 10 monclient: _finish_hunting 0 2023-09-30T02:09:10.269 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.268+0000 7f102dd9b700 1 monclient: found mon.noname-c 2023-09-30T02:09:10.269 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.268+0000 7f102dd9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:09:10.269 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.268+0000 7f102dd9b700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3300/0 2023-09-30T02:09:10.269 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.268+0000 7f102dd9b700 1 -- 172.21.15.59:0/2167290644 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f103013b470 con 0x7f1030142290 2023-09-30T02:09:10.270 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.270+0000 7f102dd9b700 1 --2- 172.21.15.59:0/2167290644 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1030142290 0x7f10300b1500 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f10300b1fc0 tx=0x7f1024009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=229c90f11f6ddf77 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:09:10.274 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.274+0000 7f102d59a700 1 -- 172.21.15.59:0/2167290644 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1024009e70 con 0x7f1030142290 2023-09-30T02:09:10.274 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.274+0000 7f102d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:09:10.274 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.274+0000 7f102d59a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:09:10.274 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.274+0000 7f102d59a700 10 monclient: dump: 2023-09-30T02:09:10.274 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:09:10.274 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:10.274 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:10.274 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:10.274 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:09:10.274 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:10.275 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:09:10.275 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:09:10.275 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:09:10.275 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:10.275 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.274+0000 7f102d59a700 10 monclient: _finish_auth 0 2023-09-30T02:09:10.275 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.274+0000 7f102d59a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:09:10.275 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.274+0000 7f102d59a700 1 -- 172.21.15.59:0/2167290644 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f102400b820 con 0x7f1030142290 2023-09-30T02:09:10.275 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.274+0000 7f1034f1a700 5 monclient: authenticate success, global_id 4639 2023-09-30T02:09:10.275 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.274+0000 7f1034f1a700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:09:10.275 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.274+0000 7f102d59a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:09:10.275 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.274+0000 7f1034f1a700 10 monclient: get_monmap_and_config success 2023-09-30T02:09:10.275 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.274+0000 7f102d59a700 1 -- 172.21.15.59:0/2167290644 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f102400bbc0 con 0x7f1030142290 2023-09-30T02:09:10.276 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.274+0000 7f1034f1a700 4 set_mon_vals no callback set 2023-09-30T02:09:10.276 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.274+0000 7f102d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:09:10.276 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.274+0000 7f102d59a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:09:10.276 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.274+0000 7f102d59a700 10 monclient: dump: 2023-09-30T02:09:10.276 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:09:10.276 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:10.276 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:10.276 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:10.276 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:09:10.276 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:10.276 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:09:10.276 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:09:10.277 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:09:10.277 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:10.279 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.275+0000 7f1034f1a700 10 monclient: shutdown 2023-09-30T02:09:10.279 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.275+0000 7f1034f1a700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:09:10.280 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.275+0000 7f1034f1a700 1 -- 172.21.15.59:0/2167290644 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1030142290 msgr2=0x7f10300b1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:10.280 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.275+0000 7f1034f1a700 1 --2- 172.21.15.59:0/2167290644 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1030142290 0x7f10300b1500 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f10300b1fc0 tx=0x7f1024009250 comp rx=0 tx=0).stop 2023-09-30T02:09:10.280 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.275+0000 7f1034f1a700 1 -- 172.21.15.59:0/2167290644 shutdown_connections 2023-09-30T02:09:10.280 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.275+0000 7f1034f1a700 1 --2- 172.21.15.59:0/2167290644 >> v2:172.21.15.59:3300/0 conn(0x7f10300a5f40 0x7f103013af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:10.280 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.275+0000 7f1034f1a700 1 --2- 172.21.15.59:0/2167290644 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1030142290 0x7f10300b1500 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:10.280 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.275+0000 7f1034f1a700 1 -- 172.21.15.59:0/2167290644 >> 172.21.15.59:0/2167290644 conn(0x7f10300631f0 msgr2=0x7f10300635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:09:10.280 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.275+0000 7f1034f1a700 1 -- 172.21.15.59:0/2167290644 shutdown_connections 2023-09-30T02:09:10.280 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.275+0000 7f1034f1a700 1 -- 172.21.15.59:0/2167290644 wait complete. 2023-09-30T02:09:10.280 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.276+0000 7f1034f1a700 10 monclient: build_initial_monmap 2023-09-30T02:09:10.280 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.276+0000 7f1034f1a700 10 monclient: monmap: 2023-09-30T02:09:10.281 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:09:10.281 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:09:10.281 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:09:10.281 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:09:10.281 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:09:10.281 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:09:10.281 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:09:10.281 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:09:10.281 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:09:10.281 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:10.281 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.276+0000 7f1034f1a700 1 librados: starting msgr at 2023-09-30T02:09:10.281 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.276+0000 7f1034f1a700 1 librados: starting objecter 2023-09-30T02:09:10.282 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.276+0000 7f1034f1a700 1 Processor -- start 2023-09-30T02:09:10.282 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.277+0000 7f1034f1a700 1 -- start start 2023-09-30T02:09:10.282 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.277+0000 7f1034f1a700 1 librados: setting wanted keys 2023-09-30T02:09:10.282 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.277+0000 7f1034f1a700 1 librados: calling monclient init 2023-09-30T02:09:10.282 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.277+0000 7f1034f1a700 10 monclient: init 2023-09-30T02:09:10.282 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.277+0000 7f1034f1a700 10 monclient: _reopen_session rank -1 2023-09-30T02:09:10.282 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.277+0000 7f1034f1a700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T02:09:10.282 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.277+0000 7f1034f1a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10300a5b70 0x7f10300a80e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:10.282 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.277+0000 7f1034f1a700 10 monclient(hunting): picked mon.noname-a con 0x7f10300a5b70 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:09:10.282 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.277+0000 7f1034f1a700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f10300a5f40 0x7f1030151450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:10.282 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.277+0000 7f1034f1a700 10 monclient(hunting): picked mon.noname-b con 0x7f10300a5f40 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:09:10.283 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.277+0000 7f1034f1a700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1030142290 0x7f1030151990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:10.283 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.277+0000 7f1034f1a700 10 monclient(hunting): picked mon.noname-c con 0x7f1030142290 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:09:10.283 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.277+0000 7f1034f1a700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:10.283 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.277+0000 7f1034f1a700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10300a7d30 con 0x7f10300a5b70 2023-09-30T02:09:10.283 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.277+0000 7f1034f1a700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:10.283 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.277+0000 7f1034f1a700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10300a78a0 con 0x7f10300a5f40 2023-09-30T02:09:10.283 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.277+0000 7f1034f1a700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:10.283 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.277+0000 7f1034f1a700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f103013f090 con 0x7f1030142290 2023-09-30T02:09:10.283 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.277+0000 7f1034f1a700 10 monclient(hunting): _renew_subs 2023-09-30T02:09:10.283 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.277+0000 7f1034f1a700 10 monclient(hunting): authenticate will time out at 1995.579918s 2023-09-30T02:09:10.283 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.278+0000 7f102dd9b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f10300a5f40 0x7f1030151450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:10.283 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.278+0000 7f102dd9b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f10300a5f40 0x7f1030151450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:36876/0 (socket says 172.21.15.59:36876) 2023-09-30T02:09:10.283 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.278+0000 7f102dd9b700 1 -- 172.21.15.59:0/33975824 learned_addr learned my addr 172.21.15.59:0/33975824 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:09:10.284 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.278+0000 7f102dd9b700 10 monclient(hunting): get_auth_request con 0x7f10300a5f40 auth_method 0 2023-09-30T02:09:10.284 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.278+0000 7f102dd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:09:10.284 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.278+0000 7f102dd9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:10.284 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.278+0000 7f102dd9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:09:10.284 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.279+0000 7f102e59c700 1 --2- 172.21.15.59:0/33975824 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10300a5b70 0x7f10300a80e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:10.284 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.279+0000 7f102dd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:09:10.284 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.279+0000 7f102dd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:09:10.284 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.279+0000 7f102dd9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:09:10.284 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.279+0000 7f102e59c700 10 monclient(hunting): get_auth_request con 0x7f10300a5b70 auth_method 0 2023-09-30T02:09:10.284 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.279+0000 7f102e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:09:10.284 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.279+0000 7f102e59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:10.284 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.279+0000 7f102e59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:09:10.285 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.279+0000 7f102ed9d700 1 --2- 172.21.15.59:0/33975824 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1030142290 0x7f1030151990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:10.285 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.280+0000 7f102e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:09:10.285 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.280+0000 7f102e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:09:10.285 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.280+0000 7f102e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:09:10.285 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.280+0000 7f102ed9d700 10 monclient(hunting): get_auth_request con 0x7f1030142290 auth_method 0 2023-09-30T02:09:10.285 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.280+0000 7f102ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:09:10.285 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.280+0000 7f102ed9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:10.285 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.280+0000 7f102ed9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:09:10.287 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.281+0000 7f102ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:09:10.288 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.281+0000 7f102ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:09:10.288 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.281+0000 7f102ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:09:10.288 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.282+0000 7f102dd9b700 10 monclient(hunting): handle_auth_done global_id 4642 payload 824 2023-09-30T02:09:10.288 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.282+0000 7f102dd9b700 1 -- 172.21.15.59:0/33975824 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1030142290 msgr2=0x7f1030151990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:10.288 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.282+0000 7f102dd9b700 1 --2- 172.21.15.59:0/33975824 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1030142290 0x7f1030151990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:10.288 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.282+0000 7f102dd9b700 1 -- 172.21.15.59:0/33975824 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10300a5b70 msgr2=0x7f10300a80e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:10.288 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.282+0000 7f102dd9b700 1 --2- 172.21.15.59:0/33975824 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10300a5b70 0x7f10300a80e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:10.288 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.282+0000 7f102dd9b700 10 monclient: _finish_hunting 0 2023-09-30T02:09:10.288 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.282+0000 7f102dd9b700 1 monclient: found mon.noname-b 2023-09-30T02:09:10.288 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.282+0000 7f102dd9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:09:10.289 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.282+0000 7f102dd9b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:09:10.289 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.282+0000 7f102dd9b700 1 -- 172.21.15.59:0/33975824 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10240094f0 con 0x7f10300a5f40 2023-09-30T02:09:10.290 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.291+0000 7f102dd9b700 1 --2- 172.21.15.59:0/33975824 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f10300a5f40 0x7f1030151450 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f1024006fd0 tx=0x7f1024016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:09:10.290 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.291+0000 7f101e7fc700 10 client.?.objecter ms_handle_connect 0x7f10300a5f40 2023-09-30T02:09:10.290 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.291+0000 7f101e7fc700 10 client.?.objecter resend_mon_ops 2023-09-30T02:09:10.295 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.295+0000 7f101e7fc700 1 -- 172.21.15.59:0/33975824 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1024011d00 con 0x7f10300a5f40 2023-09-30T02:09:10.298 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.295+0000 7f101e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:09:10.298 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.295+0000 7f101e7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:09:10.298 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.295+0000 7f101e7fc700 10 monclient: dump: 2023-09-30T02:09:10.298 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:09:10.298 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:10.298 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:10.299 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:10.299 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:09:10.299 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:10.299 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:09:10.299 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:09:10.299 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:09:10.299 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:10.299 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.295+0000 7f101e7fc700 10 monclient: _finish_auth 0 2023-09-30T02:09:10.299 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.295+0000 7f101e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:09:10.299 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.295+0000 7f101e7fc700 1 -- 172.21.15.59:0/33975824 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1024011e80 con 0x7f10300a5f40 2023-09-30T02:09:10.299 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.295+0000 7f1034f1a700 5 monclient: authenticate success, global_id 4642 2023-09-30T02:09:10.300 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.295+0000 7f1034f1a700 10 monclient: _renew_subs 2023-09-30T02:09:10.300 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.295+0000 7f1034f1a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:09:10.300 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.295+0000 7f1034f1a700 1 -- 172.21.15.59:0/33975824 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f103013edc0 con 0x7f10300a5f40 2023-09-30T02:09:10.300 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.295+0000 7f101e7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:09:10.300 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.295+0000 7f101e7fc700 1 -- 172.21.15.59:0/33975824 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f102401b6c0 con 0x7f10300a5f40 2023-09-30T02:09:10.300 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.295+0000 7f101e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:09:10.300 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.295+0000 7f101e7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:09:10.300 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.295+0000 7f101e7fc700 10 monclient: dump: 2023-09-30T02:09:10.300 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:09:10.300 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:10.300 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:10.300 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:10.301 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:09:10.301 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:10.301 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:09:10.301 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:09:10.301 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:09:10.301 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:10.301 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.295+0000 7f102cd99700 4 set_mon_vals no callback set 2023-09-30T02:09:10.301 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.295+0000 7f1034f1a700 10 client.4642.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:09:10.301 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.295+0000 7f1034f1a700 10 monclient: _renew_subs 2023-09-30T02:09:10.301 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.295+0000 7f1034f1a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:09:10.301 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.295+0000 7f1034f1a700 1 -- 172.21.15.59:0/33975824 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f103013e180 con 0x7f10300a5f40 2023-09-30T02:09:10.301 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.295+0000 7f1034f1a700 1 librados: init done 2023-09-30T02:09:10.302 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.296+0000 7f1034f1a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:09:10.302 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.296+0000 7f1034f1a700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:09:10.302 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.296+0000 7f1034f1a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:09:10.302 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.296+0000 7f1034f1a700 1 -- 172.21.15.59:0/33975824 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1030064a00 con 0x7f10300a5f40 2023-09-30T02:09:10.345 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.345+0000 7f101e7fc700 1 -- 172.21.15.59:0/33975824 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f102401ba30 con 0x7f10300a5f40 2023-09-30T02:09:10.346 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.346+0000 7f101e7fc700 1 --2- 172.21.15.59:0/33975824 >> v2:172.21.15.110:6800/115870 conn(0x7f1010068910 0x7f101006adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:10.346 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.346+0000 7f101e7fc700 1 -- 172.21.15.59:0/33975824 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f1024085d00 con 0x7f10300a5f40 2023-09-30T02:09:10.346 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.346+0000 7f101e7fc700 10 client.4642.objecter ms_dispatch 0x7f1030001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:09:10.346 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.346+0000 7f101e7fc700 3 client.4642.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:09:10.346 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.346+0000 7f101e7fc700 3 client.4642.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:09:10.346 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.346+0000 7f102e59c700 1 --2- 172.21.15.59:0/33975824 >> v2:172.21.15.110:6800/115870 conn(0x7f1010068910 0x7f101006adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:10.346 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.346+0000 7f101e7fc700 20 client.4642.objecter dump_active .. 0 homeless 2023-09-30T02:09:10.346 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.346+0000 7f102e59c700 10 monclient: get_auth_request con 0x7f1010068910 auth_method 0 2023-09-30T02:09:10.347 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.347+0000 7f101e7fc700 10 client.4642.objecter ms_handle_connect 0x7f1010068910 2023-09-30T02:09:10.347 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.347+0000 7f102e59c700 1 --2- 172.21.15.59:0/33975824 >> v2:172.21.15.110:6800/115870 conn(0x7f1010068910 0x7f101006adc0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1020003a30 tx=0x7f1020007ea0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:09:10.559 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.560+0000 7f101e7fc700 1 -- 172.21.15.59:0/33975824 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f10240535a0 con 0x7f10300a5f40 2023-09-30T02:09:10.560 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.560+0000 7f101e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:09:10.560 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.560+0000 7f101e7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:09:10.811 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.812+0000 7f1034f1a700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:09:10.811 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.812+0000 7f1034f1a700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:09:10.812 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.812+0000 7f1034f1a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:09:10.812 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.812+0000 7f1034f1a700 1 -- 172.21.15.59:0/33975824 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f10300a79e0 con 0x7f10300a5f40 2023-09-30T02:09:10.816 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.816+0000 7f101e7fc700 1 -- 172.21.15.59:0/33975824 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f1024056bc0 con 0x7f10300a5f40 2023-09-30T02:09:10.816 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.816+0000 7f101e7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:09:10.816 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.816+0000 7f101e7fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:09:10.816 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:09:10.816 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:09:10.819 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.819+0000 7f1034f1a700 10 librados: watch_flush enter 2023-09-30T02:09:10.819 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.819+0000 7f1034f1a700 10 librados: watch_flush exit 2023-09-30T02:09:10.819 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.819+0000 7f1034f1a700 20 client.4642.objecter shutdown clearing up homeless session... 2023-09-30T02:09:10.819 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.819+0000 7f1034f1a700 10 client.4642.objecter successfully canceled tick 2023-09-30T02:09:10.819 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.819+0000 7f1034f1a700 1 -- 172.21.15.59:0/33975824 >> v2:172.21.15.110:6800/115870 conn(0x7f1010068910 msgr2=0x7f101006adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:10.820 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.819+0000 7f1034f1a700 1 --2- 172.21.15.59:0/33975824 >> v2:172.21.15.110:6800/115870 conn(0x7f1010068910 0x7f101006adc0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1020003a30 tx=0x7f1020007ea0 comp rx=0 tx=0).stop 2023-09-30T02:09:10.820 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.819+0000 7f1034f1a700 10 monclient: shutdown 2023-09-30T02:09:10.820 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.819+0000 7f1034f1a700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:09:10.820 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.819+0000 7f1034f1a700 1 -- 172.21.15.59:0/33975824 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f10300a5f40 msgr2=0x7f1030151450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:10.820 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.819+0000 7f1034f1a700 1 --2- 172.21.15.59:0/33975824 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f10300a5f40 0x7f1030151450 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f1024006fd0 tx=0x7f1024016000 comp rx=0 tx=0).stop 2023-09-30T02:09:10.820 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.819+0000 7f1034f1a700 1 -- 172.21.15.59:0/33975824 shutdown_connections 2023-09-30T02:09:10.820 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.819+0000 7f1034f1a700 1 --2- 172.21.15.59:0/33975824 >> v2:172.21.15.110:6800/115870 conn(0x7f1010068910 0x7f101006adc0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:10.820 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.819+0000 7f1034f1a700 1 --2- 172.21.15.59:0/33975824 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1030142290 0x7f1030151990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:10.820 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.819+0000 7f1034f1a700 1 --2- 172.21.15.59:0/33975824 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f10300a5b70 0x7f10300a80e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:10.820 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.819+0000 7f1034f1a700 1 --2- 172.21.15.59:0/33975824 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f10300a5f40 0x7f1030151450 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:10.820 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.819+0000 7f1034f1a700 1 -- 172.21.15.59:0/33975824 >> 172.21.15.59:0/33975824 conn(0x7f10300631f0 msgr2=0x7f10300afdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:09:10.821 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.819+0000 7f1034f1a700 1 -- 172.21.15.59:0/33975824 shutdown_connections 2023-09-30T02:09:10.821 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.819+0000 7f1034f1a700 1 -- 172.21.15.59:0/33975824 wait complete. 2023-09-30T02:09:10.821 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:10.819+0000 7f1034f1a700 1 librados: shutdown 2023-09-30T02:09:13.833 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:09:13.976 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:13.976+0000 7f8bacc93700 10 monclient: get_monmap_and_config 2023-09-30T02:09:13.976 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:13.976+0000 7f8bacc93700 10 monclient: build_initial_monmap 2023-09-30T02:09:13.976 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:13.976+0000 7f8bacc93700 10 monclient: monmap: 2023-09-30T02:09:13.976 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:09:13.976 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:13.976 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:09:13.976691+0000 2023-09-30T02:09:13.976 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:09:13.976691+0000 2023-09-30T02:09:13.976 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:09:13.976 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:13.977 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:09:13.977 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:09:13.977 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:09:13.977 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:09:13.977 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:09:13.977 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:13.977 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:13.976+0000 7f8bacc93700 1 Processor -- start 2023-09-30T02:09:13.977 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:13.977+0000 7f8bacc93700 1 -- start start 2023-09-30T02:09:13.977 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:13.977+0000 7f8bacc93700 10 monclient: init 2023-09-30T02:09:13.978 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:13.977+0000 7f8bacc93700 10 monclient: _reopen_session rank -1 2023-09-30T02:09:13.978 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:13.977+0000 7f8bacc93700 10 monclient: _add_conns ranks=[2,4,3,0,1] 2023-09-30T02:09:13.978 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:13.977+0000 7f8bacc93700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8ba80a5b70 0x7f8ba80b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:13.979 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:13.977+0000 7f8bacc93700 10 monclient(hunting): picked mon.noname-b con 0x7f8ba80a5b70 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:09:13.979 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:13.977+0000 7f8bacc93700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f8ba80a5f40 0x7f8ba8141b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:13.979 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:13.977+0000 7f8bacc93700 10 monclient(hunting): picked mon.noname-c con 0x7f8ba80a5f40 addr v2:172.21.15.110:3300/0 2023-09-30T02:09:13.979 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:13.977+0000 7f8bacc93700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f8ba81420c0 0x7f8ba813e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:13.979 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:13.977+0000 7f8bacc93700 10 monclient(hunting): picked mon.noname-a con 0x7f8ba81420c0 addr v2:172.21.15.59:3300/0 2023-09-30T02:09:13.979 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:13.977+0000 7f8bacc93700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:13.979 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:13.977+0000 7f8bacc93700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ba80a63b0 con 0x7f8ba81420c0 2023-09-30T02:09:13.979 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:13.977+0000 7f8bacc93700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:13.979 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:13.977+0000 7f8bacc93700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ba80a75a0 con 0x7f8ba80a5f40 2023-09-30T02:09:13.979 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:13.977+0000 7f8bacc93700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:13.979 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:13.977+0000 7f8bacc93700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ba80af3f0 con 0x7f8ba80a5b70 2023-09-30T02:09:13.979 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:13.977+0000 7f8bacc93700 10 monclient(hunting): _renew_subs 2023-09-30T02:09:13.980 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:13.977+0000 7f8bacc93700 10 monclient(hunting): authenticate will time out at 1999.279971s 2023-09-30T02:09:13.994 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:13.995+0000 7f8ba5d9b700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f8ba80a5f40 0x7f8ba8141b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:13.994 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:13.995+0000 7f8ba5d9b700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f8ba80a5f40 0x7f8ba8141b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:36890/0 (socket says 172.21.15.59:36890) 2023-09-30T02:09:13.995 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:13.995+0000 7f8ba5d9b700 1 -- 172.21.15.59:0/1280362343 learned_addr learned my addr 172.21.15.59:0/1280362343 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:09:13.995 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:13.995+0000 7f8ba5d9b700 10 monclient(hunting): get_auth_request con 0x7f8ba80a5f40 auth_method 0 2023-09-30T02:09:13.995 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:13.995+0000 7f8ba5d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:09:13.995 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:13.995+0000 7f8ba5d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:13.995 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:13.995+0000 7f8ba5d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:09:13.995 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:13.996+0000 7f8ba5d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:09:13.995 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:13.996+0000 7f8ba5d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:09:13.996 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:13.996+0000 7f8ba5d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:09:13.996 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:13.997+0000 7f8ba6d9d700 1 --2- 172.21.15.59:0/1280362343 >> v2:172.21.15.59:3300/0 conn(0x7f8ba81420c0 0x7f8ba813e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:13.998 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:13.998+0000 7f8ba5d9b700 10 monclient(hunting): handle_auth_done global_id 4645 payload 274 2023-09-30T02:09:13.999 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:13.998+0000 7f8ba5d9b700 1 -- 172.21.15.59:0/1280362343 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8ba80a5b70 msgr2=0x7f8ba80b1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:09:13.999 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:13.998+0000 7f8ba5d9b700 1 --2- 172.21.15.59:0/1280362343 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8ba80a5b70 0x7f8ba80b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:13.999 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:13.998+0000 7f8ba5d9b700 1 -- 172.21.15.59:0/1280362343 >> v2:172.21.15.59:3300/0 conn(0x7f8ba81420c0 msgr2=0x7f8ba813e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:09:13.999 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:13.998+0000 7f8ba5d9b700 1 --2- 172.21.15.59:0/1280362343 >> v2:172.21.15.59:3300/0 conn(0x7f8ba81420c0 0x7f8ba813e230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:13.999 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:13.998+0000 7f8ba5d9b700 10 monclient: _finish_hunting 0 2023-09-30T02:09:13.999 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:13.998+0000 7f8ba5d9b700 1 monclient: found mon.noname-c 2023-09-30T02:09:13.999 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:13.999+0000 7f8ba5d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:09:13.999 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:13.999+0000 7f8ba5d9b700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3300/0 2023-09-30T02:09:13.999 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:13.999+0000 7f8ba5d9b700 1 -- 172.21.15.59:0/1280362343 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ba8142500 con 0x7f8ba80a5f40 2023-09-30T02:09:14.000 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.001+0000 7f8ba5d9b700 1 --2- 172.21.15.59:0/1280362343 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8ba80a5f40 0x7f8ba8141b80 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f8ba8069af0 tx=0x7f8b9c009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=180beb660ce797cd server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:09:14.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.008+0000 7f8ba559a700 1 -- 172.21.15.59:0/1280362343 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b9c00b860 con 0x7f8ba80a5f40 2023-09-30T02:09:14.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.008+0000 7f8ba559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:09:14.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.008+0000 7f8ba559a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:09:14.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.008+0000 7f8ba559a700 10 monclient: dump: 2023-09-30T02:09:14.008 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:09:14.008 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:14.009 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:14.009 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:14.009 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:09:14.009 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:14.009 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:09:14.009 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:09:14.009 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:09:14.009 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:14.009 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.008+0000 7f8ba559a700 10 monclient: _finish_auth 0 2023-09-30T02:09:14.009 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.008+0000 7f8ba559a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:09:14.009 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.008+0000 7f8ba559a700 1 -- 172.21.15.59:0/1280362343 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b9c00b9c0 con 0x7f8ba80a5f40 2023-09-30T02:09:14.010 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.008+0000 7f8ba559a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:09:14.010 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.008+0000 7f8ba559a700 1 -- 172.21.15.59:0/1280362343 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b9c00bd60 con 0x7f8ba80a5f40 2023-09-30T02:09:14.010 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.008+0000 7f8bacc93700 5 monclient: authenticate success, global_id 4645 2023-09-30T02:09:14.010 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.008+0000 7f8bacc93700 10 monclient: get_monmap_and_config success 2023-09-30T02:09:14.010 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.008+0000 7f8bacc93700 4 set_mon_vals no callback set 2023-09-30T02:09:14.010 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.008+0000 7f8ba559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:09:14.010 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.008+0000 7f8ba559a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:09:14.010 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.008+0000 7f8ba559a700 10 monclient: dump: 2023-09-30T02:09:14.010 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:09:14.010 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:14.010 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:14.011 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:14.011 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:09:14.011 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:14.011 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:09:14.011 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:09:14.011 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:09:14.011 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:14.014 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.009+0000 7f8bacc93700 10 monclient: shutdown 2023-09-30T02:09:14.014 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.009+0000 7f8bacc93700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:09:14.014 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.009+0000 7f8bacc93700 1 -- 172.21.15.59:0/1280362343 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8ba80a5f40 msgr2=0x7f8ba8141b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:14.014 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.009+0000 7f8bacc93700 1 --2- 172.21.15.59:0/1280362343 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8ba80a5f40 0x7f8ba8141b80 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f8ba8069af0 tx=0x7f8b9c009250 comp rx=0 tx=0).stop 2023-09-30T02:09:14.014 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.009+0000 7f8bacc93700 1 -- 172.21.15.59:0/1280362343 shutdown_connections 2023-09-30T02:09:14.014 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.009+0000 7f8bacc93700 1 --2- 172.21.15.59:0/1280362343 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8ba80a5b70 0x7f8ba80b1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:14.014 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.009+0000 7f8bacc93700 1 --2- 172.21.15.59:0/1280362343 >> v2:172.21.15.59:3300/0 conn(0x7f8ba81420c0 0x7f8ba813e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:14.014 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.009+0000 7f8bacc93700 1 --2- 172.21.15.59:0/1280362343 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8ba80a5f40 0x7f8ba8141b80 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:14.015 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.009+0000 7f8bacc93700 1 -- 172.21.15.59:0/1280362343 >> 172.21.15.59:0/1280362343 conn(0x7f8ba80631f0 msgr2=0x7f8ba80635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:09:14.015 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.009+0000 7f8bacc93700 1 -- 172.21.15.59:0/1280362343 shutdown_connections 2023-09-30T02:09:14.015 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.010+0000 7f8bacc93700 1 -- 172.21.15.59:0/1280362343 wait complete. 2023-09-30T02:09:14.015 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.010+0000 7f8bacc93700 10 monclient: build_initial_monmap 2023-09-30T02:09:14.015 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.010+0000 7f8bacc93700 10 monclient: monmap: 2023-09-30T02:09:14.015 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:09:14.015 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:09:14.015 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:09:14.015 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:09:14.015 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:09:14.015 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:09:14.015 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:09:14.016 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:09:14.016 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:09:14.016 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:14.016 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.011+0000 7f8bacc93700 1 librados: starting msgr at 2023-09-30T02:09:14.016 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.011+0000 7f8bacc93700 1 librados: starting objecter 2023-09-30T02:09:14.016 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.011+0000 7f8bacc93700 1 Processor -- start 2023-09-30T02:09:14.016 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.011+0000 7f8bacc93700 1 -- start start 2023-09-30T02:09:14.016 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.011+0000 7f8bacc93700 1 librados: setting wanted keys 2023-09-30T02:09:14.016 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.011+0000 7f8bacc93700 1 librados: calling monclient init 2023-09-30T02:09:14.016 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.011+0000 7f8bacc93700 10 monclient: init 2023-09-30T02:09:14.016 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.012+0000 7f8bacc93700 10 monclient: _reopen_session rank -1 2023-09-30T02:09:14.016 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.012+0000 7f8bacc93700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T02:09:14.017 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.012+0000 7f8bacc93700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ba80a5b70 0x7f8ba80aa2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:14.017 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.012+0000 7f8bacc93700 10 monclient(hunting): picked mon.noname-a con 0x7f8ba80a5b70 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:09:14.017 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.012+0000 7f8bacc93700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8ba80a5f40 0x7f8ba80aa8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:14.017 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.012+0000 7f8bacc93700 10 monclient(hunting): picked mon.noname-c con 0x7f8ba80a5f40 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:09:14.017 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.012+0000 7f8bacc93700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8ba81420c0 0x7f8ba80ade80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:14.017 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.012+0000 7f8bacc93700 10 monclient(hunting): picked mon.noname-b con 0x7f8ba81420c0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:09:14.017 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.012+0000 7f8bacc93700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:14.017 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.012+0000 7f8bacc93700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ba8137920 con 0x7f8ba80a5b70 2023-09-30T02:09:14.017 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.012+0000 7f8bacc93700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:14.018 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.012+0000 7f8bacc93700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ba80ab200 con 0x7f8ba81420c0 2023-09-30T02:09:14.018 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.012+0000 7f8bacc93700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:14.018 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.012+0000 7f8bacc93700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ba80a94c0 con 0x7f8ba80a5f40 2023-09-30T02:09:14.018 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.012+0000 7f8bacc93700 10 monclient(hunting): _renew_subs 2023-09-30T02:09:14.018 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.012+0000 7f8bacc93700 10 monclient(hunting): authenticate will time out at 1999.314148s 2023-09-30T02:09:14.018 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.012+0000 7f8ba659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ba80a5b70 0x7f8ba80aa2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:14.018 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.013+0000 7f8ba659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ba80a5b70 0x7f8ba80aa2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:37678/0 (socket says 172.21.15.59:37678) 2023-09-30T02:09:14.018 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.013+0000 7f8ba659c700 1 -- 172.21.15.59:0/2475725300 learned_addr learned my addr 172.21.15.59:0/2475725300 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:09:14.018 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.013+0000 7f8ba659c700 10 monclient(hunting): get_auth_request con 0x7f8ba80a5b70 auth_method 0 2023-09-30T02:09:14.018 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.013+0000 7f8ba659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:09:14.018 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.013+0000 7f8ba659c700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:14.019 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.013+0000 7f8ba659c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:09:14.019 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.013+0000 7f8ba6d9d700 1 --2- 172.21.15.59:0/2475725300 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8ba81420c0 0x7f8ba80ade80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:14.019 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.013+0000 7f8ba6d9d700 10 monclient(hunting): get_auth_request con 0x7f8ba81420c0 auth_method 0 2023-09-30T02:09:14.019 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.013+0000 7f8ba6d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:09:14.019 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.013+0000 7f8ba6d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:14.019 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.013+0000 7f8ba6d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:09:14.019 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.014+0000 7f8ba659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:09:14.019 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.014+0000 7f8ba659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:09:14.019 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.014+0000 7f8ba659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:09:14.019 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.014+0000 7f8ba6d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:09:14.019 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.014+0000 7f8ba6d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:09:14.019 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.014+0000 7f8ba6d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:09:14.020 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.017+0000 7f8ba5d9b700 1 --2- 172.21.15.59:0/2475725300 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8ba80a5f40 0x7f8ba80aa8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:14.020 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.017+0000 7f8ba6d9d700 10 monclient(hunting): handle_auth_done global_id 4648 payload 824 2023-09-30T02:09:14.021 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.018+0000 7f8ba6d9d700 1 -- 172.21.15.59:0/2475725300 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8ba80a5f40 msgr2=0x7f8ba80aa8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:14.021 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.018+0000 7f8ba6d9d700 1 --2- 172.21.15.59:0/2475725300 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8ba80a5f40 0x7f8ba80aa8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:14.021 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.018+0000 7f8ba6d9d700 1 -- 172.21.15.59:0/2475725300 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ba80a5b70 msgr2=0x7f8ba80aa2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:14.021 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.018+0000 7f8ba6d9d700 1 --2- 172.21.15.59:0/2475725300 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ba80a5b70 0x7f8ba80aa2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:14.021 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.018+0000 7f8ba6d9d700 10 monclient: _finish_hunting 0 2023-09-30T02:09:14.021 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.018+0000 7f8ba6d9d700 1 monclient: found mon.noname-b 2023-09-30T02:09:14.021 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.018+0000 7f8ba6d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:09:14.021 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.018+0000 7f8ba6d9d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:09:14.021 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.018+0000 7f8ba6d9d700 1 -- 172.21.15.59:0/2475725300 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b9c0094f0 con 0x7f8ba81420c0 2023-09-30T02:09:14.021 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.020+0000 7f8ba6d9d700 1 --2- 172.21.15.59:0/2475725300 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8ba81420c0 0x7f8ba80ade80 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f8ba000cce0 tx=0x7f8ba0007710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:09:14.021 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.020+0000 7f8b967fc700 10 client.?.objecter ms_handle_connect 0x7f8ba81420c0 2023-09-30T02:09:14.021 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.021+0000 7f8b967fc700 10 client.?.objecter resend_mon_ops 2023-09-30T02:09:14.025 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.024+0000 7f8b967fc700 1 -- 172.21.15.59:0/2475725300 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ba000a8d0 con 0x7f8ba81420c0 2023-09-30T02:09:14.025 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.024+0000 7f8b967fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:09:14.025 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.024+0000 7f8b967fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:09:14.025 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.024+0000 7f8b967fc700 10 monclient: dump: 2023-09-30T02:09:14.026 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:09:14.026 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:14.026 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:14.026 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:14.026 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:09:14.027 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:14.027 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:09:14.027 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:09:14.027 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:09:14.027 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:14.027 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.024+0000 7f8b967fc700 10 monclient: _finish_auth 0 2023-09-30T02:09:14.027 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.024+0000 7f8b967fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:09:14.027 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.024+0000 7f8b967fc700 1 -- 172.21.15.59:0/2475725300 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ba000aa30 con 0x7f8ba81420c0 2023-09-30T02:09:14.027 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.024+0000 7f8bacc93700 5 monclient: authenticate success, global_id 4648 2023-09-30T02:09:14.028 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.024+0000 7f8bacc93700 10 monclient: _renew_subs 2023-09-30T02:09:14.028 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.024+0000 7f8bacc93700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:09:14.028 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.024+0000 7f8bacc93700 1 -- 172.21.15.59:0/2475725300 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ba80aae30 con 0x7f8ba81420c0 2023-09-30T02:09:14.028 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.024+0000 7f8b967fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:09:14.028 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.024+0000 7f8b967fc700 1 -- 172.21.15.59:0/2475725300 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ba000add0 con 0x7f8ba81420c0 2023-09-30T02:09:14.028 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.024+0000 7f8b967fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:09:14.028 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.024+0000 7f8b967fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:09:14.028 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.024+0000 7f8b967fc700 10 monclient: dump: 2023-09-30T02:09:14.028 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:09:14.028 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:14.028 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:14.029 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:14.029 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:09:14.029 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:14.029 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:09:14.029 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:09:14.029 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:09:14.029 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:14.029 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.024+0000 7f8ba4d99700 4 set_mon_vals no callback set 2023-09-30T02:09:14.029 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.024+0000 7f8bacc93700 10 client.4648.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:09:14.029 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.024+0000 7f8bacc93700 10 monclient: _renew_subs 2023-09-30T02:09:14.029 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.024+0000 7f8bacc93700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:09:14.030 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.024+0000 7f8bacc93700 1 -- 172.21.15.59:0/2475725300 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ba80ae580 con 0x7f8ba81420c0 2023-09-30T02:09:14.030 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.024+0000 7f8bacc93700 1 librados: init done 2023-09-30T02:09:14.030 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.025+0000 7f8bacc93700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:09:14.030 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.025+0000 7f8bacc93700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:09:14.030 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.025+0000 7f8bacc93700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:09:14.030 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.025+0000 7f8bacc93700 1 -- 172.21.15.59:0/2475725300 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ba8064a00 con 0x7f8ba81420c0 2023-09-30T02:09:14.069 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.069+0000 7f8b967fc700 1 -- 172.21.15.59:0/2475725300 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f8ba0003020 con 0x7f8ba81420c0 2023-09-30T02:09:14.069 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.070+0000 7f8b967fc700 1 --2- 172.21.15.59:0/2475725300 >> v2:172.21.15.110:6800/115870 conn(0x7f8b88068910 0x7f8b8806adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:14.070 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.070+0000 7f8b967fc700 1 -- 172.21.15.59:0/2475725300 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f8ba0013070 con 0x7f8ba81420c0 2023-09-30T02:09:14.070 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.070+0000 7f8b967fc700 10 client.4648.objecter ms_dispatch 0x7f8ba8001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:09:14.070 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.070+0000 7f8b967fc700 3 client.4648.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:09:14.070 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.070+0000 7f8b967fc700 3 client.4648.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:09:14.070 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.070+0000 7f8b967fc700 20 client.4648.objecter dump_active .. 0 homeless 2023-09-30T02:09:14.070 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.070+0000 7f8ba659c700 1 --2- 172.21.15.59:0/2475725300 >> v2:172.21.15.110:6800/115870 conn(0x7f8b88068910 0x7f8b8806adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:14.070 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.070+0000 7f8ba659c700 10 monclient: get_auth_request con 0x7f8b88068910 auth_method 0 2023-09-30T02:09:14.071 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.071+0000 7f8ba659c700 1 --2- 172.21.15.59:0/2475725300 >> v2:172.21.15.110:6800/115870 conn(0x7f8b88068910 0x7f8b8806adc0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f8b98009ca0 tx=0x7f8b98009250 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:09:14.071 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.071+0000 7f8b967fc700 10 client.4648.objecter ms_handle_connect 0x7f8b88068910 2023-09-30T02:09:14.287 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.287+0000 7f8b967fc700 1 -- 172.21.15.59:0/2475725300 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8ba0052090 con 0x7f8ba81420c0 2023-09-30T02:09:14.287 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.287+0000 7f8b967fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:09:14.288 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.287+0000 7f8b967fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:09:14.551 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.551+0000 7f8bacc93700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:09:14.552 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.551+0000 7f8bacc93700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:09:14.552 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.551+0000 7f8bacc93700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:09:14.552 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.551+0000 7f8bacc93700 1 -- 172.21.15.59:0/2475725300 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f8ba806a720 con 0x7f8ba81420c0 2023-09-30T02:09:14.556 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.556+0000 7f8b967fc700 1 -- 172.21.15.59:0/2475725300 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f8ba00556b0 con 0x7f8ba81420c0 2023-09-30T02:09:14.556 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.556+0000 7f8b967fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:09:14.556 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.556+0000 7f8b967fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:09:14.556 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:09:14.556 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:09:14.558 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.557+0000 7f8bacc93700 10 librados: watch_flush enter 2023-09-30T02:09:14.559 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.558+0000 7f8bacc93700 10 librados: watch_flush exit 2023-09-30T02:09:14.559 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.558+0000 7f8bacc93700 20 client.4648.objecter shutdown clearing up homeless session... 2023-09-30T02:09:14.559 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.558+0000 7f8bacc93700 10 client.4648.objecter successfully canceled tick 2023-09-30T02:09:14.559 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.558+0000 7f8bacc93700 1 -- 172.21.15.59:0/2475725300 >> v2:172.21.15.110:6800/115870 conn(0x7f8b88068910 msgr2=0x7f8b8806adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:14.559 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.558+0000 7f8bacc93700 1 --2- 172.21.15.59:0/2475725300 >> v2:172.21.15.110:6800/115870 conn(0x7f8b88068910 0x7f8b8806adc0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f8b98009ca0 tx=0x7f8b98009250 comp rx=0 tx=0).stop 2023-09-30T02:09:14.559 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.558+0000 7f8bacc93700 10 monclient: shutdown 2023-09-30T02:09:14.559 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.558+0000 7f8bacc93700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:09:14.559 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.558+0000 7f8bacc93700 1 -- 172.21.15.59:0/2475725300 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8ba81420c0 msgr2=0x7f8ba80ade80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:14.559 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.558+0000 7f8bacc93700 1 --2- 172.21.15.59:0/2475725300 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8ba81420c0 0x7f8ba80ade80 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f8ba000cce0 tx=0x7f8ba0007710 comp rx=0 tx=0).stop 2023-09-30T02:09:14.559 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.558+0000 7f8bacc93700 1 -- 172.21.15.59:0/2475725300 shutdown_connections 2023-09-30T02:09:14.559 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.558+0000 7f8bacc93700 1 --2- 172.21.15.59:0/2475725300 >> v2:172.21.15.110:6800/115870 conn(0x7f8b88068910 0x7f8b8806adc0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:14.560 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.558+0000 7f8bacc93700 1 --2- 172.21.15.59:0/2475725300 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8ba81420c0 0x7f8ba80ade80 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:14.560 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.558+0000 7f8bacc93700 1 --2- 172.21.15.59:0/2475725300 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ba80a5b70 0x7f8ba80aa2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:14.560 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.558+0000 7f8bacc93700 1 --2- 172.21.15.59:0/2475725300 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8ba80a5f40 0x7f8ba80aa8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:14.560 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.558+0000 7f8bacc93700 1 -- 172.21.15.59:0/2475725300 >> 172.21.15.59:0/2475725300 conn(0x7f8ba80631f0 msgr2=0x7f8ba80af530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:09:14.560 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.558+0000 7f8bacc93700 1 -- 172.21.15.59:0/2475725300 shutdown_connections 2023-09-30T02:09:14.560 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.558+0000 7f8bacc93700 1 -- 172.21.15.59:0/2475725300 wait complete. 2023-09-30T02:09:14.560 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:14.558+0000 7f8bacc93700 1 librados: shutdown 2023-09-30T02:09:17.572 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:09:17.717 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.717+0000 7ff5ef6b9700 10 monclient: get_monmap_and_config 2023-09-30T02:09:17.718 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.717+0000 7ff5ef6b9700 10 monclient: build_initial_monmap 2023-09-30T02:09:17.718 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.717+0000 7ff5ef6b9700 10 monclient: monmap: 2023-09-30T02:09:17.718 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:09:17.718 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:17.718 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:09:17.718786+0000 2023-09-30T02:09:17.718 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:09:17.718786+0000 2023-09-30T02:09:17.718 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:09:17.718 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:17.718 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:09:17.718 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:09:17.719 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:09:17.719 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:09:17.719 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:09:17.719 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:17.720 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.718+0000 7ff5ef6b9700 1 Processor -- start 2023-09-30T02:09:17.720 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.718+0000 7ff5ef6b9700 1 -- start start 2023-09-30T02:09:17.720 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.718+0000 7ff5ef6b9700 10 monclient: init 2023-09-30T02:09:17.720 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.718+0000 7ff5ef6b9700 10 monclient: _reopen_session rank -1 2023-09-30T02:09:17.720 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.718+0000 7ff5ef6b9700 10 monclient: _add_conns ranks=[3,2,1,0,4] 2023-09-30T02:09:17.720 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.718+0000 7ff5ef6b9700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7ff5e80a5b70 0x7ff5e80b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:17.720 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.718+0000 7ff5ef6b9700 10 monclient(hunting): picked mon.noname-a con 0x7ff5e80a5b70 addr v2:172.21.15.59:3300/0 2023-09-30T02:09:17.721 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.718+0000 7ff5ef6b9700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff5e80a5f40 0x7ff5e8141b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:17.721 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.718+0000 7ff5ef6b9700 10 monclient(hunting): picked mon.noname-b con 0x7ff5e80a5f40 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:09:17.721 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.718+0000 7ff5ef6b9700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff5e81420c0 addr v1:172.21.15.110:6789/0 2023-09-30T02:09:17.721 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.718+0000 7ff5ef6b9700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5e80a63b0 con 0x7ff5e81420c0 2023-09-30T02:09:17.721 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.719+0000 7ff5ef6b9700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:17.721 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.719+0000 7ff5ef6b9700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5e80a75a0 con 0x7ff5e80a5b70 2023-09-30T02:09:17.721 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.719+0000 7ff5ef6b9700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:17.721 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.719+0000 7ff5ef6b9700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5e80af3f0 con 0x7ff5e80a5f40 2023-09-30T02:09:17.721 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.719+0000 7ff5ef6b9700 10 monclient(hunting): _renew_subs 2023-09-30T02:09:17.721 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.719+0000 7ff5ef6b9700 10 monclient(hunting): authenticate will time out at 2003.022027s 2023-09-30T02:09:17.721 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.719+0000 7ff5ecc54700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff5e80a5f40 0x7ff5e8141b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:17.721 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.719+0000 7ff5edc56700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7ff5e81420c0 0x7ff5e813e230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.59:42756/0 (socket says 172.21.15.59:42756) 2023-09-30T02:09:17.722 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.719+0000 7ff5edc56700 1 -- 172.21.15.59:0/803474563 learned_addr learned my addr 172.21.15.59:0/803474563 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:09:17.722 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.720+0000 7ff5ecc54700 10 monclient(hunting): get_auth_request con 0x7ff5e80a5f40 auth_method 0 2023-09-30T02:09:17.722 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.720+0000 7ff5ecc54700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:09:17.722 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.720+0000 7ff5ecc54700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:17.722 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.720+0000 7ff5ecc54700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:09:17.722 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.720+0000 7ff5ecc54700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:09:17.722 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.720+0000 7ff5ecc54700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:09:17.722 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.720+0000 7ff5ecc54700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:09:17.723 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.723+0000 7ff5ecc54700 10 monclient(hunting): handle_auth_done global_id 4562 payload 274 2023-09-30T02:09:17.723 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.723+0000 7ff5ecc54700 1 -- 172.21.15.59:0/803474563 >> v2:172.21.15.59:3300/0 conn(0x7ff5e80a5b70 msgr2=0x7ff5e80b1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:09:17.723 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.723+0000 7ff5ecc54700 1 --2- 172.21.15.59:0/803474563 >> v2:172.21.15.59:3300/0 conn(0x7ff5e80a5b70 0x7ff5e80b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:17.723 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.723+0000 7ff5ecc54700 1 -- 172.21.15.59:0/803474563 >> v1:172.21.15.110:6789/0 conn(0x7ff5e81420c0 legacy=0x7ff5e813e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:17.723 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.723+0000 7ff5ecc54700 10 monclient: _finish_hunting 0 2023-09-30T02:09:17.724 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.723+0000 7ff5ecc54700 1 monclient: found mon.noname-b 2023-09-30T02:09:17.724 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.723+0000 7ff5ecc54700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:09:17.724 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.723+0000 7ff5ecc54700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.59:3301/0 2023-09-30T02:09:17.724 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.723+0000 7ff5ecc54700 1 -- 172.21.15.59:0/803474563 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5e8142500 con 0x7ff5e80a5f40 2023-09-30T02:09:17.725 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.725+0000 7ff5ecc54700 1 --2- 172.21.15.59:0/803474563 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff5e80a5f40 0x7ff5e8141b80 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff5dc0098e0 tx=0x7ff5dc0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90f3945e0b575d66 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:09:17.731 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.731+0000 7ff5e7fff700 1 -- 172.21.15.59:0/803474563 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5dc009e80 con 0x7ff5e80a5f40 2023-09-30T02:09:17.731 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.731+0000 7ff5e7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:09:17.731 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.731+0000 7ff5e7fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:09:17.731 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.731+0000 7ff5e7fff700 10 monclient: dump: 2023-09-30T02:09:17.731 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:09:17.732 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:17.732 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:17.732 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:17.732 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:09:17.732 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:17.732 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:09:17.732 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:09:17.732 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:09:17.732 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:17.732 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.731+0000 7ff5e7fff700 10 monclient: _finish_auth 0 2023-09-30T02:09:17.732 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.731+0000 7ff5e7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:09:17.732 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.731+0000 7ff5e7fff700 1 -- 172.21.15.59:0/803474563 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5dc00b860 con 0x7ff5e80a5f40 2023-09-30T02:09:17.733 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.731+0000 7ff5ef6b9700 5 monclient: authenticate success, global_id 4562 2023-09-30T02:09:17.733 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.731+0000 7ff5ef6b9700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:09:17.733 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.731+0000 7ff5e7fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:09:17.733 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.731+0000 7ff5ef6b9700 10 monclient: get_monmap_and_config success 2023-09-30T02:09:17.733 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.731+0000 7ff5e7fff700 1 -- 172.21.15.59:0/803474563 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5dc00bbe0 con 0x7ff5e80a5f40 2023-09-30T02:09:17.733 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.731+0000 7ff5e7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:09:17.733 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.731+0000 7ff5ef6b9700 4 set_mon_vals no callback set 2023-09-30T02:09:17.733 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.731+0000 7ff5e7fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:09:17.733 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.731+0000 7ff5e7fff700 10 monclient: dump: 2023-09-30T02:09:17.733 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:09:17.733 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:17.733 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:17.733 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:17.734 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:09:17.734 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:17.734 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:09:17.734 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:09:17.734 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:09:17.734 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:17.736 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.732+0000 7ff5ef6b9700 10 monclient: shutdown 2023-09-30T02:09:17.736 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.732+0000 7ff5ef6b9700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:09:17.736 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.732+0000 7ff5ef6b9700 1 -- 172.21.15.59:0/803474563 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff5e80a5f40 msgr2=0x7ff5e8141b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:17.736 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.732+0000 7ff5ef6b9700 1 --2- 172.21.15.59:0/803474563 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff5e80a5f40 0x7ff5e8141b80 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff5dc0098e0 tx=0x7ff5dc0091a0 comp rx=0 tx=0).stop 2023-09-30T02:09:17.736 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.732+0000 7ff5ef6b9700 1 -- 172.21.15.59:0/803474563 shutdown_connections 2023-09-30T02:09:17.736 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.732+0000 7ff5ef6b9700 1 --2- 172.21.15.59:0/803474563 >> v2:172.21.15.59:3300/0 conn(0x7ff5e80a5b70 0x7ff5e80b1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:17.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.732+0000 7ff5ef6b9700 1 --2- 172.21.15.59:0/803474563 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff5e80a5f40 0x7ff5e8141b80 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:17.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.732+0000 7ff5ef6b9700 1 -- 172.21.15.59:0/803474563 >> 172.21.15.59:0/803474563 conn(0x7ff5e80631f0 msgr2=0x7ff5e80635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:09:17.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.732+0000 7ff5ef6b9700 1 -- 172.21.15.59:0/803474563 shutdown_connections 2023-09-30T02:09:17.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.732+0000 7ff5ef6b9700 1 -- 172.21.15.59:0/803474563 wait complete. 2023-09-30T02:09:17.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.733+0000 7ff5ef6b9700 10 monclient: build_initial_monmap 2023-09-30T02:09:17.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.733+0000 7ff5ef6b9700 10 monclient: monmap: 2023-09-30T02:09:17.737 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:09:17.737 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:09:17.737 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:09:17.737 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:09:17.737 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:09:17.737 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:09:17.737 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:09:17.738 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:09:17.738 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:09:17.738 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:17.738 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.733+0000 7ff5ef6b9700 1 librados: starting msgr at 2023-09-30T02:09:17.738 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.733+0000 7ff5ef6b9700 1 librados: starting objecter 2023-09-30T02:09:17.738 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.733+0000 7ff5ef6b9700 1 Processor -- start 2023-09-30T02:09:17.738 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.733+0000 7ff5ef6b9700 1 -- start start 2023-09-30T02:09:17.738 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.733+0000 7ff5ef6b9700 1 librados: setting wanted keys 2023-09-30T02:09:17.738 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.733+0000 7ff5ef6b9700 1 librados: calling monclient init 2023-09-30T02:09:17.738 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.733+0000 7ff5ef6b9700 10 monclient: init 2023-09-30T02:09:17.738 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.734+0000 7ff5ef6b9700 10 monclient: _reopen_session rank -1 2023-09-30T02:09:17.738 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.734+0000 7ff5ef6b9700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T02:09:17.739 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.734+0000 7ff5ef6b9700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff5e80a5b70 0x7ff5e80b5420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:17.739 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.734+0000 7ff5ef6b9700 10 monclient(hunting): picked mon.noname-a con 0x7ff5e80a5b70 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:09:17.739 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.734+0000 7ff5ef6b9700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff5e80a5f40 0x7ff5e80b2880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:17.739 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.734+0000 7ff5ef6b9700 10 monclient(hunting): picked mon.noname-b con 0x7ff5e80a5f40 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:09:17.739 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.734+0000 7ff5ef6b9700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff5e81420c0 0x7ff5e80adc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:17.739 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.734+0000 7ff5ef6b9700 10 monclient(hunting): picked mon.noname-c con 0x7ff5e81420c0 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:09:17.739 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.734+0000 7ff5ef6b9700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:17.739 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.734+0000 7ff5ef6b9700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5e80a81c0 con 0x7ff5e80a5b70 2023-09-30T02:09:17.739 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.734+0000 7ff5ef6b9700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:17.739 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.734+0000 7ff5ef6b9700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5e80a9210 con 0x7ff5e80a5f40 2023-09-30T02:09:17.739 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.734+0000 7ff5ef6b9700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:17.739 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.734+0000 7ff5ef6b9700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5e80a94f0 con 0x7ff5e81420c0 2023-09-30T02:09:17.740 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.734+0000 7ff5ef6b9700 10 monclient(hunting): _renew_subs 2023-09-30T02:09:17.740 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.734+0000 7ff5ef6b9700 10 monclient(hunting): authenticate will time out at 2003.037515s 2023-09-30T02:09:17.766 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.766+0000 7ff5edc56700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff5e81420c0 0x7ff5e80adc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:17.768 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.768+0000 7ff5ed455700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff5e80a5b70 0x7ff5e80b5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:17.772 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.772+0000 7ff5ed455700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff5e80a5b70 0x7ff5e80b5420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:58906/0 (socket says 172.21.15.59:58906) 2023-09-30T02:09:17.772 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.772+0000 7ff5ed455700 1 -- 172.21.15.59:0/1636149047 learned_addr learned my addr 172.21.15.59:0/1636149047 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:09:17.772 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.772+0000 7ff5ed455700 10 monclient(hunting): get_auth_request con 0x7ff5e80a5b70 auth_method 0 2023-09-30T02:09:17.773 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.772+0000 7ff5ed455700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:09:17.773 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.772+0000 7ff5ed455700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:17.773 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.772+0000 7ff5ed455700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:09:17.774 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.773+0000 7ff5ed455700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:09:17.774 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.773+0000 7ff5ed455700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:09:17.774 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.774+0000 7ff5ed455700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:09:17.781 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.780+0000 7ff5ed455700 10 monclient(hunting): handle_auth_done global_id 4635 payload 824 2023-09-30T02:09:17.781 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.780+0000 7ff5ed455700 1 -- 172.21.15.59:0/1636149047 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff5e81420c0 msgr2=0x7ff5e80adc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:17.781 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.780+0000 7ff5ed455700 1 --2- 172.21.15.59:0/1636149047 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff5e81420c0 0x7ff5e80adc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:17.781 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.780+0000 7ff5ed455700 1 -- 172.21.15.59:0/1636149047 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff5e80a5f40 msgr2=0x7ff5e80b2880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:17.781 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.780+0000 7ff5ed455700 1 --2- 172.21.15.59:0/1636149047 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff5e80a5f40 0x7ff5e80b2880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:17.781 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.780+0000 7ff5ed455700 10 monclient: _finish_hunting 0 2023-09-30T02:09:17.782 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.780+0000 7ff5ed455700 1 monclient: found mon.noname-a 2023-09-30T02:09:17.782 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.780+0000 7ff5ed455700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:09:17.782 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.780+0000 7ff5ed455700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.59:3300/0 2023-09-30T02:09:17.782 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.780+0000 7ff5ed455700 1 -- 172.21.15.59:0/1636149047 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5dc009420 con 0x7ff5e80a5b70 2023-09-30T02:09:17.784 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.784+0000 7ff5ed455700 1 --2- 172.21.15.59:0/1636149047 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff5e80a5b70 0x7ff5e80b5420 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ff5d800bc30 tx=0x7ff5d8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:09:17.784 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.784+0000 7ff5e57fa700 10 client.?.objecter ms_handle_connect 0x7ff5e80a5b70 2023-09-30T02:09:17.784 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.784+0000 7ff5e57fa700 10 client.?.objecter resend_mon_ops 2023-09-30T02:09:17.788 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.788+0000 7ff5e57fa700 1 -- 172.21.15.59:0/1636149047 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d80092e0 con 0x7ff5e80a5b70 2023-09-30T02:09:17.789 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.788+0000 7ff5e57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:09:17.789 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.788+0000 7ff5e57fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T02:09:17.789 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.788+0000 7ff5e57fa700 10 monclient: dump: 2023-09-30T02:09:17.789 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:09:17.789 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:17.789 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:17.789 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:17.789 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:09:17.789 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:17.789 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:09:17.789 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:09:17.789 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:09:17.790 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:17.790 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.788+0000 7ff5e57fa700 10 monclient: _finish_auth 0 2023-09-30T02:09:17.790 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.788+0000 7ff5e57fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:09:17.790 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.788+0000 7ff5ef6b9700 5 monclient: authenticate success, global_id 4635 2023-09-30T02:09:17.790 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.788+0000 7ff5ef6b9700 10 monclient: _renew_subs 2023-09-30T02:09:17.790 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.788+0000 7ff5ef6b9700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:09:17.790 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.788+0000 7ff5ef6b9700 1 -- 172.21.15.59:0/1636149047 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5e8137fe0 con 0x7ff5e80a5b70 2023-09-30T02:09:17.790 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.788+0000 7ff5ef6b9700 10 client.4635.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:09:17.790 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.788+0000 7ff5ef6b9700 10 monclient: _renew_subs 2023-09-30T02:09:17.790 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.788+0000 7ff5ef6b9700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:09:17.790 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.788+0000 7ff5ef6b9700 1 -- 172.21.15.59:0/1636149047 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5e80a7d60 con 0x7ff5e80a5b70 2023-09-30T02:09:17.791 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.788+0000 7ff5ef6b9700 1 librados: init done 2023-09-30T02:09:17.791 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.789+0000 7ff5ef6b9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:09:17.791 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.789+0000 7ff5ef6b9700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:09:17.791 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.789+0000 7ff5ef6b9700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:09:17.791 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.789+0000 7ff5ef6b9700 1 -- 172.21.15.59:0/1636149047 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5e80a8000 con 0x7ff5e80a5b70 2023-09-30T02:09:17.792 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.792+0000 7ff5e57fa700 1 -- 172.21.15.59:0/1636149047 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5d800c9b0 con 0x7ff5e80a5b70 2023-09-30T02:09:17.792 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.792+0000 7ff5e57fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:09:17.792 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:17.792+0000 7ff5e77fe700 4 set_mon_vals no callback set 2023-09-30T02:09:18.075 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:18.075+0000 7ff5e57fa700 1 -- 172.21.15.59:0/1636149047 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d800cb40 con 0x7ff5e80a5b70 2023-09-30T02:09:18.076 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:18.075+0000 7ff5e57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:09:18.076 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:18.075+0000 7ff5e57fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T02:09:18.076 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:18.075+0000 7ff5e57fa700 10 monclient: dump: 2023-09-30T02:09:18.076 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:09:18.076 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:18.076 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:18.076 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:18.076 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:09:18.076 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:18.076 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:09:18.076 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:09:18.076 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:09:18.077 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:18.091 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:18.091+0000 7ff5e57fa700 1 -- 172.21.15.59:0/1636149047 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7ff5d8012070 con 0x7ff5e80a5b70 2023-09-30T02:09:18.093 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:18.092+0000 7ff5e57fa700 1 --2- 172.21.15.59:0/1636149047 >> v2:172.21.15.110:6800/115870 conn(0x7ff5d4068890 0x7ff5d406ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:18.093 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:18.092+0000 7ff5e57fa700 1 -- 172.21.15.59:0/1636149047 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7ff5d8083aa0 con 0x7ff5e80a5b70 2023-09-30T02:09:18.093 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:18.092+0000 7ff5e57fa700 10 client.4635.objecter ms_dispatch 0x7ff5e8001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:09:18.093 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:18.092+0000 7ff5e57fa700 3 client.4635.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:09:18.093 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:18.092+0000 7ff5e57fa700 3 client.4635.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:09:18.093 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:18.092+0000 7ff5e57fa700 20 client.4635.objecter dump_active .. 0 homeless 2023-09-30T02:09:18.093 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:18.092+0000 7ff5e57fa700 1 -- 172.21.15.59:0/1636149047 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff5d80ae640 con 0x7ff5e80a5b70 2023-09-30T02:09:18.093 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:18.092+0000 7ff5e57fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:09:18.093 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:18.092+0000 7ff5e57fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:09:18.093 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:18.092+0000 7ff5ecc54700 1 --2- 172.21.15.59:0/1636149047 >> v2:172.21.15.110:6800/115870 conn(0x7ff5d4068890 0x7ff5d406ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:18.094 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:18.092+0000 7ff5ecc54700 10 monclient: get_auth_request con 0x7ff5d4068890 auth_method 0 2023-09-30T02:09:18.094 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:18.093+0000 7ff5e57fa700 10 client.4635.objecter ms_handle_connect 0x7ff5d4068890 2023-09-30T02:09:18.094 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:18.093+0000 7ff5ecc54700 1 --2- 172.21.15.59:0/1636149047 >> v2:172.21.15.110:6800/115870 conn(0x7ff5d4068890 0x7ff5d406ad40 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff5e80b5980 tx=0x7ff5dc017000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:09:18.345 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:18.344+0000 7ff5ef6b9700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:09:18.345 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:18.344+0000 7ff5ef6b9700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:09:18.345 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:18.344+0000 7ff5ef6b9700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:09:18.345 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:18.344+0000 7ff5ef6b9700 1 -- 172.21.15.59:0/1636149047 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ff5e80a8000 con 0x7ff5e80a5b70 2023-09-30T02:09:18.351 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:18.350+0000 7ff5e57fa700 1 -- 172.21.15.59:0/1636149047 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7ff5e80a8000 con 0x7ff5e80a5b70 2023-09-30T02:09:18.351 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:18.351+0000 7ff5e57fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:09:18.351 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:18.351+0000 7ff5e57fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:09:18.351 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:09:18.351 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:09:18.354 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:18.353+0000 7ff5ef6b9700 10 librados: watch_flush enter 2023-09-30T02:09:18.354 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:18.354+0000 7ff5ef6b9700 10 librados: watch_flush exit 2023-09-30T02:09:18.354 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:18.354+0000 7ff5ef6b9700 20 client.4635.objecter shutdown clearing up homeless session... 2023-09-30T02:09:18.354 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:18.354+0000 7ff5ef6b9700 10 client.4635.objecter successfully canceled tick 2023-09-30T02:09:18.354 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:18.354+0000 7ff5ef6b9700 1 -- 172.21.15.59:0/1636149047 >> v2:172.21.15.110:6800/115870 conn(0x7ff5d4068890 msgr2=0x7ff5d406ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:18.354 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:18.354+0000 7ff5ef6b9700 1 --2- 172.21.15.59:0/1636149047 >> v2:172.21.15.110:6800/115870 conn(0x7ff5d4068890 0x7ff5d406ad40 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff5e80b5980 tx=0x7ff5dc017000 comp rx=0 tx=0).stop 2023-09-30T02:09:18.355 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:18.354+0000 7ff5ef6b9700 10 monclient: shutdown 2023-09-30T02:09:18.355 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:18.354+0000 7ff5ef6b9700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:09:18.355 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:18.354+0000 7ff5ef6b9700 1 -- 172.21.15.59:0/1636149047 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff5e80a5b70 msgr2=0x7ff5e80b5420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:18.355 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:18.354+0000 7ff5ef6b9700 1 --2- 172.21.15.59:0/1636149047 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff5e80a5b70 0x7ff5e80b5420 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ff5d800bc30 tx=0x7ff5d8009840 comp rx=0 tx=0).stop 2023-09-30T02:09:18.356 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:18.354+0000 7ff5ef6b9700 1 -- 172.21.15.59:0/1636149047 shutdown_connections 2023-09-30T02:09:18.356 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:18.354+0000 7ff5ef6b9700 1 --2- 172.21.15.59:0/1636149047 >> v2:172.21.15.110:6800/115870 conn(0x7ff5d4068890 0x7ff5d406ad40 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:18.356 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:18.354+0000 7ff5ef6b9700 1 --2- 172.21.15.59:0/1636149047 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff5e81420c0 0x7ff5e80adc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:18.356 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:18.354+0000 7ff5ef6b9700 1 --2- 172.21.15.59:0/1636149047 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff5e80a5b70 0x7ff5e80b5420 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:18.356 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:18.354+0000 7ff5ef6b9700 1 --2- 172.21.15.59:0/1636149047 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff5e80a5f40 0x7ff5e80b2880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:18.356 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:18.354+0000 7ff5ef6b9700 1 -- 172.21.15.59:0/1636149047 >> 172.21.15.59:0/1636149047 conn(0x7ff5e80631f0 msgr2=0x7ff5e813ab90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:09:18.356 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:18.354+0000 7ff5ef6b9700 1 -- 172.21.15.59:0/1636149047 shutdown_connections 2023-09-30T02:09:18.356 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:18.354+0000 7ff5ef6b9700 1 -- 172.21.15.59:0/1636149047 wait complete. 2023-09-30T02:09:18.356 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:18.354+0000 7ff5ef6b9700 1 librados: shutdown 2023-09-30T02:09:21.368 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:09:21.521 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.520+0000 7f038faee700 10 monclient: get_monmap_and_config 2023-09-30T02:09:21.521 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.520+0000 7f038faee700 10 monclient: build_initial_monmap 2023-09-30T02:09:21.521 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.521+0000 7f038faee700 10 monclient: monmap: 2023-09-30T02:09:21.521 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:09:21.521 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:21.522 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:09:21.522379+0000 2023-09-30T02:09:21.522 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:09:21.522379+0000 2023-09-30T02:09:21.522 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:09:21.522 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:21.522 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:09:21.522 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:09:21.522 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:09:21.522 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:09:21.522 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:09:21.522 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:21.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.521+0000 7f038faee700 1 Processor -- start 2023-09-30T02:09:21.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.521+0000 7f038faee700 1 -- start start 2023-09-30T02:09:21.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.521+0000 7f038faee700 10 monclient: init 2023-09-30T02:09:21.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.522+0000 7f038faee700 10 monclient: _reopen_session rank -1 2023-09-30T02:09:21.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.522+0000 7f038faee700 10 monclient: _add_conns ranks=[2,4,1,0,3] 2023-09-30T02:09:21.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.522+0000 7f038faee700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f03880a5b70 0x7f03880b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:21.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.522+0000 7f038faee700 10 monclient(hunting): picked mon.noname-b con 0x7f03880a5b70 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:09:21.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.522+0000 7f038faee700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f03880a5f40 0x7f0388141b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:21.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.522+0000 7f038faee700 10 monclient(hunting): picked mon.noname-c con 0x7f03880a5f40 addr v2:172.21.15.110:3300/0 2023-09-30T02:09:21.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.522+0000 7f038faee700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f03881420c0 addr v1:172.21.15.110:6789/0 2023-09-30T02:09:21.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.522+0000 7f038faee700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03880a63b0 con 0x7f03881420c0 2023-09-30T02:09:21.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.522+0000 7f038faee700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:21.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.522+0000 7f038faee700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03880a75a0 con 0x7f03880a5f40 2023-09-30T02:09:21.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.522+0000 7f038faee700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:21.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.522+0000 7f038faee700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03880af3f0 con 0x7f03880a5b70 2023-09-30T02:09:21.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.522+0000 7f038faee700 10 monclient(hunting): _renew_subs 2023-09-30T02:09:21.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.522+0000 7f038faee700 10 monclient(hunting): authenticate will time out at 2006.825768s 2023-09-30T02:09:21.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.523+0000 7f038e08b700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f03881420c0 0x7f038813e230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.59:42764/0 (socket says 172.21.15.59:42764) 2023-09-30T02:09:21.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.523+0000 7f038e08b700 1 -- 172.21.15.59:0/3932835348 learned_addr learned my addr 172.21.15.59:0/3932835348 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:09:21.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.523+0000 7f038d089700 1 --2- 172.21.15.59:0/3932835348 >> v2:172.21.15.110:3300/0 conn(0x7f03880a5f40 0x7f0388141b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:21.526 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.525+0000 7f038d089700 10 monclient(hunting): get_auth_request con 0x7f03880a5f40 auth_method 0 2023-09-30T02:09:21.527 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.525+0000 7f038d089700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:09:21.527 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.525+0000 7f038d089700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:21.527 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.525+0000 7f038d089700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:09:21.527 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.525+0000 7f038d88a700 1 --2- 172.21.15.59:0/3932835348 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f03880a5b70 0x7f03880b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:21.527 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.526+0000 7f038d88a700 10 monclient(hunting): get_auth_request con 0x7f03880a5b70 auth_method 0 2023-09-30T02:09:21.527 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.526+0000 7f038d88a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:09:21.527 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.526+0000 7f038d88a700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:21.527 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.526+0000 7f038d88a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:09:21.527 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.526+0000 7f038d089700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:09:21.527 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.526+0000 7f038d089700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:09:21.527 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.526+0000 7f038d089700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:09:21.528 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.526+0000 7f038d88a700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:09:21.528 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.526+0000 7f038d88a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:09:21.528 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.526+0000 7f038d88a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:09:21.529 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.529+0000 7f038d089700 10 monclient(hunting): handle_auth_done global_id 4651 payload 274 2023-09-30T02:09:21.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.529+0000 7f038d089700 1 -- 172.21.15.59:0/3932835348 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f03880a5b70 msgr2=0x7f03880b1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:09:21.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.529+0000 7f038d089700 1 --2- 172.21.15.59:0/3932835348 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f03880a5b70 0x7f03880b1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:21.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.529+0000 7f038d089700 1 -- 172.21.15.59:0/3932835348 >> v1:172.21.15.110:6789/0 conn(0x7f03881420c0 legacy=0x7f038813e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:21.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.529+0000 7f038d089700 10 monclient: _finish_hunting 0 2023-09-30T02:09:21.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.529+0000 7f038d089700 1 monclient: found mon.noname-c 2023-09-30T02:09:21.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.529+0000 7f038d089700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:09:21.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.529+0000 7f038d089700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3300/0 2023-09-30T02:09:21.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.529+0000 7f038d089700 1 -- 172.21.15.59:0/3932835348 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0388142500 con 0x7f03880a5f40 2023-09-30T02:09:21.534 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.534+0000 7f038d089700 1 --2- 172.21.15.59:0/3932835348 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03880a5f40 0x7f0388141b80 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f03740098e0 tx=0x7f03740091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=186f9a682c46766d server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:09:21.539 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.538+0000 7f038c888700 1 -- 172.21.15.59:0/3932835348 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f037400b640 con 0x7f03880a5f40 2023-09-30T02:09:21.539 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.538+0000 7f038c888700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:09:21.539 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.538+0000 7f038c888700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:09:21.539 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.538+0000 7f038c888700 10 monclient: dump: 2023-09-30T02:09:21.539 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:09:21.539 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:21.539 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:21.539 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:21.540 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:09:21.540 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:21.540 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:09:21.540 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:09:21.540 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:09:21.540 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:21.540 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.538+0000 7f038c888700 10 monclient: _finish_auth 0 2023-09-30T02:09:21.540 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.538+0000 7f038c888700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:09:21.540 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.538+0000 7f038faee700 5 monclient: authenticate success, global_id 4651 2023-09-30T02:09:21.540 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.538+0000 7f038c888700 1 -- 172.21.15.59:0/3932835348 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f037400b7a0 con 0x7f03880a5f40 2023-09-30T02:09:21.541 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.538+0000 7f038c888700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:09:21.541 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.538+0000 7f038faee700 10 monclient: get_monmap_and_config success 2023-09-30T02:09:21.541 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.538+0000 7f038faee700 4 set_mon_vals no callback set 2023-09-30T02:09:21.541 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.538+0000 7f038c888700 1 -- 172.21.15.59:0/3932835348 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f037400bb40 con 0x7f03880a5f40 2023-09-30T02:09:21.541 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.538+0000 7f038c888700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:09:21.541 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.538+0000 7f038c888700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:09:21.541 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.538+0000 7f038c888700 10 monclient: dump: 2023-09-30T02:09:21.541 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:09:21.541 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:21.541 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:21.541 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:21.541 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:09:21.542 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:21.542 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:09:21.542 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:09:21.542 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:09:21.542 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:21.546 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.539+0000 7f038faee700 10 monclient: shutdown 2023-09-30T02:09:21.546 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.539+0000 7f038faee700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:09:21.546 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.539+0000 7f038faee700 1 -- 172.21.15.59:0/3932835348 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03880a5f40 msgr2=0x7f0388141b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:21.546 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.539+0000 7f038faee700 1 --2- 172.21.15.59:0/3932835348 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03880a5f40 0x7f0388141b80 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f03740098e0 tx=0x7f03740091a0 comp rx=0 tx=0).stop 2023-09-30T02:09:21.546 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.539+0000 7f038faee700 1 -- 172.21.15.59:0/3932835348 shutdown_connections 2023-09-30T02:09:21.546 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.539+0000 7f038faee700 1 --2- 172.21.15.59:0/3932835348 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f03880a5b70 0x7f03880b1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:21.546 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.539+0000 7f038faee700 1 --2- 172.21.15.59:0/3932835348 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03880a5f40 0x7f0388141b80 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:21.546 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.539+0000 7f038faee700 1 -- 172.21.15.59:0/3932835348 >> 172.21.15.59:0/3932835348 conn(0x7f03880631f0 msgr2=0x7f03880635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:09:21.547 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.539+0000 7f038faee700 1 -- 172.21.15.59:0/3932835348 shutdown_connections 2023-09-30T02:09:21.547 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.539+0000 7f038faee700 1 -- 172.21.15.59:0/3932835348 wait complete. 2023-09-30T02:09:21.547 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.540+0000 7f038faee700 10 monclient: build_initial_monmap 2023-09-30T02:09:21.547 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.540+0000 7f038faee700 10 monclient: monmap: 2023-09-30T02:09:21.547 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:09:21.547 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:09:21.547 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:09:21.547 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:09:21.547 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:09:21.547 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:09:21.548 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:09:21.548 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:09:21.548 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:09:21.548 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:21.548 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.540+0000 7f038faee700 1 librados: starting msgr at 2023-09-30T02:09:21.548 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.540+0000 7f038faee700 1 librados: starting objecter 2023-09-30T02:09:21.548 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.540+0000 7f038faee700 1 Processor -- start 2023-09-30T02:09:21.548 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.540+0000 7f038faee700 1 -- start start 2023-09-30T02:09:21.548 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.540+0000 7f038faee700 1 librados: setting wanted keys 2023-09-30T02:09:21.548 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.541+0000 7f038faee700 1 librados: calling monclient init 2023-09-30T02:09:21.548 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.541+0000 7f038faee700 10 monclient: init 2023-09-30T02:09:21.548 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.541+0000 7f038faee700 10 monclient: _reopen_session rank -1 2023-09-30T02:09:21.549 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.541+0000 7f038faee700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T02:09:21.549 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.541+0000 7f038faee700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03880a5b70 0x7f03880a95e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:21.549 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.541+0000 7f038faee700 10 monclient(hunting): picked mon.noname-b con 0x7f03880a5b70 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:09:21.549 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.541+0000 7f038faee700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03880a5f40 0x7f03880aa4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:21.549 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.541+0000 7f038faee700 10 monclient(hunting): picked mon.noname-a con 0x7f03880a5f40 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:09:21.549 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.541+0000 7f038faee700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f03881420c0 0x7f038814dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:21.549 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.541+0000 7f038faee700 10 monclient(hunting): picked mon.noname-c con 0x7f03881420c0 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:09:21.549 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.541+0000 7f038faee700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:21.549 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.541+0000 7f038faee700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03880ae760 con 0x7f03880a5f40 2023-09-30T02:09:21.549 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.541+0000 7f038faee700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:21.549 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.541+0000 7f038faee700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03880aa040 con 0x7f03880a5b70 2023-09-30T02:09:21.550 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.541+0000 7f038faee700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:21.550 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.541+0000 7f038faee700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03880aaa40 con 0x7f03881420c0 2023-09-30T02:09:21.550 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.541+0000 7f038faee700 10 monclient(hunting): _renew_subs 2023-09-30T02:09:21.550 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.541+0000 7f038faee700 10 monclient(hunting): authenticate will time out at 2006.844584s 2023-09-30T02:09:21.550 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.542+0000 7f038d88a700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03880a5b70 0x7f03880a95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:21.550 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.542+0000 7f038e08b700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f03881420c0 0x7f038814dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:21.550 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.542+0000 7f038d089700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03880a5f40 0x7f03880aa4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:21.550 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.542+0000 7f038d88a700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03880a5b70 0x7f03880a95e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:38892/0 (socket says 172.21.15.59:38892) 2023-09-30T02:09:21.550 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.542+0000 7f038d88a700 1 -- 172.21.15.59:0/2048478399 learned_addr learned my addr 172.21.15.59:0/2048478399 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:09:21.550 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.542+0000 7f038d88a700 10 monclient(hunting): get_auth_request con 0x7f03880a5b70 auth_method 0 2023-09-30T02:09:21.550 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.542+0000 7f038d88a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:09:21.551 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.542+0000 7f038d88a700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:21.551 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.542+0000 7f038d88a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:09:21.551 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.542+0000 7f038e08b700 10 monclient(hunting): get_auth_request con 0x7f03881420c0 auth_method 0 2023-09-30T02:09:21.551 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.542+0000 7f038e08b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:09:21.551 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.542+0000 7f038e08b700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:21.551 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.542+0000 7f038e08b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:09:21.551 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.542+0000 7f038d089700 10 monclient(hunting): get_auth_request con 0x7f03880a5f40 auth_method 0 2023-09-30T02:09:21.551 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.542+0000 7f038d089700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:09:21.551 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.542+0000 7f038d089700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:21.551 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.542+0000 7f038d089700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:09:21.551 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.542+0000 7f038d88a700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:09:21.551 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.542+0000 7f038d88a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:09:21.551 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.542+0000 7f038d88a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:09:21.552 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.543+0000 7f038e08b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:09:21.552 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.543+0000 7f038e08b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:09:21.552 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.543+0000 7f038e08b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:09:21.552 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.543+0000 7f038d089700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:09:21.552 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.543+0000 7f038d089700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:09:21.552 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.543+0000 7f038d089700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:09:21.553 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.546+0000 7f038d88a700 10 monclient(hunting): handle_auth_done global_id 4654 payload 824 2023-09-30T02:09:21.553 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.546+0000 7f038d88a700 1 -- 172.21.15.59:0/2048478399 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f03881420c0 msgr2=0x7f038814dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:21.553 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.546+0000 7f038d88a700 1 --2- 172.21.15.59:0/2048478399 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f03881420c0 0x7f038814dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:21.553 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.546+0000 7f038d88a700 1 -- 172.21.15.59:0/2048478399 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03880a5f40 msgr2=0x7f03880aa4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:21.553 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.546+0000 7f038d88a700 1 --2- 172.21.15.59:0/2048478399 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03880a5f40 0x7f03880aa4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:21.553 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.546+0000 7f038d88a700 10 monclient: _finish_hunting 0 2023-09-30T02:09:21.553 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.546+0000 7f038d88a700 1 monclient: found mon.noname-b 2023-09-30T02:09:21.553 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.546+0000 7f038d88a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:09:21.553 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.546+0000 7f038d88a700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:09:21.554 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.546+0000 7f038d88a700 1 -- 172.21.15.59:0/2048478399 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0374009420 con 0x7f03880a5b70 2023-09-30T02:09:21.554 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.549+0000 7f038d88a700 1 --2- 172.21.15.59:0/2048478399 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03880a5b70 0x7f03880a95e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f037c00ecb0 tx=0x7f037c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:09:21.554 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.549+0000 7f0385ffb700 10 client.?.objecter ms_handle_connect 0x7f03880a5b70 2023-09-30T02:09:21.554 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.549+0000 7f0385ffb700 10 client.?.objecter resend_mon_ops 2023-09-30T02:09:21.555 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.553+0000 7f0385ffb700 1 -- 172.21.15.59:0/2048478399 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f037c00c850 con 0x7f03880a5b70 2023-09-30T02:09:21.555 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.553+0000 7f0385ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:09:21.555 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.553+0000 7f0385ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:09:21.555 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.553+0000 7f0385ffb700 10 monclient: dump: 2023-09-30T02:09:21.555 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:09:21.555 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:21.555 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:21.555 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:21.556 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:09:21.556 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:21.556 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:09:21.556 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:09:21.556 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:09:21.556 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:21.556 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.553+0000 7f0385ffb700 10 monclient: _finish_auth 0 2023-09-30T02:09:21.556 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.553+0000 7f0385ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:09:21.556 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.553+0000 7f0385ffb700 1 -- 172.21.15.59:0/2048478399 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f037c00c9b0 con 0x7f03880a5b70 2023-09-30T02:09:21.556 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.553+0000 7f038faee700 5 monclient: authenticate success, global_id 4654 2023-09-30T02:09:21.556 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.553+0000 7f038faee700 10 monclient: _renew_subs 2023-09-30T02:09:21.556 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.553+0000 7f038faee700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:09:21.557 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.553+0000 7f038faee700 1 -- 172.21.15.59:0/2048478399 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f038814e2e0 con 0x7f03880a5b70 2023-09-30T02:09:21.557 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.553+0000 7f0385ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:09:21.557 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.553+0000 7f0385ffb700 1 -- 172.21.15.59:0/2048478399 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f037c00cd70 con 0x7f03880a5b70 2023-09-30T02:09:21.557 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.553+0000 7f0385ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:09:21.557 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.553+0000 7f0385ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:09:21.557 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.553+0000 7f0385ffb700 10 monclient: dump: 2023-09-30T02:09:21.557 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:09:21.557 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:21.557 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:21.557 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:21.557 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:09:21.557 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:21.557 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:09:21.558 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:09:21.558 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:09:21.558 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:21.558 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.553+0000 7f0387fff700 4 set_mon_vals no callback set 2023-09-30T02:09:21.558 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.553+0000 7f038faee700 10 client.4654.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:09:21.558 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.553+0000 7f038faee700 10 monclient: _renew_subs 2023-09-30T02:09:21.558 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.553+0000 7f038faee700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:09:21.558 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.553+0000 7f038faee700 1 -- 172.21.15.59:0/2048478399 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f038813d830 con 0x7f03880a5b70 2023-09-30T02:09:21.558 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.553+0000 7f038faee700 1 librados: init done 2023-09-30T02:09:21.558 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.554+0000 7f038faee700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:09:21.558 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.554+0000 7f038faee700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:09:21.559 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.554+0000 7f038faee700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:09:21.559 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.554+0000 7f038faee700 1 -- 172.21.15.59:0/2048478399 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03880b1330 con 0x7f03880a5b70 2023-09-30T02:09:21.710 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.709+0000 7f0385ffb700 1 -- 172.21.15.59:0/2048478399 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f037c013680 con 0x7f03880a5b70 2023-09-30T02:09:21.710 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.710+0000 7f0385ffb700 1 --2- 172.21.15.59:0/2048478399 >> v2:172.21.15.110:6800/115870 conn(0x7f036c0689e0 0x7f036c06ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:21.711 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.710+0000 7f0385ffb700 1 -- 172.21.15.59:0/2048478399 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f037c084810 con 0x7f03880a5b70 2023-09-30T02:09:21.711 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.710+0000 7f0385ffb700 10 client.4654.objecter ms_dispatch 0x7f0388001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:09:21.711 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.710+0000 7f0385ffb700 3 client.4654.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:09:21.711 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.710+0000 7f0385ffb700 3 client.4654.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:09:21.711 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.710+0000 7f0385ffb700 20 client.4654.objecter dump_active .. 0 homeless 2023-09-30T02:09:21.711 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.710+0000 7f038d089700 1 --2- 172.21.15.59:0/2048478399 >> v2:172.21.15.110:6800/115870 conn(0x7f036c0689e0 0x7f036c06ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:21.711 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.710+0000 7f038d089700 10 monclient: get_auth_request con 0x7f036c0689e0 auth_method 0 2023-09-30T02:09:21.711 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.710+0000 7f038d089700 1 --2- 172.21.15.59:0/2048478399 >> v2:172.21.15.110:6800/115870 conn(0x7f036c0689e0 0x7f036c06ae90 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f03740093f0 tx=0x7f0374017000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:09:21.712 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.711+0000 7f0385ffb700 10 client.4654.objecter ms_handle_connect 0x7f036c0689e0 2023-09-30T02:09:21.819 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.819+0000 7f0385ffb700 1 -- 172.21.15.59:0/2048478399 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f037c052130 con 0x7f03880a5b70 2023-09-30T02:09:21.820 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.819+0000 7f0385ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:09:21.820 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:21.819+0000 7f0385ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:09:22.073 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:22.072+0000 7f038faee700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:09:22.073 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:22.072+0000 7f038faee700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:09:22.073 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:22.072+0000 7f038faee700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:09:22.073 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:22.072+0000 7f038faee700 1 -- 172.21.15.59:0/2048478399 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f0388141be0 con 0x7f03880a5b70 2023-09-30T02:09:22.113 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:22.113+0000 7f0385ffb700 1 -- 172.21.15.59:0/2048478399 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f037c055750 con 0x7f03880a5b70 2023-09-30T02:09:22.113 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:22.113+0000 7f0385ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:09:22.113 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:22.113+0000 7f0385ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:09:22.114 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:09:22.114 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:09:22.116 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:22.115+0000 7f038faee700 10 librados: watch_flush enter 2023-09-30T02:09:22.116 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:22.115+0000 7f038faee700 10 librados: watch_flush exit 2023-09-30T02:09:22.116 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:22.115+0000 7f038faee700 20 client.4654.objecter shutdown clearing up homeless session... 2023-09-30T02:09:22.116 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:22.115+0000 7f038faee700 10 client.4654.objecter successfully canceled tick 2023-09-30T02:09:22.116 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:22.115+0000 7f038faee700 1 -- 172.21.15.59:0/2048478399 >> v2:172.21.15.110:6800/115870 conn(0x7f036c0689e0 msgr2=0x7f036c06ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:22.117 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:22.115+0000 7f038faee700 1 --2- 172.21.15.59:0/2048478399 >> v2:172.21.15.110:6800/115870 conn(0x7f036c0689e0 0x7f036c06ae90 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f03740093f0 tx=0x7f0374017000 comp rx=0 tx=0).stop 2023-09-30T02:09:22.117 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:22.115+0000 7f038faee700 10 monclient: shutdown 2023-09-30T02:09:22.117 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:22.115+0000 7f038faee700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:09:22.117 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:22.115+0000 7f038faee700 1 -- 172.21.15.59:0/2048478399 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03880a5b70 msgr2=0x7f03880a95e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:22.117 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:22.115+0000 7f038faee700 1 --2- 172.21.15.59:0/2048478399 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03880a5b70 0x7f03880a95e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f037c00ecb0 tx=0x7f037c0091a0 comp rx=0 tx=0).stop 2023-09-30T02:09:22.117 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:22.115+0000 7f038faee700 1 -- 172.21.15.59:0/2048478399 shutdown_connections 2023-09-30T02:09:22.117 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:22.115+0000 7f038faee700 1 --2- 172.21.15.59:0/2048478399 >> v2:172.21.15.110:6800/115870 conn(0x7f036c0689e0 0x7f036c06ae90 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:22.117 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:22.115+0000 7f038faee700 1 --2- 172.21.15.59:0/2048478399 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f03880a5b70 0x7f03880a95e0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:22.117 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:22.115+0000 7f038faee700 1 --2- 172.21.15.59:0/2048478399 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f03881420c0 0x7f038814dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:22.117 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:22.115+0000 7f038faee700 1 --2- 172.21.15.59:0/2048478399 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f03880a5f40 0x7f03880aa4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:22.118 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:22.115+0000 7f038faee700 1 -- 172.21.15.59:0/2048478399 >> 172.21.15.59:0/2048478399 conn(0x7f03880631f0 msgr2=0x7f03880635c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:09:22.118 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:22.115+0000 7f038faee700 1 -- 172.21.15.59:0/2048478399 shutdown_connections 2023-09-30T02:09:22.118 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:22.115+0000 7f038faee700 1 -- 172.21.15.59:0/2048478399 wait complete. 2023-09-30T02:09:22.118 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:22.115+0000 7f038faee700 1 librados: shutdown 2023-09-30T02:09:25.130 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:09:25.269 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.269+0000 7f394b6ae700 10 monclient: get_monmap_and_config 2023-09-30T02:09:25.269 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.269+0000 7f394b6ae700 10 monclient: build_initial_monmap 2023-09-30T02:09:25.269 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.269+0000 7f394b6ae700 10 monclient: monmap: 2023-09-30T02:09:25.270 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:09:25.270 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:25.270 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:09:25.270564+0000 2023-09-30T02:09:25.270 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:09:25.270564+0000 2023-09-30T02:09:25.270 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:09:25.270 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:25.270 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:09:25.270 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:09:25.270 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:09:25.270 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:09:25.271 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:09:25.271 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:25.272 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.269+0000 7f394b6ae700 1 Processor -- start 2023-09-30T02:09:25.272 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.270+0000 7f394b6ae700 1 -- start start 2023-09-30T02:09:25.272 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.270+0000 7f394b6ae700 10 monclient: init 2023-09-30T02:09:25.272 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.270+0000 7f394b6ae700 10 monclient: _reopen_session rank -1 2023-09-30T02:09:25.273 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.270+0000 7f394b6ae700 10 monclient: _add_conns ranks=[0,1,4,3,2] 2023-09-30T02:09:25.273 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.270+0000 7f394b6ae700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f39440a5b70 addr v1:172.21.15.59:6789/0 2023-09-30T02:09:25.273 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.270+0000 7f394b6ae700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3944142290 addr v1:172.21.15.110:6789/0 2023-09-30T02:09:25.273 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.270+0000 7f394b6ae700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f39440b1500 0x7f39440a5f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:25.273 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.270+0000 7f394b6ae700 10 monclient(hunting): picked mon.noname-c con 0x7f39440b1500 addr v2:172.21.15.110:3300/0 2023-09-30T02:09:25.273 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.270+0000 7f394b6ae700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39440b1920 con 0x7f39440a5b70 2023-09-30T02:09:25.273 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.270+0000 7f394b6ae700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39440a75a0 con 0x7f3944142290 2023-09-30T02:09:25.273 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.270+0000 7f394b6ae700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:25.273 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.270+0000 7f394b6ae700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39440af3f0 con 0x7f39440b1500 2023-09-30T02:09:25.273 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.270+0000 7f394b6ae700 10 monclient(hunting): _renew_subs 2023-09-30T02:09:25.274 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.270+0000 7f394b6ae700 10 monclient(hunting): authenticate will time out at 2010.573817s 2023-09-30T02:09:25.274 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.271+0000 7f394944a700 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f39440a5b70 0x7f3944141b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:47080/0 (socket says 172.21.15.59:47080) 2023-09-30T02:09:25.274 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.271+0000 7f394944a700 1 -- 172.21.15.59:0/1387107613 learned_addr learned my addr 172.21.15.59:0/1387107613 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:09:25.274 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.271+0000 7f3949c4b700 1 --2- 172.21.15.59:0/1387107613 >> v2:172.21.15.110:3300/0 conn(0x7f39440b1500 0x7f39440a5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:25.274 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.272+0000 7f3949c4b700 10 monclient(hunting): get_auth_request con 0x7f39440b1500 auth_method 0 2023-09-30T02:09:25.274 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.272+0000 7f3949c4b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:09:25.274 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.272+0000 7f3949c4b700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:25.275 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.272+0000 7f3949c4b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:09:25.275 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.273+0000 7f3949c4b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:09:25.275 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.273+0000 7f3949c4b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:09:25.275 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.273+0000 7f3949c4b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:09:25.276 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.274+0000 7f393bfff700 1 -- 172.21.15.59:0/1387107613 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2033605810 0 0) 0x7f3940001110 con 0x7f39440a5b70 2023-09-30T02:09:25.276 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.274+0000 7f393bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:25.276 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.274+0000 7f393bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:09:25.276 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.274+0000 7f393bfff700 10 monclient(hunting): my global_id is 4641 2023-09-30T02:09:25.276 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.274+0000 7f393bfff700 1 -- 172.21.15.59:0/1387107613 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3930003570 con 0x7f39440a5b70 2023-09-30T02:09:25.276 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.276+0000 7f3949c4b700 10 monclient(hunting): handle_auth_done global_id 4657 payload 274 2023-09-30T02:09:25.276 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.276+0000 7f3949c4b700 1 -- 172.21.15.59:0/1387107613 >> v1:172.21.15.110:6789/0 conn(0x7f3944142290 legacy=0x7f394413af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:25.277 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.276+0000 7f3949c4b700 1 -- 172.21.15.59:0/1387107613 >> v1:172.21.15.59:6789/0 conn(0x7f39440a5b70 legacy=0x7f3944141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:25.277 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.276+0000 7f3949c4b700 10 monclient: _finish_hunting 0 2023-09-30T02:09:25.277 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.276+0000 7f3949c4b700 1 monclient: found mon.noname-c 2023-09-30T02:09:25.277 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.276+0000 7f3949c4b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:09:25.277 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.276+0000 7f3949c4b700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3300/0 2023-09-30T02:09:25.277 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.276+0000 7f3949c4b700 1 -- 172.21.15.59:0/1387107613 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f394413b640 con 0x7f39440b1500 2023-09-30T02:09:25.280 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.280+0000 7f3949c4b700 1 --2- 172.21.15.59:0/1387107613 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f39440b1500 0x7f39440a5f40 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f393c009e60 tx=0x7f393c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9a45189c73939c86 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:09:25.286 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.284+0000 7f393bfff700 1 -- 172.21.15.59:0/1387107613 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f393c00e070 con 0x7f39440b1500 2023-09-30T02:09:25.286 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.284+0000 7f393bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:09:25.287 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.284+0000 7f393bfff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:09:25.287 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.284+0000 7f393bfff700 10 monclient: dump: 2023-09-30T02:09:25.287 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:09:25.287 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:25.287 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:25.287 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:25.287 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:09:25.288 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:25.288 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:09:25.288 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:09:25.288 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:09:25.288 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:25.288 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.284+0000 7f393bfff700 10 monclient: _finish_auth 0 2023-09-30T02:09:25.288 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.284+0000 7f393bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:09:25.288 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.284+0000 7f393bfff700 1 -- 172.21.15.59:0/1387107613 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f393c00bb60 con 0x7f39440b1500 2023-09-30T02:09:25.289 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.284+0000 7f393bfff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:09:25.289 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.284+0000 7f393bfff700 1 -- 172.21.15.59:0/1387107613 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f393c012430 con 0x7f39440b1500 2023-09-30T02:09:25.289 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.284+0000 7f394b6ae700 5 monclient: authenticate success, global_id 4657 2023-09-30T02:09:25.289 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.284+0000 7f394b6ae700 10 monclient: get_monmap_and_config success 2023-09-30T02:09:25.289 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.284+0000 7f394b6ae700 4 set_mon_vals no callback set 2023-09-30T02:09:25.289 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.284+0000 7f393bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:09:25.289 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.284+0000 7f393bfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:09:25.289 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.284+0000 7f393bfff700 10 monclient: dump: 2023-09-30T02:09:25.289 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:09:25.289 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:25.289 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:25.289 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:25.289 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:09:25.290 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:25.290 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:09:25.290 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:09:25.290 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:09:25.290 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:25.290 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.285+0000 7f394b6ae700 10 monclient: shutdown 2023-09-30T02:09:25.290 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.285+0000 7f394b6ae700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:09:25.290 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.285+0000 7f394b6ae700 1 -- 172.21.15.59:0/1387107613 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f39440b1500 msgr2=0x7f39440a5f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:25.290 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.285+0000 7f394b6ae700 1 --2- 172.21.15.59:0/1387107613 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f39440b1500 0x7f39440a5f40 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f393c009e60 tx=0x7f393c009780 comp rx=0 tx=0).stop 2023-09-30T02:09:25.290 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.285+0000 7f394b6ae700 1 -- 172.21.15.59:0/1387107613 shutdown_connections 2023-09-30T02:09:25.290 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.285+0000 7f394b6ae700 1 --2- 172.21.15.59:0/1387107613 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f39440b1500 0x7f39440a5f40 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:25.291 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.285+0000 7f394b6ae700 1 -- 172.21.15.59:0/1387107613 >> 172.21.15.59:0/1387107613 conn(0x7f39440631f0 msgr2=0x7f39440635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:09:25.291 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.285+0000 7f394b6ae700 1 -- 172.21.15.59:0/1387107613 shutdown_connections 2023-09-30T02:09:25.291 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.285+0000 7f394b6ae700 1 -- 172.21.15.59:0/1387107613 wait complete. 2023-09-30T02:09:25.291 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.286+0000 7f394b6ae700 10 monclient: build_initial_monmap 2023-09-30T02:09:25.291 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.286+0000 7f394b6ae700 10 monclient: monmap: 2023-09-30T02:09:25.291 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:09:25.291 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:09:25.291 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:09:25.291 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:09:25.291 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:09:25.291 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:09:25.291 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:09:25.291 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:09:25.292 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:09:25.292 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:25.292 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.286+0000 7f394b6ae700 1 librados: starting msgr at 2023-09-30T02:09:25.292 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.286+0000 7f394b6ae700 1 librados: starting objecter 2023-09-30T02:09:25.292 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.286+0000 7f394b6ae700 1 Processor -- start 2023-09-30T02:09:25.292 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.286+0000 7f394b6ae700 1 -- start start 2023-09-30T02:09:25.292 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.286+0000 7f394b6ae700 1 librados: setting wanted keys 2023-09-30T02:09:25.292 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.286+0000 7f394b6ae700 1 librados: calling monclient init 2023-09-30T02:09:25.292 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.286+0000 7f394b6ae700 10 monclient: init 2023-09-30T02:09:25.295 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.287+0000 7f394b6ae700 10 monclient: _reopen_session rank -1 2023-09-30T02:09:25.295 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.287+0000 7f394b6ae700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T02:09:25.295 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.287+0000 7f394b6ae700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39440a5b70 0x7f39441403a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:25.295 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.287+0000 7f394b6ae700 10 monclient(hunting): picked mon.noname-a con 0x7f39440a5b70 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:09:25.295 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.287+0000 7f394b6ae700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f39440b1500 0x7f394413e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:25.295 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.287+0000 7f394b6ae700 10 monclient(hunting): picked mon.noname-c con 0x7f39440b1500 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:09:25.295 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.287+0000 7f394b6ae700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3944142290 0x7f3944136c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:25.295 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.287+0000 7f394b6ae700 10 monclient(hunting): picked mon.noname-b con 0x7f3944142290 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:09:25.295 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.287+0000 7f394b6ae700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:25.296 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.287+0000 7f394b6ae700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f394413e7b0 con 0x7f39440a5b70 2023-09-30T02:09:25.296 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.287+0000 7f394b6ae700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:25.296 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.287+0000 7f394b6ae700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39441410d0 con 0x7f3944142290 2023-09-30T02:09:25.296 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.287+0000 7f394b6ae700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:25.296 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.287+0000 7f394b6ae700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f394413d510 con 0x7f39440b1500 2023-09-30T02:09:25.296 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.287+0000 7f394b6ae700 10 monclient(hunting): _renew_subs 2023-09-30T02:09:25.296 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.287+0000 7f394b6ae700 10 monclient(hunting): authenticate will time out at 2010.590575s 2023-09-30T02:09:25.296 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.288+0000 7f3948c49700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f39440b1500 0x7f394413e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:25.296 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.288+0000 7f394944a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39440a5b70 0x7f39441403a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:25.296 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.288+0000 7f3949c4b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3944142290 0x7f3944136c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:25.296 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.288+0000 7f3949c4b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3944142290 0x7f3944136c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:38916/0 (socket says 172.21.15.59:38916) 2023-09-30T02:09:25.296 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.288+0000 7f3949c4b700 1 -- 172.21.15.59:0/706173838 learned_addr learned my addr 172.21.15.59:0/706173838 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:09:25.297 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.288+0000 7f3949c4b700 10 monclient(hunting): get_auth_request con 0x7f3944142290 auth_method 0 2023-09-30T02:09:25.297 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.288+0000 7f3949c4b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:09:25.297 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.288+0000 7f3949c4b700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:25.297 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.288+0000 7f3949c4b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:09:25.297 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.288+0000 7f3948c49700 10 monclient(hunting): get_auth_request con 0x7f39440b1500 auth_method 0 2023-09-30T02:09:25.297 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.288+0000 7f3948c49700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:09:25.297 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.288+0000 7f3948c49700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:25.297 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.288+0000 7f3948c49700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:09:25.297 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.288+0000 7f394944a700 10 monclient(hunting): get_auth_request con 0x7f39440a5b70 auth_method 0 2023-09-30T02:09:25.297 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.288+0000 7f394944a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:09:25.297 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.288+0000 7f394944a700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:25.297 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.288+0000 7f394944a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:09:25.298 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.289+0000 7f3949c4b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:09:25.298 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.289+0000 7f3949c4b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:09:25.298 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.289+0000 7f3949c4b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:09:25.298 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.289+0000 7f3948c49700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:09:25.298 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.289+0000 7f3948c49700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:09:25.298 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.289+0000 7f3948c49700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:09:25.298 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.289+0000 7f394944a700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:09:25.298 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.289+0000 7f394944a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:09:25.298 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.289+0000 7f394944a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:09:25.298 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.292+0000 7f3949c4b700 10 monclient(hunting): handle_auth_done global_id 4660 payload 824 2023-09-30T02:09:25.298 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.292+0000 7f3949c4b700 1 -- 172.21.15.59:0/706173838 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f39440b1500 msgr2=0x7f394413e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:25.299 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.292+0000 7f3949c4b700 1 --2- 172.21.15.59:0/706173838 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f39440b1500 0x7f394413e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:25.299 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.292+0000 7f3949c4b700 1 -- 172.21.15.59:0/706173838 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39440a5b70 msgr2=0x7f39441403a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:25.299 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.292+0000 7f3949c4b700 1 --2- 172.21.15.59:0/706173838 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39440a5b70 0x7f39441403a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:25.299 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.292+0000 7f3949c4b700 10 monclient: _finish_hunting 0 2023-09-30T02:09:25.299 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.292+0000 7f3949c4b700 1 monclient: found mon.noname-b 2023-09-30T02:09:25.299 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.292+0000 7f3949c4b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:09:25.299 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.292+0000 7f3949c4b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:09:25.299 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.292+0000 7f3949c4b700 1 -- 172.21.15.59:0/706173838 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f393c009a20 con 0x7f3944142290 2023-09-30T02:09:25.299 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.294+0000 7f3949c4b700 1 --2- 172.21.15.59:0/706173838 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3944142290 0x7f3944136c60 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f393c0099f0 tx=0x7f393c019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:09:25.299 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.294+0000 7f39397fa700 10 client.?.objecter ms_handle_connect 0x7f3944142290 2023-09-30T02:09:25.299 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.294+0000 7f39397fa700 10 client.?.objecter resend_mon_ops 2023-09-30T02:09:25.301 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.298+0000 7f39397fa700 1 -- 172.21.15.59:0/706173838 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f393c009420 con 0x7f3944142290 2023-09-30T02:09:25.301 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.298+0000 7f39397fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:09:25.301 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.298+0000 7f39397fa700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:09:25.301 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.298+0000 7f39397fa700 10 monclient: dump: 2023-09-30T02:09:25.301 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:09:25.301 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:25.301 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:25.301 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:25.301 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:09:25.301 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:25.301 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:09:25.302 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:09:25.302 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:09:25.302 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:25.302 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.298+0000 7f39397fa700 10 monclient: _finish_auth 0 2023-09-30T02:09:25.302 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.298+0000 7f39397fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:09:25.302 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.298+0000 7f39397fa700 1 -- 172.21.15.59:0/706173838 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f393c027470 con 0x7f3944142290 2023-09-30T02:09:25.302 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.298+0000 7f394b6ae700 5 monclient: authenticate success, global_id 4660 2023-09-30T02:09:25.302 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.298+0000 7f394b6ae700 10 monclient: _renew_subs 2023-09-30T02:09:25.302 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.298+0000 7f394b6ae700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:09:25.302 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.298+0000 7f394b6ae700 1 -- 172.21.15.59:0/706173838 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39441371a0 con 0x7f3944142290 2023-09-30T02:09:25.302 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.298+0000 7f39397fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:09:25.303 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.298+0000 7f39397fa700 1 -- 172.21.15.59:0/706173838 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f393c027830 con 0x7f3944142290 2023-09-30T02:09:25.303 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.298+0000 7f39397fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:09:25.303 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.298+0000 7f39397fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:09:25.303 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.298+0000 7f39397fa700 10 monclient: dump: 2023-09-30T02:09:25.303 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:09:25.303 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:25.303 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:25.303 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:25.303 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:09:25.303 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:25.303 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:09:25.303 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:09:25.304 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:09:25.304 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:25.304 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.298+0000 7f393b7fe700 4 set_mon_vals no callback set 2023-09-30T02:09:25.304 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.298+0000 7f394b6ae700 10 client.4660.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:09:25.304 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.298+0000 7f394b6ae700 10 monclient: _renew_subs 2023-09-30T02:09:25.304 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.298+0000 7f394b6ae700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:09:25.304 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.298+0000 7f394b6ae700 1 -- 172.21.15.59:0/706173838 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39441376e0 con 0x7f3944142290 2023-09-30T02:09:25.304 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.298+0000 7f394b6ae700 1 librados: init done 2023-09-30T02:09:25.304 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.299+0000 7f394b6ae700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:09:25.304 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.299+0000 7f394b6ae700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:09:25.304 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.299+0000 7f394b6ae700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:09:25.305 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.299+0000 7f394b6ae700 1 -- 172.21.15.59:0/706173838 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f394413e590 con 0x7f3944142290 2023-09-30T02:09:25.317 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.317+0000 7f39397fa700 1 -- 172.21.15.59:0/706173838 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f393c024070 con 0x7f3944142290 2023-09-30T02:09:25.318 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.318+0000 7f39397fa700 1 --2- 172.21.15.59:0/706173838 >> v2:172.21.15.110:6800/115870 conn(0x7f39280688c0 0x7f392806ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:25.318 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.318+0000 7f394944a700 1 --2- 172.21.15.59:0/706173838 >> v2:172.21.15.110:6800/115870 conn(0x7f39280688c0 0x7f392806ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:25.318 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.318+0000 7f394944a700 10 monclient: get_auth_request con 0x7f39280688c0 auth_method 0 2023-09-30T02:09:25.318 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.318+0000 7f39397fa700 10 client.4660.objecter ms_handle_connect 0x7f39280688c0 2023-09-30T02:09:25.318 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.318+0000 7f394944a700 1 --2- 172.21.15.59:0/706173838 >> v2:172.21.15.110:6800/115870 conn(0x7f39280688c0 0x7f392806ad70 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f3940001270 tx=0x7f3940007bf0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:09:25.554 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.554+0000 7f39397fa700 1 -- 172.21.15.59:0/706173838 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f393c025370 con 0x7f3944142290 2023-09-30T02:09:25.555 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.554+0000 7f39397fa700 10 client.4660.objecter ms_dispatch 0x7f3944001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:09:25.555 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.554+0000 7f39397fa700 3 client.4660.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:09:25.555 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.554+0000 7f39397fa700 3 client.4660.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:09:25.555 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.554+0000 7f39397fa700 20 client.4660.objecter dump_active .. 0 homeless 2023-09-30T02:09:25.576 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.576+0000 7f39397fa700 1 -- 172.21.15.59:0/706173838 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f393c054420 con 0x7f3944142290 2023-09-30T02:09:25.576 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.576+0000 7f39397fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:09:25.576 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.576+0000 7f39397fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:09:25.843 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.842+0000 7f394b6ae700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:09:25.843 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.842+0000 7f394b6ae700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:09:25.843 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.842+0000 7f394b6ae700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:09:25.843 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.842+0000 7f394b6ae700 1 -- 172.21.15.59:0/706173838 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f39441376e0 con 0x7f3944142290 2023-09-30T02:09:25.847 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.847+0000 7f39397fa700 1 -- 172.21.15.59:0/706173838 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f39441376e0 con 0x7f3944142290 2023-09-30T02:09:25.847 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.847+0000 7f39397fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:09:25.847 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.847+0000 7f39397fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:09:25.847 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:09:25.847 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:09:25.850 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.849+0000 7f394b6ae700 10 librados: watch_flush enter 2023-09-30T02:09:25.850 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.849+0000 7f394b6ae700 10 librados: watch_flush exit 2023-09-30T02:09:25.850 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.849+0000 7f394b6ae700 20 client.4660.objecter shutdown clearing up homeless session... 2023-09-30T02:09:25.850 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.849+0000 7f394b6ae700 10 client.4660.objecter successfully canceled tick 2023-09-30T02:09:25.851 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.849+0000 7f394b6ae700 1 -- 172.21.15.59:0/706173838 >> v2:172.21.15.110:6800/115870 conn(0x7f39280688c0 msgr2=0x7f392806ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:25.851 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.849+0000 7f394b6ae700 1 --2- 172.21.15.59:0/706173838 >> v2:172.21.15.110:6800/115870 conn(0x7f39280688c0 0x7f392806ad70 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f3940001270 tx=0x7f3940007bf0 comp rx=0 tx=0).stop 2023-09-30T02:09:25.851 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.849+0000 7f394b6ae700 10 monclient: shutdown 2023-09-30T02:09:25.851 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.849+0000 7f394b6ae700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:09:25.851 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.849+0000 7f394b6ae700 1 -- 172.21.15.59:0/706173838 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3944142290 msgr2=0x7f3944136c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:25.851 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.849+0000 7f394b6ae700 1 --2- 172.21.15.59:0/706173838 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3944142290 0x7f3944136c60 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f393c0099f0 tx=0x7f393c019000 comp rx=0 tx=0).stop 2023-09-30T02:09:25.851 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.849+0000 7f394b6ae700 1 -- 172.21.15.59:0/706173838 shutdown_connections 2023-09-30T02:09:25.851 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.849+0000 7f394b6ae700 1 --2- 172.21.15.59:0/706173838 >> v2:172.21.15.110:6800/115870 conn(0x7f39280688c0 0x7f392806ad70 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:25.851 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.849+0000 7f394b6ae700 1 --2- 172.21.15.59:0/706173838 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3944142290 0x7f3944136c60 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:25.851 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.849+0000 7f394b6ae700 1 --2- 172.21.15.59:0/706173838 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39440a5b70 0x7f39441403a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:25.852 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.849+0000 7f394b6ae700 1 --2- 172.21.15.59:0/706173838 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f39440b1500 0x7f394413e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:25.852 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.849+0000 7f394b6ae700 1 -- 172.21.15.59:0/706173838 >> 172.21.15.59:0/706173838 conn(0x7f39440631f0 msgr2=0x7f39440635c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:09:25.852 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.849+0000 7f394b6ae700 1 -- 172.21.15.59:0/706173838 shutdown_connections 2023-09-30T02:09:25.852 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.849+0000 7f394b6ae700 1 -- 172.21.15.59:0/706173838 wait complete. 2023-09-30T02:09:25.852 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:25.849+0000 7f394b6ae700 1 librados: shutdown 2023-09-30T02:09:28.864 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:09:29.004 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.004+0000 7f446c8d2700 10 monclient: get_monmap_and_config 2023-09-30T02:09:29.005 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.004+0000 7f446c8d2700 10 monclient: build_initial_monmap 2023-09-30T02:09:29.005 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.004+0000 7f446c8d2700 10 monclient: monmap: 2023-09-30T02:09:29.005 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:09:29.005 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:29.005 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:09:29.005693+0000 2023-09-30T02:09:29.005 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:09:29.005693+0000 2023-09-30T02:09:29.005 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:09:29.005 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:29.005 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:09:29.005 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:09:29.006 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:09:29.006 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:09:29.006 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:09:29.006 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:29.007 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.004+0000 7f446c8d2700 1 Processor -- start 2023-09-30T02:09:29.007 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.005+0000 7f446c8d2700 1 -- start start 2023-09-30T02:09:29.007 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.005+0000 7f446c8d2700 10 monclient: init 2023-09-30T02:09:29.007 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.006+0000 7f446c8d2700 10 monclient: _reopen_session rank -1 2023-09-30T02:09:29.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.006+0000 7f446c8d2700 10 monclient: _add_conns ranks=[3,4,0,2,1] 2023-09-30T02:09:29.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.006+0000 7f446c8d2700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f44680a5b70 0x7f44680b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:29.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.006+0000 7f446c8d2700 10 monclient(hunting): picked mon.noname-a con 0x7f44680a5b70 addr v2:172.21.15.59:3300/0 2023-09-30T02:09:29.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.006+0000 7f446c8d2700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f44680a5f40 0x7f4468141b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:29.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.006+0000 7f446c8d2700 10 monclient(hunting): picked mon.noname-c con 0x7f44680a5f40 addr v2:172.21.15.110:3300/0 2023-09-30T02:09:29.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.006+0000 7f446c8d2700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f44681420c0 addr v1:172.21.15.59:6789/0 2023-09-30T02:09:29.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.006+0000 7f446c8d2700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44680a75a0 con 0x7f44681420c0 2023-09-30T02:09:29.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.006+0000 7f446c8d2700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:29.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.006+0000 7f446c8d2700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44680a63e0 con 0x7f44680a5b70 2023-09-30T02:09:29.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.006+0000 7f446c8d2700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:29.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.006+0000 7f446c8d2700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44680af3f0 con 0x7f44680a5f40 2023-09-30T02:09:29.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.006+0000 7f446c8d2700 10 monclient(hunting): _renew_subs 2023-09-30T02:09:29.009 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.006+0000 7f446c8d2700 10 monclient(hunting): authenticate will time out at 2014.309421s 2023-09-30T02:09:29.009 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.007+0000 7f4465d9b700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f44680a5f40 0x7f4468141b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:29.009 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.007+0000 7f4465d9b700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f44680a5f40 0x7f4468141b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:55160/0 (socket says 172.21.15.59:55160) 2023-09-30T02:09:29.009 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.007+0000 7f4465d9b700 1 -- 172.21.15.59:0/648003699 learned_addr learned my addr 172.21.15.59:0/648003699 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:09:29.009 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.007+0000 7f4465d9b700 10 monclient(hunting): get_auth_request con 0x7f44680a5f40 auth_method 0 2023-09-30T02:09:29.009 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.007+0000 7f4465d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:09:29.009 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.007+0000 7f4465d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:29.009 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.007+0000 7f4465d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:09:29.010 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.008+0000 7f4465d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:09:29.010 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.008+0000 7f4465d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:09:29.010 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.008+0000 7f4465d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:09:29.011 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.010+0000 7f4465d9b700 10 monclient(hunting): handle_auth_done global_id 4663 payload 274 2023-09-30T02:09:29.011 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.010+0000 7f4465d9b700 1 -- 172.21.15.59:0/648003699 >> v2:172.21.15.59:3300/0 conn(0x7f44680a5b70 msgr2=0x7f44680b1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:09:29.011 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.010+0000 7f4465d9b700 1 --2- 172.21.15.59:0/648003699 >> v2:172.21.15.59:3300/0 conn(0x7f44680a5b70 0x7f44680b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:29.011 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.010+0000 7f4465d9b700 1 -- 172.21.15.59:0/648003699 >> v1:172.21.15.59:6789/0 conn(0x7f44681420c0 legacy=0x7f446813e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:09:29.011 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.010+0000 7f4465d9b700 10 monclient: _finish_hunting 0 2023-09-30T02:09:29.011 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.010+0000 7f4465d9b700 1 monclient: found mon.noname-c 2023-09-30T02:09:29.011 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.010+0000 7f4465d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:09:29.011 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.010+0000 7f4465d9b700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3300/0 2023-09-30T02:09:29.011 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.010+0000 7f4465d9b700 1 -- 172.21.15.59:0/648003699 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44681424e0 con 0x7f44680a5f40 2023-09-30T02:09:29.013 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.012+0000 7f4465d9b700 1 --2- 172.21.15.59:0/648003699 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f44680a5f40 0x7f4468141b80 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f44680b1fc0 tx=0x7f4450009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f00bbbc7e47f682d server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:09:29.017 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.017+0000 7f446559a700 1 -- 172.21.15.59:0/648003699 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f445000b820 con 0x7f44680a5f40 2023-09-30T02:09:29.017 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.017+0000 7f446559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:09:29.017 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.017+0000 7f446559a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:09:29.018 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.017+0000 7f446559a700 10 monclient: dump: 2023-09-30T02:09:29.018 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:09:29.018 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:29.018 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:29.018 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:29.018 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:09:29.018 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:29.018 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:09:29.018 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:09:29.018 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:09:29.018 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:29.019 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.017+0000 7f446559a700 10 monclient: _finish_auth 0 2023-09-30T02:09:29.019 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.017+0000 7f446559a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:09:29.019 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.017+0000 7f446559a700 1 -- 172.21.15.59:0/648003699 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f445000b980 con 0x7f44680a5f40 2023-09-30T02:09:29.019 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.017+0000 7f446c8d2700 5 monclient: authenticate success, global_id 4663 2023-09-30T02:09:29.019 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.017+0000 7f446c8d2700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:09:29.019 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.017+0000 7f446559a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:09:29.019 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.017+0000 7f446559a700 1 -- 172.21.15.59:0/648003699 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f445000bd20 con 0x7f44680a5f40 2023-09-30T02:09:29.019 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.017+0000 7f446c8d2700 10 monclient: get_monmap_and_config success 2023-09-30T02:09:29.019 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.017+0000 7f446c8d2700 4 set_mon_vals no callback set 2023-09-30T02:09:29.019 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.017+0000 7f446559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:09:29.019 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.017+0000 7f446559a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:09:29.020 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.017+0000 7f446559a700 10 monclient: dump: 2023-09-30T02:09:29.020 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:09:29.020 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:29.020 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:29.020 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:29.020 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:09:29.020 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:29.020 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:09:29.020 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:09:29.020 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:09:29.020 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:29.023 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.017+0000 7f446c8d2700 10 monclient: shutdown 2023-09-30T02:09:29.023 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.017+0000 7f446c8d2700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:09:29.024 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.017+0000 7f446c8d2700 1 -- 172.21.15.59:0/648003699 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f44680a5f40 msgr2=0x7f4468141b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:29.024 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.017+0000 7f446c8d2700 1 --2- 172.21.15.59:0/648003699 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f44680a5f40 0x7f4468141b80 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f44680b1fc0 tx=0x7f4450009250 comp rx=0 tx=0).stop 2023-09-30T02:09:29.024 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.018+0000 7f446c8d2700 1 -- 172.21.15.59:0/648003699 shutdown_connections 2023-09-30T02:09:29.024 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.018+0000 7f446c8d2700 1 --2- 172.21.15.59:0/648003699 >> v2:172.21.15.59:3300/0 conn(0x7f44680a5b70 0x7f44680b1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:29.024 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.018+0000 7f446c8d2700 1 --2- 172.21.15.59:0/648003699 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f44680a5f40 0x7f4468141b80 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:29.024 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.018+0000 7f446c8d2700 1 -- 172.21.15.59:0/648003699 >> 172.21.15.59:0/648003699 conn(0x7f44680631f0 msgr2=0x7f44680635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:09:29.024 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.018+0000 7f446c8d2700 1 -- 172.21.15.59:0/648003699 shutdown_connections 2023-09-30T02:09:29.024 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.018+0000 7f446c8d2700 1 -- 172.21.15.59:0/648003699 wait complete. 2023-09-30T02:09:29.024 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.018+0000 7f446c8d2700 10 monclient: build_initial_monmap 2023-09-30T02:09:29.024 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.018+0000 7f446c8d2700 10 monclient: monmap: 2023-09-30T02:09:29.024 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:09:29.024 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:09:29.025 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:09:29.025 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:09:29.025 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:09:29.025 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:09:29.025 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:09:29.025 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:09:29.025 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:09:29.025 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:29.025 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.019+0000 7f446c8d2700 1 librados: starting msgr at 2023-09-30T02:09:29.025 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.019+0000 7f446c8d2700 1 librados: starting objecter 2023-09-30T02:09:29.025 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.019+0000 7f446c8d2700 1 Processor -- start 2023-09-30T02:09:29.025 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.019+0000 7f446c8d2700 1 -- start start 2023-09-30T02:09:29.026 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.019+0000 7f446c8d2700 1 librados: setting wanted keys 2023-09-30T02:09:29.026 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.019+0000 7f446c8d2700 1 librados: calling monclient init 2023-09-30T02:09:29.026 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.019+0000 7f446c8d2700 10 monclient: init 2023-09-30T02:09:29.026 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.019+0000 7f446c8d2700 10 monclient: _reopen_session rank -1 2023-09-30T02:09:29.026 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.019+0000 7f446c8d2700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T02:09:29.026 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.019+0000 7f446c8d2700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44680a5b70 0x7f446813b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:29.026 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.019+0000 7f446c8d2700 10 monclient(hunting): picked mon.noname-a con 0x7f44680a5b70 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:09:29.026 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.019+0000 7f446c8d2700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f44680a5f40 0x7f44680a9c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:29.026 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.019+0000 7f446c8d2700 10 monclient(hunting): picked mon.noname-b con 0x7f44680a5f40 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:09:29.026 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.019+0000 7f446c8d2700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f44681420c0 0x7f44681512f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:29.026 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.019+0000 7f446c8d2700 10 monclient(hunting): picked mon.noname-c con 0x7f44681420c0 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:09:29.026 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.019+0000 7f446c8d2700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:29.027 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.019+0000 7f446c8d2700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44680aa3f0 con 0x7f44680a5b70 2023-09-30T02:09:29.027 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.019+0000 7f446c8d2700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:29.027 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.019+0000 7f446c8d2700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44680aba10 con 0x7f44680a5f40 2023-09-30T02:09:29.027 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.019+0000 7f446c8d2700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:29.027 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.019+0000 7f446c8d2700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44680aa880 con 0x7f44681420c0 2023-09-30T02:09:29.027 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.019+0000 7f446c8d2700 10 monclient(hunting): _renew_subs 2023-09-30T02:09:29.027 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.019+0000 7f446c8d2700 10 monclient(hunting): authenticate will time out at 2014.322898s 2023-09-30T02:09:29.027 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.020+0000 7f4466d9d700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f44681420c0 0x7f44681512f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:29.027 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.020+0000 7f4466d9d700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f44681420c0 0x7f44681512f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:41550/0 (socket says 172.21.15.59:41550) 2023-09-30T02:09:29.027 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.020+0000 7f4466d9d700 1 -- 172.21.15.59:0/1619614368 learned_addr learned my addr 172.21.15.59:0/1619614368 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:09:29.027 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.020+0000 7f4466d9d700 10 monclient(hunting): get_auth_request con 0x7f44681420c0 auth_method 0 2023-09-30T02:09:29.028 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.020+0000 7f4466d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:09:29.028 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.020+0000 7f4466d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:29.028 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.020+0000 7f4466d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:09:29.028 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.020+0000 7f4465d9b700 1 --2- 172.21.15.59:0/1619614368 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f44680a5f40 0x7f44680a9c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:29.028 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.021+0000 7f4465d9b700 10 monclient(hunting): get_auth_request con 0x7f44680a5f40 auth_method 0 2023-09-30T02:09:29.028 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.021+0000 7f4465d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:09:29.028 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.021+0000 7f4465d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:29.028 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.021+0000 7f4465d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:09:29.028 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.021+0000 7f4466d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:09:29.028 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.021+0000 7f4466d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:09:29.029 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.021+0000 7f4466d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:09:29.029 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.021+0000 7f4465d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:09:29.029 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.021+0000 7f4465d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:09:29.029 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.021+0000 7f4465d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:09:29.030 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.025+0000 7f4465d9b700 10 monclient(hunting): handle_auth_done global_id 4666 payload 824 2023-09-30T02:09:29.030 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.025+0000 7f4465d9b700 1 -- 172.21.15.59:0/1619614368 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f44681420c0 msgr2=0x7f44681512f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:29.030 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.025+0000 7f4465d9b700 1 --2- 172.21.15.59:0/1619614368 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f44681420c0 0x7f44681512f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:29.030 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.025+0000 7f4465d9b700 1 -- 172.21.15.59:0/1619614368 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44680a5b70 msgr2=0x7f446813b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:29.030 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.025+0000 7f4465d9b700 1 --2- 172.21.15.59:0/1619614368 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44680a5b70 0x7f446813b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:29.030 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.025+0000 7f4465d9b700 10 monclient: _finish_hunting 0 2023-09-30T02:09:29.031 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.025+0000 7f4465d9b700 1 monclient: found mon.noname-b 2023-09-30T02:09:29.031 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.025+0000 7f4465d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:09:29.031 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.025+0000 7f4465d9b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:09:29.031 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.025+0000 7f4465d9b700 1 -- 172.21.15.59:0/1619614368 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44500094f0 con 0x7f44680a5f40 2023-09-30T02:09:29.031 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.027+0000 7f4465d9b700 1 --2- 172.21.15.59:0/1619614368 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f44680a5f40 0x7f44680a9c90 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f44680b0640 tx=0x7f4450016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:09:29.031 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.027+0000 7f445e7fc700 10 client.?.objecter ms_handle_connect 0x7f44680a5f40 2023-09-30T02:09:29.031 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.027+0000 7f445e7fc700 10 client.?.objecter resend_mon_ops 2023-09-30T02:09:29.032 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.031+0000 7f445e7fc700 1 -- 172.21.15.59:0/1619614368 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4450026480 con 0x7f44680a5f40 2023-09-30T02:09:29.033 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.031+0000 7f445e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:09:29.033 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.031+0000 7f445e7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:09:29.033 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.031+0000 7f445e7fc700 10 monclient: dump: 2023-09-30T02:09:29.033 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:09:29.033 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:29.033 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:29.033 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:29.033 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:09:29.033 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:29.033 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:09:29.033 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:09:29.034 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:09:29.034 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:29.034 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.031+0000 7f445e7fc700 10 monclient: _finish_auth 0 2023-09-30T02:09:29.034 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.031+0000 7f445e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:09:29.034 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.031+0000 7f445e7fc700 1 -- 172.21.15.59:0/1619614368 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f445001f070 con 0x7f44680a5f40 2023-09-30T02:09:29.034 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.031+0000 7f446c8d2700 5 monclient: authenticate success, global_id 4666 2023-09-30T02:09:29.034 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.031+0000 7f446c8d2700 10 monclient: _renew_subs 2023-09-30T02:09:29.034 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.031+0000 7f446c8d2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:09:29.034 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.031+0000 7f446c8d2700 1 -- 172.21.15.59:0/1619614368 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44680a92d0 con 0x7f44680a5f40 2023-09-30T02:09:29.034 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.031+0000 7f445e7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:09:29.034 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.031+0000 7f445e7fc700 1 -- 172.21.15.59:0/1619614368 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4450026850 con 0x7f44680a5f40 2023-09-30T02:09:29.035 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.031+0000 7f445e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:09:29.035 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.031+0000 7f445e7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:09:29.035 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.031+0000 7f445e7fc700 10 monclient: dump: 2023-09-30T02:09:29.035 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:09:29.035 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:29.035 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:29.035 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:29.035 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:09:29.035 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:29.035 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:09:29.035 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:09:29.036 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:09:29.036 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:29.036 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.031+0000 7f4464d99700 4 set_mon_vals no callback set 2023-09-30T02:09:29.036 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.031+0000 7f446c8d2700 10 client.4666.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:09:29.036 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.031+0000 7f446c8d2700 10 monclient: _renew_subs 2023-09-30T02:09:29.036 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.031+0000 7f446c8d2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:09:29.036 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.031+0000 7f446c8d2700 1 -- 172.21.15.59:0/1619614368 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4468151a70 con 0x7f44680a5f40 2023-09-30T02:09:29.036 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.031+0000 7f446c8d2700 1 librados: init done 2023-09-30T02:09:29.036 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.032+0000 7f446c8d2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:09:29.036 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.032+0000 7f446c8d2700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:09:29.036 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.032+0000 7f446c8d2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:09:29.037 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.032+0000 7f446c8d2700 1 -- 172.21.15.59:0/1619614368 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44680aa6d0 con 0x7f44680a5f40 2023-09-30T02:09:29.056 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.056+0000 7f445e7fc700 1 -- 172.21.15.59:0/1619614368 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f445001a440 con 0x7f44680a5f40 2023-09-30T02:09:29.058 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.057+0000 7f445e7fc700 1 --2- 172.21.15.59:0/1619614368 >> v2:172.21.15.110:6800/115870 conn(0x7f4448068960 0x7f444806ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:29.058 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.057+0000 7f445e7fc700 1 -- 172.21.15.59:0/1619614368 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f4450021070 con 0x7f44680a5f40 2023-09-30T02:09:29.058 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.057+0000 7f445e7fc700 10 client.4666.objecter ms_dispatch 0x7f4468001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:09:29.058 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.057+0000 7f445e7fc700 3 client.4666.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:09:29.058 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.057+0000 7f445e7fc700 3 client.4666.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:09:29.058 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.057+0000 7f445e7fc700 20 client.4666.objecter dump_active .. 0 homeless 2023-09-30T02:09:29.058 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.057+0000 7f446659c700 1 --2- 172.21.15.59:0/1619614368 >> v2:172.21.15.110:6800/115870 conn(0x7f4448068960 0x7f444806ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:29.058 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.057+0000 7f446659c700 10 monclient: get_auth_request con 0x7f4448068960 auth_method 0 2023-09-30T02:09:29.058 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.058+0000 7f445e7fc700 10 client.4666.objecter ms_handle_connect 0x7f4448068960 2023-09-30T02:09:29.059 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.058+0000 7f446659c700 1 --2- 172.21.15.59:0/1619614368 >> v2:172.21.15.110:6800/115870 conn(0x7f4448068960 0x7f444806ae10 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f445800acf0 tx=0x7f4458009370 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:09:29.344 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.344+0000 7f445e7fc700 1 -- 172.21.15.59:0/1619614368 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4450053d60 con 0x7f44680a5f40 2023-09-30T02:09:29.345 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.344+0000 7f445e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:09:29.345 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.344+0000 7f445e7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:09:29.600 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.600+0000 7f446c8d2700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:09:29.600 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.600+0000 7f446c8d2700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:09:29.600 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.600+0000 7f446c8d2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:09:29.600 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.600+0000 7f446c8d2700 1 -- 172.21.15.59:0/1619614368 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f44680b1570 con 0x7f44680a5f40 2023-09-30T02:09:29.604 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.605+0000 7f445e7fc700 1 -- 172.21.15.59:0/1619614368 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f4450057380 con 0x7f44680a5f40 2023-09-30T02:09:29.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.605+0000 7f445e7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:09:29.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.605+0000 7f445e7fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:09:29.605 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:09:29.605 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:09:29.607 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.606+0000 7f446c8d2700 10 librados: watch_flush enter 2023-09-30T02:09:29.607 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.606+0000 7f446c8d2700 10 librados: watch_flush exit 2023-09-30T02:09:29.607 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.606+0000 7f446c8d2700 20 client.4666.objecter shutdown clearing up homeless session... 2023-09-30T02:09:29.607 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.606+0000 7f446c8d2700 10 client.4666.objecter successfully canceled tick 2023-09-30T02:09:29.607 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.606+0000 7f446c8d2700 1 -- 172.21.15.59:0/1619614368 >> v2:172.21.15.110:6800/115870 conn(0x7f4448068960 msgr2=0x7f444806ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:29.607 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.606+0000 7f446c8d2700 1 --2- 172.21.15.59:0/1619614368 >> v2:172.21.15.110:6800/115870 conn(0x7f4448068960 0x7f444806ae10 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f445800acf0 tx=0x7f4458009370 comp rx=0 tx=0).stop 2023-09-30T02:09:29.608 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.606+0000 7f446c8d2700 10 monclient: shutdown 2023-09-30T02:09:29.608 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.606+0000 7f446c8d2700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:09:29.608 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.606+0000 7f446c8d2700 1 -- 172.21.15.59:0/1619614368 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f44680a5f40 msgr2=0x7f44680a9c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:29.608 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.606+0000 7f446c8d2700 1 --2- 172.21.15.59:0/1619614368 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f44680a5f40 0x7f44680a9c90 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f44680b0640 tx=0x7f4450016000 comp rx=0 tx=0).stop 2023-09-30T02:09:29.608 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.606+0000 7f446c8d2700 1 -- 172.21.15.59:0/1619614368 shutdown_connections 2023-09-30T02:09:29.608 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.606+0000 7f446c8d2700 1 --2- 172.21.15.59:0/1619614368 >> v2:172.21.15.110:6800/115870 conn(0x7f4448068960 0x7f444806ae10 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:29.608 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.606+0000 7f446c8d2700 1 --2- 172.21.15.59:0/1619614368 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f44681420c0 0x7f44681512f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:29.608 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.606+0000 7f446c8d2700 1 --2- 172.21.15.59:0/1619614368 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44680a5b70 0x7f446813b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:29.608 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.606+0000 7f446c8d2700 1 --2- 172.21.15.59:0/1619614368 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f44680a5f40 0x7f44680a9c90 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:29.608 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.606+0000 7f446c8d2700 1 -- 172.21.15.59:0/1619614368 >> 172.21.15.59:0/1619614368 conn(0x7f44680631f0 msgr2=0x7f4468137c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:09:29.608 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.607+0000 7f446c8d2700 1 -- 172.21.15.59:0/1619614368 shutdown_connections 2023-09-30T02:09:29.608 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.607+0000 7f446c8d2700 1 -- 172.21.15.59:0/1619614368 wait complete. 2023-09-30T02:09:29.608 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:29.607+0000 7f446c8d2700 1 librados: shutdown 2023-09-30T02:09:32.621 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:09:32.759 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.759+0000 7f148cd8a700 10 monclient: get_monmap_and_config 2023-09-30T02:09:32.760 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.759+0000 7f148cd8a700 10 monclient: build_initial_monmap 2023-09-30T02:09:32.760 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.759+0000 7f148cd8a700 10 monclient: monmap: 2023-09-30T02:09:32.760 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:09:32.760 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:32.760 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:09:32.760722+0000 2023-09-30T02:09:32.760 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:09:32.760722+0000 2023-09-30T02:09:32.760 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:09:32.760 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:32.760 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:09:32.760 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:09:32.760 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:09:32.761 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:09:32.761 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:09:32.761 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:32.762 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.759+0000 7f148cd8a700 1 Processor -- start 2023-09-30T02:09:32.762 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.760+0000 7f148cd8a700 1 -- start start 2023-09-30T02:09:32.762 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.760+0000 7f148cd8a700 10 monclient: init 2023-09-30T02:09:32.762 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.760+0000 7f148cd8a700 10 monclient: _reopen_session rank -1 2023-09-30T02:09:32.762 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.760+0000 7f148cd8a700 10 monclient: _add_conns ranks=[3,0,1,4,2] 2023-09-30T02:09:32.762 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.760+0000 7f148cd8a700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f14880a5b70 0x7f14880b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:32.762 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.760+0000 7f148cd8a700 10 monclient(hunting): picked mon.noname-a con 0x7f14880a5b70 addr v2:172.21.15.59:3300/0 2023-09-30T02:09:32.762 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.760+0000 7f148cd8a700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f14880a5f40 addr v1:172.21.15.59:6789/0 2023-09-30T02:09:32.763 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.760+0000 7f148cd8a700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1488142290 addr v1:172.21.15.110:6789/0 2023-09-30T02:09:32.763 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.760+0000 7f148cd8a700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14880a75a0 con 0x7f14880a5f40 2023-09-30T02:09:32.763 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.760+0000 7f148cd8a700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14880af3f0 con 0x7f1488142290 2023-09-30T02:09:32.763 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.760+0000 7f148cd8a700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:32.763 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.760+0000 7f148cd8a700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14880a63e0 con 0x7f14880a5b70 2023-09-30T02:09:32.763 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.760+0000 7f148cd8a700 10 monclient(hunting): _renew_subs 2023-09-30T02:09:32.763 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.760+0000 7f148cd8a700 10 monclient(hunting): authenticate will time out at 2018.063914s 2023-09-30T02:09:32.763 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.761+0000 7f148659c700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f14880a5b70 0x7f14880b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:32.763 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.761+0000 7f1486d9d700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f1488142290 0x7f148813af30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.59:56932/0 (socket says 172.21.15.59:56932) 2023-09-30T02:09:32.763 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.761+0000 7f1486d9d700 1 -- 172.21.15.59:0/2052550531 learned_addr learned my addr 172.21.15.59:0/2052550531 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:09:32.764 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.762+0000 7f148659c700 10 monclient(hunting): get_auth_request con 0x7f14880a5b70 auth_method 0 2023-09-30T02:09:32.764 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.762+0000 7f148659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:09:32.764 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.762+0000 7f148659c700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:32.764 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.762+0000 7f148659c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:09:32.765 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.764+0000 7f148659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:09:32.765 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.764+0000 7f148659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:09:32.765 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.764+0000 7f148659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:09:32.765 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.764+0000 7f148559a700 1 -- 172.21.15.59:0/2052550531 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 879793622 0 0) 0x7f14880af3f0 con 0x7f1488142290 2023-09-30T02:09:32.765 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.764+0000 7f148559a700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:32.765 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.764+0000 7f148559a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:09:32.765 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.764+0000 7f148559a700 10 monclient(hunting): my global_id is 4669 2023-09-30T02:09:32.765 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.764+0000 7f148559a700 1 -- 172.21.15.59:0/2052550531 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1470003570 con 0x7f1488142290 2023-09-30T02:09:32.767 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.766+0000 7f148559a700 1 -- 172.21.15.59:0/2052550531 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1401889882 0 0) 0x7f14880a75a0 con 0x7f14880a5f40 2023-09-30T02:09:32.767 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.766+0000 7f148559a700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:32.767 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.766+0000 7f148559a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:09:32.767 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.766+0000 7f148559a700 10 monclient(hunting): my global_id is 4650 2023-09-30T02:09:32.767 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.766+0000 7f148559a700 1 -- 172.21.15.59:0/2052550531 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14880af3f0 con 0x7f14880a5f40 2023-09-30T02:09:32.771 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.768+0000 7f148559a700 1 -- 172.21.15.59:0/2052550531 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3972508124 0 0) 0x7f1470003570 con 0x7f1488142290 2023-09-30T02:09:32.771 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.768+0000 7f148559a700 1 -- 172.21.15.59:0/2052550531 >> v2:172.21.15.59:3300/0 conn(0x7f14880a5b70 msgr2=0x7f14880b1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:09:32.771 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.769+0000 7f148559a700 1 --2- 172.21.15.59:0/2052550531 >> v2:172.21.15.59:3300/0 conn(0x7f14880a5b70 0x7f14880b1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:32.771 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.769+0000 7f148559a700 1 -- 172.21.15.59:0/2052550531 >> v1:172.21.15.59:6789/0 conn(0x7f14880a5f40 legacy=0x7f1488141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:32.771 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.769+0000 7f148559a700 10 monclient: _finish_hunting 0 2023-09-30T02:09:32.771 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.769+0000 7f148559a700 1 monclient: found mon.noname-c-legacy 2023-09-30T02:09:32.771 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.769+0000 7f148559a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:09:32.771 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.769+0000 7f148559a700 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.110:6789/0 2023-09-30T02:09:32.771 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.769+0000 7f148559a700 1 -- 172.21.15.59:0/2052550531 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f148813b640 con 0x7f1488142290 2023-09-30T02:09:32.771 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.769+0000 7f148559a700 10 monclient: _finish_auth 0 2023-09-30T02:09:32.772 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.769+0000 7f148559a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:09:32.772 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.769+0000 7f148cd8a700 5 monclient: authenticate success, global_id 4669 2023-09-30T02:09:32.772 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.769+0000 7f148559a700 1 -- 172.21.15.59:0/2052550531 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2174283827 0 0) 0x7f14800017f0 con 0x7f1488142290 2023-09-30T02:09:32.772 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.769+0000 7f148cd8a700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:09:32.772 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.769+0000 7f148559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:09:32.772 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.769+0000 7f148559a700 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2023-09-30T02:09:32.772 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.769+0000 7f148559a700 10 monclient: dump: 2023-09-30T02:09:32.772 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:09:32.772 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:32.772 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:32.772 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:32.773 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:09:32.773 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:32.773 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:09:32.773 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:09:32.773 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:09:32.773 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:32.773 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.769+0000 7f148559a700 1 monclient: mon.b has (v2) addrs [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] but i'm connected to v1:172.21.15.110:6789/0, reconnecting 2023-09-30T02:09:32.773 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.769+0000 7f148559a700 10 monclient: _reopen_session rank -1 2023-09-30T02:09:32.773 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.769+0000 7f148559a700 1 -- 172.21.15.59:0/2052550531 >> v1:172.21.15.110:6789/0 conn(0x7f1488142290 legacy=0x7f148813af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:32.773 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.769+0000 7f148559a700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T02:09:32.773 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.769+0000 7f148559a700 1 --2- 172.21.15.59:0/2052550531 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1470003c40 0x7f1470004030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:32.773 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.769+0000 7f148559a700 10 monclient(hunting): picked mon.b con 0x7f1470003c40 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:09:32.774 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.769+0000 7f148559a700 1 --2- 172.21.15.59:0/2052550531 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1470004810 0x7f1470004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:32.774 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.769+0000 7f148559a700 10 monclient(hunting): picked mon.c con 0x7f1470004810 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:09:32.774 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.769+0000 7f148559a700 1 --2- 172.21.15.59:0/2052550531 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1470005210 0x7f147000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:32.774 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.769+0000 7f148559a700 10 monclient(hunting): picked mon.a con 0x7f1470005210 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:09:32.774 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.769+0000 7f148559a700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:32.774 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.769+0000 7f148559a700 1 -- 172.21.15.59:0/2052550531 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14880a63e0 con 0x7f1470005210 2023-09-30T02:09:32.774 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.769+0000 7f148559a700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:32.774 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.769+0000 7f148559a700 1 -- 172.21.15.59:0/2052550531 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f147000c1b0 con 0x7f1470003c40 2023-09-30T02:09:32.774 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.769+0000 7f148559a700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:32.774 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.769+0000 7f148559a700 1 -- 172.21.15.59:0/2052550531 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f147000c370 con 0x7f1470004810 2023-09-30T02:09:32.774 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.769+0000 7f148559a700 10 monclient(hunting): _renew_subs 2023-09-30T02:09:32.774 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.769+0000 7f148559a700 10 monclient(hunting): _finish_auth 0 2023-09-30T02:09:32.775 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.769+0000 7f148cd8a700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-09-30T02:09:32.775 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.769+0000 7f1485d9b700 1 --2- 172.21.15.59:0/2052550531 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1470004810 0x7f1470004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:32.775 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.770+0000 7f1485d9b700 10 monclient(hunting): get_auth_request con 0x7f1470004810 auth_method 0 2023-09-30T02:09:32.775 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.770+0000 7f1485d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:09:32.775 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.770+0000 7f1485d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:32.775 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.770+0000 7f1485d9b700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T02:09:32.775 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.770+0000 7f1485d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:09:32.775 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.770+0000 7f1485d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:09:32.775 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.770+0000 7f1485d9b700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T02:09:32.777 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.772+0000 7f148659c700 1 --2- 172.21.15.59:0/2052550531 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1470003c40 0x7f1470004030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:32.777 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.773+0000 7f148659c700 10 monclient(hunting): get_auth_request con 0x7f1470003c40 auth_method 0 2023-09-30T02:09:32.777 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.773+0000 7f148659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:09:32.777 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.773+0000 7f148659c700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:32.777 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.773+0000 7f148659c700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T02:09:32.777 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.773+0000 7f148659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:09:32.777 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.773+0000 7f148659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:09:32.777 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.773+0000 7f148659c700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T02:09:32.777 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.773+0000 7f1485d9b700 10 monclient(hunting): handle_auth_done global_id 4669 payload 293 2023-09-30T02:09:32.777 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.773+0000 7f1485d9b700 1 -- 172.21.15.59:0/2052550531 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1470003c40 msgr2=0x7f1470004030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:09:32.777 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.773+0000 7f1485d9b700 1 --2- 172.21.15.59:0/2052550531 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1470003c40 0x7f1470004030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:32.778 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.773+0000 7f1485d9b700 1 -- 172.21.15.59:0/2052550531 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1470005210 msgr2=0x7f147000ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:09:32.778 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.773+0000 7f1485d9b700 1 --2- 172.21.15.59:0/2052550531 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1470005210 0x7f147000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:32.778 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.773+0000 7f1485d9b700 10 monclient: _finish_hunting 0 2023-09-30T02:09:32.778 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.773+0000 7f1485d9b700 1 monclient: found mon.c 2023-09-30T02:09:32.778 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.773+0000 7f1486d9d700 1 -- 172.21.15.59:0/2052550531 reap_dead start 2023-09-30T02:09:32.778 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.773+0000 7f1485d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:09:32.778 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.773+0000 7f1485d9b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:09:32.778 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.773+0000 7f1485d9b700 1 -- 172.21.15.59:0/2052550531 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1470003570 con 0x7f1470004810 2023-09-30T02:09:32.778 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.773+0000 7f1485d9b700 10 monclient: _finish_auth 0 2023-09-30T02:09:32.778 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.773+0000 7f1485d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:09:32.778 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.776+0000 7f1485d9b700 1 --2- 172.21.15.59:0/2052550531 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1470004810 0x7f1470004c80 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f14780019e0 tx=0x7f1478007d30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b20d1592a9d4f7ac server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:09:32.782 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.781+0000 7f148559a700 1 -- 172.21.15.59:0/2052550531 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f147800c5d0 con 0x7f1470004810 2023-09-30T02:09:32.782 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.781+0000 7f148559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:09:32.782 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.781+0000 7f148559a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:09:32.782 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.781+0000 7f148559a700 10 monclient: dump: 2023-09-30T02:09:32.782 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:09:32.782 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:32.783 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:32.783 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:32.783 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:09:32.783 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:32.783 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:09:32.783 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:09:32.783 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:09:32.783 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:32.783 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.781+0000 7f148559a700 1 -- 172.21.15.59:0/2052550531 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f147800c730 con 0x7f1470004810 2023-09-30T02:09:32.783 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.781+0000 7f148559a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:09:32.783 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.781+0000 7f148559a700 1 -- 172.21.15.59:0/2052550531 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f147800cad0 con 0x7f1470004810 2023-09-30T02:09:32.784 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.781+0000 7f148559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:09:32.784 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.781+0000 7f148559a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:09:32.784 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.781+0000 7f148559a700 10 monclient: dump: 2023-09-30T02:09:32.784 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:09:32.784 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:32.784 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:32.784 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:32.784 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:09:32.784 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:32.784 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:09:32.784 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:09:32.784 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:09:32.784 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:32.785 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.782+0000 7f148cd8a700 10 monclient: get_monmap_and_config success 2023-09-30T02:09:32.785 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.782+0000 7f148cd8a700 4 set_mon_vals no callback set 2023-09-30T02:09:32.786 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.783+0000 7f148cd8a700 10 monclient: shutdown 2023-09-30T02:09:32.786 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.783+0000 7f148cd8a700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:09:32.786 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.783+0000 7f148cd8a700 1 -- 172.21.15.59:0/2052550531 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1470004810 msgr2=0x7f1470004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:32.786 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.783+0000 7f148cd8a700 1 --2- 172.21.15.59:0/2052550531 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1470004810 0x7f1470004c80 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f14780019e0 tx=0x7f1478007d30 comp rx=0 tx=0).stop 2023-09-30T02:09:32.786 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.783+0000 7f148cd8a700 1 -- 172.21.15.59:0/2052550531 shutdown_connections 2023-09-30T02:09:32.786 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.783+0000 7f148cd8a700 1 --2- 172.21.15.59:0/2052550531 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1470004810 0x7f1470004c80 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:32.786 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.783+0000 7f148cd8a700 1 -- 172.21.15.59:0/2052550531 >> 172.21.15.59:0/2052550531 conn(0x7f14880631f0 msgr2=0x7f14880635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:09:32.786 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.784+0000 7f148cd8a700 1 -- 172.21.15.59:0/2052550531 shutdown_connections 2023-09-30T02:09:32.786 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.784+0000 7f148cd8a700 1 -- 172.21.15.59:0/2052550531 wait complete. 2023-09-30T02:09:32.786 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.785+0000 7f148cd8a700 10 monclient: build_initial_monmap 2023-09-30T02:09:32.787 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.785+0000 7f148cd8a700 10 monclient: monmap: 2023-09-30T02:09:32.787 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:09:32.787 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:09:32.787 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:09:32.787 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:09:32.787 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:09:32.787 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:09:32.787 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:09:32.787 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:09:32.787 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:09:32.787 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:32.787 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.785+0000 7f148cd8a700 1 librados: starting msgr at 2023-09-30T02:09:32.788 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.785+0000 7f148cd8a700 1 librados: starting objecter 2023-09-30T02:09:32.788 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.785+0000 7f148cd8a700 1 Processor -- start 2023-09-30T02:09:32.788 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.785+0000 7f148cd8a700 1 -- start start 2023-09-30T02:09:32.788 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.785+0000 7f148cd8a700 1 librados: setting wanted keys 2023-09-30T02:09:32.788 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.785+0000 7f148cd8a700 1 librados: calling monclient init 2023-09-30T02:09:32.788 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.785+0000 7f148cd8a700 10 monclient: init 2023-09-30T02:09:32.790 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.786+0000 7f148cd8a700 10 monclient: _reopen_session rank -1 2023-09-30T02:09:32.790 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.786+0000 7f148cd8a700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T02:09:32.790 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.786+0000 7f148cd8a700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1470004810 0x7f148813e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:32.790 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.786+0000 7f148cd8a700 10 monclient(hunting): picked mon.noname-c con 0x7f1470004810 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:09:32.790 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.786+0000 7f148cd8a700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f148806e900 0x7f14881363e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:32.790 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.786+0000 7f148cd8a700 10 monclient(hunting): picked mon.noname-b con 0x7f148806e900 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:09:32.791 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.786+0000 7f148cd8a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1488136920 0x7f1488155250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:32.791 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.786+0000 7f148cd8a700 10 monclient(hunting): picked mon.noname-a con 0x7f1488136920 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:09:32.791 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.786+0000 7f148cd8a700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:32.791 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.786+0000 7f148cd8a700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f148806e270 con 0x7f1488136920 2023-09-30T02:09:32.791 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.786+0000 7f148cd8a700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:32.791 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.786+0000 7f148cd8a700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f148806d240 con 0x7f148806e900 2023-09-30T02:09:32.791 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.786+0000 7f148cd8a700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:32.791 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.786+0000 7f148cd8a700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f148806e4e0 con 0x7f1470004810 2023-09-30T02:09:32.791 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.786+0000 7f148cd8a700 10 monclient(hunting): _renew_subs 2023-09-30T02:09:32.791 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.786+0000 7f148cd8a700 10 monclient(hunting): authenticate will time out at 2018.089402s 2023-09-30T02:09:32.791 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.786+0000 7f148659c700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1470004810 0x7f148813e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:32.791 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.787+0000 7f1486d9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1488136920 0x7f1488155250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:32.791 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.787+0000 7f1485d9b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f148806e900 0x7f14881363e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:32.792 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.787+0000 7f148659c700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1470004810 0x7f148813e160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:41566/0 (socket says 172.21.15.59:41566) 2023-09-30T02:09:32.792 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.787+0000 7f148659c700 1 -- 172.21.15.59:0/1523568817 learned_addr learned my addr 172.21.15.59:0/1523568817 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:09:32.792 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.787+0000 7f148659c700 10 monclient(hunting): get_auth_request con 0x7f1470004810 auth_method 0 2023-09-30T02:09:32.792 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.787+0000 7f148659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:09:32.792 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.787+0000 7f148659c700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:32.792 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.787+0000 7f148659c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:09:32.792 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.787+0000 7f1485d9b700 10 monclient(hunting): get_auth_request con 0x7f148806e900 auth_method 0 2023-09-30T02:09:32.792 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.787+0000 7f1485d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:09:32.792 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.787+0000 7f1485d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:32.792 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.787+0000 7f1485d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:09:32.792 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.787+0000 7f1486d9d700 10 monclient(hunting): get_auth_request con 0x7f1488136920 auth_method 0 2023-09-30T02:09:32.792 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.787+0000 7f1486d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:09:32.793 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.787+0000 7f1486d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:32.793 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.787+0000 7f1486d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:09:32.793 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.787+0000 7f148659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:09:32.793 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.787+0000 7f148659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:09:32.793 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.788+0000 7f148659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:09:32.793 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.788+0000 7f1485d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:09:32.793 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.788+0000 7f1485d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:09:32.793 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.788+0000 7f1485d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:09:32.793 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.788+0000 7f1486d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:09:32.793 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.788+0000 7f1486d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:09:32.793 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.788+0000 7f1486d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:09:32.794 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.791+0000 7f1485d9b700 10 monclient(hunting): handle_auth_done global_id 4672 payload 824 2023-09-30T02:09:32.794 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.791+0000 7f1485d9b700 1 -- 172.21.15.59:0/1523568817 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1470004810 msgr2=0x7f148813e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:32.794 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.791+0000 7f1485d9b700 1 --2- 172.21.15.59:0/1523568817 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1470004810 0x7f148813e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:32.794 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.791+0000 7f1485d9b700 1 -- 172.21.15.59:0/1523568817 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1488136920 msgr2=0x7f1488155250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:32.794 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.791+0000 7f1485d9b700 1 --2- 172.21.15.59:0/1523568817 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1488136920 0x7f1488155250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:32.794 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.791+0000 7f1485d9b700 10 monclient: _finish_hunting 0 2023-09-30T02:09:32.795 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.791+0000 7f1485d9b700 1 monclient: found mon.noname-b 2023-09-30T02:09:32.795 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.791+0000 7f1485d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:09:32.795 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.791+0000 7f1485d9b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:09:32.795 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.791+0000 7f1485d9b700 1 -- 172.21.15.59:0/1523568817 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f148806be80 con 0x7f148806e900 2023-09-30T02:09:32.795 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.795+0000 7f1485d9b700 1 --2- 172.21.15.59:0/1523568817 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f148806e900 0x7f14881363e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1478014000 tx=0x7f1478015000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:09:32.795 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.795+0000 7f14767fc700 10 client.?.objecter ms_handle_connect 0x7f148806e900 2023-09-30T02:09:32.795 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.795+0000 7f14767fc700 10 client.?.objecter resend_mon_ops 2023-09-30T02:09:32.800 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.799+0000 7f14767fc700 1 -- 172.21.15.59:0/1523568817 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f147800c5d0 con 0x7f148806e900 2023-09-30T02:09:32.800 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.799+0000 7f14767fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:09:32.801 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.799+0000 7f14767fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:09:32.801 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.799+0000 7f14767fc700 10 monclient: dump: 2023-09-30T02:09:32.801 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:09:32.801 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:32.801 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:32.801 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:32.801 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:09:32.801 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:32.801 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:09:32.801 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:09:32.801 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:09:32.801 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:32.802 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.799+0000 7f14767fc700 10 monclient: _finish_auth 0 2023-09-30T02:09:32.802 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.799+0000 7f14767fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:09:32.802 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.799+0000 7f14767fc700 1 -- 172.21.15.59:0/1523568817 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1478001360 con 0x7f148806e900 2023-09-30T02:09:32.802 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.799+0000 7f148cd8a700 5 monclient: authenticate success, global_id 4672 2023-09-30T02:09:32.802 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.799+0000 7f148cd8a700 10 monclient: _renew_subs 2023-09-30T02:09:32.802 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.799+0000 7f148cd8a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:09:32.802 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.799+0000 7f148cd8a700 1 -- 172.21.15.59:0/1523568817 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f148806c160 con 0x7f148806e900 2023-09-30T02:09:32.802 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.799+0000 7f14767fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:09:32.802 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.799+0000 7f14767fc700 1 -- 172.21.15.59:0/1523568817 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f147800ce80 con 0x7f148806e900 2023-09-30T02:09:32.802 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.799+0000 7f14767fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:09:32.802 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.799+0000 7f14767fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:09:32.803 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.799+0000 7f14767fc700 10 monclient: dump: 2023-09-30T02:09:32.803 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:09:32.803 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:32.803 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:32.803 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:32.803 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:09:32.803 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:32.803 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:09:32.803 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:09:32.803 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:09:32.803 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:32.803 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.799+0000 7f1484d99700 4 set_mon_vals no callback set 2023-09-30T02:09:32.804 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.799+0000 7f148cd8a700 10 client.4672.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:09:32.804 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.799+0000 7f148cd8a700 10 monclient: _renew_subs 2023-09-30T02:09:32.804 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.799+0000 7f148cd8a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:09:32.804 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.799+0000 7f148cd8a700 1 -- 172.21.15.59:0/1523568817 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1488136d80 con 0x7f148806e900 2023-09-30T02:09:32.804 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.799+0000 7f148cd8a700 1 librados: init done 2023-09-30T02:09:32.804 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.800+0000 7f148cd8a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:09:32.804 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.800+0000 7f148cd8a700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:09:32.804 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.800+0000 7f148cd8a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:09:32.804 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:32.800+0000 7f148cd8a700 1 -- 172.21.15.59:0/1523568817 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f148813bce0 con 0x7f148806e900 2023-09-30T02:09:33.059 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:33.055+0000 7f14767fc700 1 -- 172.21.15.59:0/1523568817 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f1478019070 con 0x7f148806e900 2023-09-30T02:09:33.059 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:33.055+0000 7f14767fc700 1 --2- 172.21.15.59:0/1523568817 >> v2:172.21.15.110:6800/115870 conn(0x7f14680689e0 0x7f146806ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:33.059 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:33.055+0000 7f14767fc700 1 -- 172.21.15.59:0/1523568817 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f1478085740 con 0x7f148806e900 2023-09-30T02:09:33.059 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:33.055+0000 7f14767fc700 10 client.4672.objecter ms_dispatch 0x7f1488001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:09:33.059 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:33.055+0000 7f14767fc700 3 client.4672.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:09:33.059 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:33.055+0000 7f14767fc700 3 client.4672.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:09:33.060 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:33.056+0000 7f14767fc700 20 client.4672.objecter dump_active .. 0 homeless 2023-09-30T02:09:33.060 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:33.056+0000 7f148659c700 1 --2- 172.21.15.59:0/1523568817 >> v2:172.21.15.110:6800/115870 conn(0x7f14680689e0 0x7f146806ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:33.060 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:33.056+0000 7f148659c700 10 monclient: get_auth_request con 0x7f14680689e0 auth_method 0 2023-09-30T02:09:33.060 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:33.056+0000 7f148659c700 1 --2- 172.21.15.59:0/1523568817 >> v2:172.21.15.110:6800/115870 conn(0x7f14680689e0 0x7f146806ae90 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f147c006ac0 tx=0x7f147c0091a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:09:33.060 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:33.056+0000 7f14767fc700 10 client.4672.objecter ms_handle_connect 0x7f14680689e0 2023-09-30T02:09:33.064 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:33.064+0000 7f14767fc700 1 -- 172.21.15.59:0/1523568817 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1478053060 con 0x7f148806e900 2023-09-30T02:09:33.064 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:33.064+0000 7f14767fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:09:33.065 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:33.064+0000 7f14767fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:09:33.328 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:33.328+0000 7f148cd8a700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:09:33.329 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:33.328+0000 7f148cd8a700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:09:33.329 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:33.328+0000 7f148cd8a700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:09:33.329 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:33.328+0000 7f148cd8a700 1 -- 172.21.15.59:0/1523568817 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f148806e620 con 0x7f148806e900 2023-09-30T02:09:33.333 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:33.333+0000 7f14767fc700 1 -- 172.21.15.59:0/1523568817 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f148806e620 con 0x7f148806e900 2023-09-30T02:09:33.333 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:33.333+0000 7f14767fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:09:33.333 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:33.333+0000 7f14767fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:09:33.333 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:09:33.333 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:09:33.336 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:33.335+0000 7f148cd8a700 10 librados: watch_flush enter 2023-09-30T02:09:33.336 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:33.335+0000 7f148cd8a700 10 librados: watch_flush exit 2023-09-30T02:09:33.336 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:33.335+0000 7f148cd8a700 20 client.4672.objecter shutdown clearing up homeless session... 2023-09-30T02:09:33.336 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:33.335+0000 7f148cd8a700 10 client.4672.objecter successfully canceled tick 2023-09-30T02:09:33.336 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:33.335+0000 7f148cd8a700 1 -- 172.21.15.59:0/1523568817 >> v2:172.21.15.110:6800/115870 conn(0x7f14680689e0 msgr2=0x7f146806ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:33.336 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:33.335+0000 7f148cd8a700 1 --2- 172.21.15.59:0/1523568817 >> v2:172.21.15.110:6800/115870 conn(0x7f14680689e0 0x7f146806ae90 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f147c006ac0 tx=0x7f147c0091a0 comp rx=0 tx=0).stop 2023-09-30T02:09:33.337 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:33.335+0000 7f148cd8a700 10 monclient: shutdown 2023-09-30T02:09:33.337 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:33.335+0000 7f148cd8a700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:09:33.337 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:33.335+0000 7f148cd8a700 1 -- 172.21.15.59:0/1523568817 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f148806e900 msgr2=0x7f14881363e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:33.337 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:33.335+0000 7f148cd8a700 1 --2- 172.21.15.59:0/1523568817 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f148806e900 0x7f14881363e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1478014000 tx=0x7f1478015000 comp rx=0 tx=0).stop 2023-09-30T02:09:33.337 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:33.335+0000 7f148cd8a700 1 -- 172.21.15.59:0/1523568817 shutdown_connections 2023-09-30T02:09:33.337 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:33.335+0000 7f148cd8a700 1 --2- 172.21.15.59:0/1523568817 >> v2:172.21.15.110:6800/115870 conn(0x7f14680689e0 0x7f146806ae90 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:33.337 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:33.335+0000 7f148cd8a700 1 --2- 172.21.15.59:0/1523568817 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1488136920 0x7f1488155250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:33.338 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:33.335+0000 7f148cd8a700 1 --2- 172.21.15.59:0/1523568817 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1470004810 0x7f148813e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:33.338 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:33.335+0000 7f148cd8a700 1 --2- 172.21.15.59:0/1523568817 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f148806e900 0x7f14881363e0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:33.338 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:33.335+0000 7f148cd8a700 1 -- 172.21.15.59:0/1523568817 >> 172.21.15.59:0/1523568817 conn(0x7f14880631f0 msgr2=0x7f14880b0be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:09:33.338 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:33.335+0000 7f148cd8a700 1 -- 172.21.15.59:0/1523568817 shutdown_connections 2023-09-30T02:09:33.338 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:33.335+0000 7f148cd8a700 1 -- 172.21.15.59:0/1523568817 wait complete. 2023-09-30T02:09:33.338 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:33.335+0000 7f148cd8a700 1 librados: shutdown 2023-09-30T02:09:34.067 INFO:tasks.ceph.mon.a.smithi059.stderr:==00:00:04:43.932 116256== Warning: unimplemented fcntl command: 1036 2023-09-30T02:09:34.073 INFO:tasks.ceph.mon.a.smithi059.stderr:==00:00:04:43.938 116256== Warning: unimplemented fcntl command: 1036 2023-09-30T02:09:34.106 INFO:tasks.ceph.mon.a.smithi059.stderr:==00:00:04:43.971 116256== Warning: unimplemented fcntl command: 1036 2023-09-30T02:09:34.129 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:04:43.994 116257== Warning: unimplemented fcntl command: 1036 2023-09-30T02:09:34.130 INFO:tasks.ceph.mon.b.smithi110.stderr:==00:00:04:43.977 115869== Warning: unimplemented fcntl command: 1036 2023-09-30T02:09:34.132 INFO:tasks.ceph.mon.b.smithi110.stderr:==00:00:04:43.980 115869== Warning: unimplemented fcntl command: 1036 2023-09-30T02:09:34.134 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:04:44.000 116257== Warning: unimplemented fcntl command: 1036 2023-09-30T02:09:34.160 INFO:tasks.ceph.mon.b.smithi110.stderr:==00:00:04:44.008 115869== Warning: unimplemented fcntl command: 1036 2023-09-30T02:09:34.165 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:04:44.030 116257== Warning: unimplemented fcntl command: 1036 2023-09-30T02:09:36.350 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:09:36.488 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.488+0000 7f5e8e8c1700 10 monclient: get_monmap_and_config 2023-09-30T02:09:36.488 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.488+0000 7f5e8e8c1700 10 monclient: build_initial_monmap 2023-09-30T02:09:36.488 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.488+0000 7f5e8e8c1700 10 monclient: monmap: 2023-09-30T02:09:36.488 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:09:36.489 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:36.489 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:09:36.489496+0000 2023-09-30T02:09:36.489 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:09:36.489496+0000 2023-09-30T02:09:36.489 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:09:36.489 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:36.489 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:09:36.489 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:09:36.489 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:09:36.489 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:09:36.489 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:09:36.489 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:36.491 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.488+0000 7f5e8e8c1700 1 Processor -- start 2023-09-30T02:09:36.491 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.488+0000 7f5e8e8c1700 1 -- start start 2023-09-30T02:09:36.491 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.488+0000 7f5e8e8c1700 10 monclient: init 2023-09-30T02:09:36.491 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.489+0000 7f5e8e8c1700 10 monclient: _reopen_session rank -1 2023-09-30T02:09:36.492 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.489+0000 7f5e8e8c1700 10 monclient: _add_conns ranks=[2,4,1,3,0] 2023-09-30T02:09:36.492 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.489+0000 7f5e8e8c1700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5e880a5b70 0x7f5e880b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:36.492 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.489+0000 7f5e8e8c1700 10 monclient(hunting): picked mon.noname-b con 0x7f5e880a5b70 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:09:36.492 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.489+0000 7f5e8e8c1700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e880a5f40 0x7f5e88141b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:36.492 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.489+0000 7f5e8e8c1700 10 monclient(hunting): picked mon.noname-c con 0x7f5e880a5f40 addr v2:172.21.15.110:3300/0 2023-09-30T02:09:36.492 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.489+0000 7f5e8e8c1700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5e881420c0 addr v1:172.21.15.110:6789/0 2023-09-30T02:09:36.492 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.489+0000 7f5e8e8c1700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e880a63b0 con 0x7f5e881420c0 2023-09-30T02:09:36.492 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.489+0000 7f5e8e8c1700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:36.492 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.489+0000 7f5e8e8c1700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e880a75a0 con 0x7f5e880a5f40 2023-09-30T02:09:36.492 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.489+0000 7f5e8e8c1700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:36.492 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.489+0000 7f5e8e8c1700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e880af3f0 con 0x7f5e880a5b70 2023-09-30T02:09:36.492 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.489+0000 7f5e8e8c1700 10 monclient(hunting): _renew_subs 2023-09-30T02:09:36.493 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.489+0000 7f5e8e8c1700 10 monclient(hunting): authenticate will time out at 2021.792713s 2023-09-30T02:09:36.493 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.490+0000 7f5e87fff700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5e880a5b70 0x7f5e880b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:36.493 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.490+0000 7f5e877fe700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f5e880a5f40 0x7f5e88141b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:36.493 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.490+0000 7f5e87fff700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5e880a5b70 0x7f5e880b1500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:41580/0 (socket says 172.21.15.59:41580) 2023-09-30T02:09:36.493 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.490+0000 7f5e87fff700 1 -- 172.21.15.59:0/2822481419 learned_addr learned my addr 172.21.15.59:0/2822481419 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:09:36.493 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.490+0000 7f5e87fff700 10 monclient(hunting): get_auth_request con 0x7f5e880a5b70 auth_method 0 2023-09-30T02:09:36.493 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.490+0000 7f5e87fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:09:36.493 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.490+0000 7f5e87fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:36.493 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.490+0000 7f5e87fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:09:36.493 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.491+0000 7f5e877fe700 10 monclient(hunting): get_auth_request con 0x7f5e880a5f40 auth_method 0 2023-09-30T02:09:36.493 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.491+0000 7f5e877fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:09:36.493 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.491+0000 7f5e877fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:36.494 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.491+0000 7f5e877fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:09:36.494 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.491+0000 7f5e87fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:09:36.494 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.491+0000 7f5e87fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:09:36.494 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.491+0000 7f5e87fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:09:36.495 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.492+0000 7f5e877fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:09:36.495 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.492+0000 7f5e877fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:09:36.495 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.492+0000 7f5e877fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:09:36.495 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.494+0000 7f5e87fff700 10 monclient(hunting): handle_auth_done global_id 4580 payload 274 2023-09-30T02:09:36.495 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.494+0000 7f5e87fff700 1 -- 172.21.15.59:0/2822481419 >> v2:172.21.15.110:3300/0 conn(0x7f5e880a5f40 msgr2=0x7f5e88141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:09:36.495 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.494+0000 7f5e87fff700 1 --2- 172.21.15.59:0/2822481419 >> v2:172.21.15.110:3300/0 conn(0x7f5e880a5f40 0x7f5e88141b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:36.495 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.494+0000 7f5e87fff700 1 -- 172.21.15.59:0/2822481419 >> v1:172.21.15.110:6789/0 conn(0x7f5e881420c0 legacy=0x7f5e8813e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:09:36.495 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.494+0000 7f5e87fff700 10 monclient: _finish_hunting 0 2023-09-30T02:09:36.496 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.494+0000 7f5e87fff700 1 monclient: found mon.noname-b 2023-09-30T02:09:36.496 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.494+0000 7f5e87fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:09:36.496 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.494+0000 7f5e87fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.59:3301/0 2023-09-30T02:09:36.496 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.494+0000 7f5e87fff700 1 -- 172.21.15.59:0/2822481419 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e88142500 con 0x7f5e880a5b70 2023-09-30T02:09:36.497 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.497+0000 7f5e87fff700 1 --2- 172.21.15.59:0/2822481419 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5e880a5b70 0x7f5e880b1500 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f5e780099b0 tx=0x7f5e78009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=59a82c3d7941a4d7 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:09:36.504 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.504+0000 7f5e86ffd700 1 -- 172.21.15.59:0/2822481419 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e78009ea0 con 0x7f5e880a5b70 2023-09-30T02:09:36.504 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.504+0000 7f5e86ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:09:36.504 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.504+0000 7f5e86ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:09:36.504 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.504+0000 7f5e86ffd700 10 monclient: dump: 2023-09-30T02:09:36.504 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:09:36.505 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:36.505 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:36.505 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:36.505 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:09:36.505 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:36.505 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:09:36.505 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:09:36.505 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:09:36.505 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:36.505 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.504+0000 7f5e86ffd700 10 monclient: _finish_auth 0 2023-09-30T02:09:36.505 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.504+0000 7f5e86ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:09:36.505 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.504+0000 7f5e86ffd700 1 -- 172.21.15.59:0/2822481419 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e7800b820 con 0x7f5e880a5b70 2023-09-30T02:09:36.506 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.504+0000 7f5e86ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:09:36.506 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.504+0000 7f5e86ffd700 1 -- 172.21.15.59:0/2822481419 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e7800bbc0 con 0x7f5e880a5b70 2023-09-30T02:09:36.506 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.504+0000 7f5e8e8c1700 5 monclient: authenticate success, global_id 4580 2023-09-30T02:09:36.506 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.504+0000 7f5e8e8c1700 10 monclient: get_monmap_and_config success 2023-09-30T02:09:36.506 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.504+0000 7f5e8e8c1700 4 set_mon_vals no callback set 2023-09-30T02:09:36.506 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.504+0000 7f5e86ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:09:36.506 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.504+0000 7f5e86ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:09:36.506 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.504+0000 7f5e86ffd700 10 monclient: dump: 2023-09-30T02:09:36.506 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:09:36.506 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:36.506 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:36.506 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:36.507 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:09:36.507 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:36.507 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:09:36.507 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:09:36.507 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:09:36.507 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:36.510 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.504+0000 7f5e8e8c1700 10 monclient: shutdown 2023-09-30T02:09:36.510 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.504+0000 7f5e8e8c1700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:09:36.510 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.504+0000 7f5e8e8c1700 1 -- 172.21.15.59:0/2822481419 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5e880a5b70 msgr2=0x7f5e880b1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:36.510 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.504+0000 7f5e8e8c1700 1 --2- 172.21.15.59:0/2822481419 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5e880a5b70 0x7f5e880b1500 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f5e780099b0 tx=0x7f5e78009250 comp rx=0 tx=0).stop 2023-09-30T02:09:36.510 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.505+0000 7f5e8e8c1700 1 -- 172.21.15.59:0/2822481419 shutdown_connections 2023-09-30T02:09:36.510 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.505+0000 7f5e8e8c1700 1 --2- 172.21.15.59:0/2822481419 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5e880a5b70 0x7f5e880b1500 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:36.510 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.505+0000 7f5e8e8c1700 1 --2- 172.21.15.59:0/2822481419 >> v2:172.21.15.110:3300/0 conn(0x7f5e880a5f40 0x7f5e88141b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:36.510 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.505+0000 7f5e8e8c1700 1 -- 172.21.15.59:0/2822481419 >> 172.21.15.59:0/2822481419 conn(0x7f5e880631f0 msgr2=0x7f5e880635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:09:36.511 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.505+0000 7f5e8e8c1700 1 -- 172.21.15.59:0/2822481419 shutdown_connections 2023-09-30T02:09:36.511 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.505+0000 7f5e8e8c1700 1 -- 172.21.15.59:0/2822481419 wait complete. 2023-09-30T02:09:36.511 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.506+0000 7f5e8e8c1700 10 monclient: build_initial_monmap 2023-09-30T02:09:36.511 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.506+0000 7f5e8e8c1700 10 monclient: monmap: 2023-09-30T02:09:36.511 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:09:36.511 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:09:36.511 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:09:36.511 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:09:36.511 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:09:36.511 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:09:36.511 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:09:36.511 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:09:36.512 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:09:36.512 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:36.512 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.506+0000 7f5e8e8c1700 1 librados: starting msgr at 2023-09-30T02:09:36.512 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.506+0000 7f5e8e8c1700 1 librados: starting objecter 2023-09-30T02:09:36.512 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.506+0000 7f5e8e8c1700 1 Processor -- start 2023-09-30T02:09:36.512 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.506+0000 7f5e8e8c1700 1 -- start start 2023-09-30T02:09:36.512 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.506+0000 7f5e8e8c1700 1 librados: setting wanted keys 2023-09-30T02:09:36.512 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.506+0000 7f5e8e8c1700 1 librados: calling monclient init 2023-09-30T02:09:36.512 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.506+0000 7f5e8e8c1700 10 monclient: init 2023-09-30T02:09:36.512 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.507+0000 7f5e8e8c1700 10 monclient: _reopen_session rank -1 2023-09-30T02:09:36.512 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.507+0000 7f5e8e8c1700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T02:09:36.512 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.507+0000 7f5e8e8c1700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5e880a5b70 0x7f5e8814e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:36.512 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.507+0000 7f5e8e8c1700 10 monclient(hunting): picked mon.noname-b con 0x7f5e880a5b70 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:09:36.513 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.507+0000 7f5e8e8c1700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5e880a5f40 0x7f5e8814e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:36.513 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.507+0000 7f5e8e8c1700 10 monclient(hunting): picked mon.noname-a con 0x7f5e880a5f40 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:09:36.513 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.507+0000 7f5e8e8c1700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5e881420c0 0x7f5e880a7930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:36.513 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.507+0000 7f5e8e8c1700 10 monclient(hunting): picked mon.noname-c con 0x7f5e881420c0 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:09:36.513 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.507+0000 7f5e8e8c1700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:36.513 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.507+0000 7f5e8e8c1700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e8814ef30 con 0x7f5e880a5f40 2023-09-30T02:09:36.513 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.507+0000 7f5e8e8c1700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:36.513 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.507+0000 7f5e8e8c1700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e880a9780 con 0x7f5e880a5b70 2023-09-30T02:09:36.513 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.507+0000 7f5e8e8c1700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:36.513 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.507+0000 7f5e8e8c1700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e880b2ad0 con 0x7f5e881420c0 2023-09-30T02:09:36.513 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.507+0000 7f5e8e8c1700 10 monclient(hunting): _renew_subs 2023-09-30T02:09:36.513 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.507+0000 7f5e8e8c1700 10 monclient(hunting): authenticate will time out at 2021.810754s 2023-09-30T02:09:36.514 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.508+0000 7f5e87fff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5e880a5b70 0x7f5e8814e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:36.514 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.508+0000 7f5e8ce5e700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5e881420c0 0x7f5e880a7930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:36.514 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.508+0000 7f5e87fff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5e880a5b70 0x7f5e8814e4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:55210/0 (socket says 172.21.15.59:55210) 2023-09-30T02:09:36.514 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.508+0000 7f5e87fff700 1 -- 172.21.15.59:0/1034612387 learned_addr learned my addr 172.21.15.59:0/1034612387 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:09:36.514 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.508+0000 7f5e87fff700 10 monclient(hunting): get_auth_request con 0x7f5e880a5b70 auth_method 0 2023-09-30T02:09:36.514 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.508+0000 7f5e87fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:09:36.514 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.508+0000 7f5e87fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:36.514 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.508+0000 7f5e87fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:09:36.514 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.508+0000 7f5e8ce5e700 10 monclient(hunting): get_auth_request con 0x7f5e881420c0 auth_method 0 2023-09-30T02:09:36.514 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.508+0000 7f5e8ce5e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:09:36.514 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.508+0000 7f5e8ce5e700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:36.514 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.508+0000 7f5e8ce5e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:09:36.515 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.509+0000 7f5e87fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:09:36.515 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.509+0000 7f5e87fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:09:36.515 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.509+0000 7f5e87fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:09:36.515 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.509+0000 7f5e8ce5e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:09:36.515 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.509+0000 7f5e8ce5e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:09:36.515 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.509+0000 7f5e8ce5e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:09:36.515 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.509+0000 7f5e877fe700 1 --2- 172.21.15.59:0/1034612387 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5e880a5f40 0x7f5e8814e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:36.516 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.511+0000 7f5e877fe700 10 monclient(hunting): get_auth_request con 0x7f5e880a5f40 auth_method 0 2023-09-30T02:09:36.516 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.511+0000 7f5e877fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:09:36.516 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.511+0000 7f5e877fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:36.516 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.511+0000 7f5e877fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:09:36.516 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.512+0000 7f5e877fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:09:36.517 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.512+0000 7f5e877fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:09:36.517 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.512+0000 7f5e877fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:09:36.517 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.513+0000 7f5e8ce5e700 10 monclient(hunting): handle_auth_done global_id 4583 payload 824 2023-09-30T02:09:36.517 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.513+0000 7f5e8ce5e700 1 -- 172.21.15.59:0/1034612387 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5e880a5b70 msgr2=0x7f5e8814e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:36.517 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.513+0000 7f5e8ce5e700 1 --2- 172.21.15.59:0/1034612387 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5e880a5b70 0x7f5e8814e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:36.517 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.513+0000 7f5e8ce5e700 1 -- 172.21.15.59:0/1034612387 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5e880a5f40 msgr2=0x7f5e8814e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:36.517 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.513+0000 7f5e8ce5e700 1 --2- 172.21.15.59:0/1034612387 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5e880a5f40 0x7f5e8814e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:36.517 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.513+0000 7f5e8ce5e700 10 monclient: _finish_hunting 0 2023-09-30T02:09:36.517 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.513+0000 7f5e8ce5e700 1 monclient: found mon.noname-c 2023-09-30T02:09:36.517 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.513+0000 7f5e8ce5e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:09:36.517 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.513+0000 7f5e8ce5e700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.59:3301/0 2023-09-30T02:09:36.517 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.513+0000 7f5e8ce5e700 1 -- 172.21.15.59:0/1034612387 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e780094f0 con 0x7f5e881420c0 2023-09-30T02:09:36.517 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.513+0000 7f5e87fff700 1 --2- 172.21.15.59:0/1034612387 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5e880a5b70 0x7f5e8814e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T02:09:36.518 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.516+0000 7f5e8ce5e700 1 --2- 172.21.15.59:0/1034612387 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5e881420c0 0x7f5e880a7930 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5e8000bb00 tx=0x7f5e80009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:09:36.518 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.516+0000 7f5e6ffff700 10 client.?.objecter ms_handle_connect 0x7f5e881420c0 2023-09-30T02:09:36.518 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.516+0000 7f5e6ffff700 10 client.?.objecter resend_mon_ops 2023-09-30T02:09:36.579 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.578+0000 7f5e6ffff700 1 -- 172.21.15.59:0/1034612387 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e8000c7c0 con 0x7f5e881420c0 2023-09-30T02:09:36.579 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.578+0000 7f5e6ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:09:36.579 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.578+0000 7f5e6ffff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:09:36.580 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.578+0000 7f5e6ffff700 10 monclient: dump: 2023-09-30T02:09:36.580 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:09:36.580 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:36.580 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:36.580 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:36.580 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:09:36.580 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:36.580 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:09:36.580 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:09:36.580 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:09:36.580 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:36.580 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.578+0000 7f5e6ffff700 10 monclient: _finish_auth 0 2023-09-30T02:09:36.581 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.578+0000 7f5e6ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:09:36.581 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.578+0000 7f5e6ffff700 1 -- 172.21.15.59:0/1034612387 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e8000c920 con 0x7f5e881420c0 2023-09-30T02:09:36.581 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.578+0000 7f5e6ffff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:09:36.581 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.578+0000 7f5e6ffff700 1 -- 172.21.15.59:0/1034612387 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e8000cce0 con 0x7f5e881420c0 2023-09-30T02:09:36.581 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.578+0000 7f5e8e8c1700 5 monclient: authenticate success, global_id 4583 2023-09-30T02:09:36.581 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.578+0000 7f5e8e8c1700 10 monclient: _renew_subs 2023-09-30T02:09:36.581 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.578+0000 7f5e8e8c1700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:09:36.581 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.578+0000 7f5e8e8c1700 1 -- 172.21.15.59:0/1034612387 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e880b4950 con 0x7f5e881420c0 2023-09-30T02:09:36.581 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.578+0000 7f5e6ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:09:36.581 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.578+0000 7f5e6ffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:09:36.581 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.578+0000 7f5e867fc700 4 set_mon_vals no callback set 2023-09-30T02:09:36.581 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.578+0000 7f5e6ffff700 10 monclient: dump: 2023-09-30T02:09:36.582 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:09:36.582 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:36.582 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:36.582 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:36.582 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:09:36.582 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:36.582 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:09:36.582 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:09:36.582 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:09:36.582 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:36.582 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.578+0000 7f5e8e8c1700 10 client.4583.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:09:36.583 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.578+0000 7f5e8e8c1700 10 monclient: _renew_subs 2023-09-30T02:09:36.583 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.578+0000 7f5e8e8c1700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:09:36.583 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.578+0000 7f5e8e8c1700 1 -- 172.21.15.59:0/1034612387 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e880a8030 con 0x7f5e881420c0 2023-09-30T02:09:36.583 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.578+0000 7f5e8e8c1700 1 librados: init done 2023-09-30T02:09:36.583 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.579+0000 7f5e8e8c1700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:09:36.584 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.579+0000 7f5e8e8c1700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:09:36.584 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.579+0000 7f5e8e8c1700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:09:36.584 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.579+0000 7f5e8e8c1700 1 -- 172.21.15.59:0/1034612387 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e8814e1d0 con 0x7f5e881420c0 2023-09-30T02:09:36.602 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.601+0000 7f5e6ffff700 1 -- 172.21.15.59:0/1034612387 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f5e80014620 con 0x7f5e881420c0 2023-09-30T02:09:36.602 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.602+0000 7f5e6ffff700 1 --2- 172.21.15.59:0/1034612387 >> v2:172.21.15.110:6800/115870 conn(0x7f5e74068960 0x7f5e7406ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:36.603 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.602+0000 7f5e87fff700 1 --2- 172.21.15.59:0/1034612387 >> v2:172.21.15.110:6800/115870 conn(0x7f5e74068960 0x7f5e7406ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:36.603 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.602+0000 7f5e87fff700 10 monclient: get_auth_request con 0x7f5e74068960 auth_method 0 2023-09-30T02:09:36.603 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.603+0000 7f5e6ffff700 10 client.4583.objecter ms_handle_connect 0x7f5e74068960 2023-09-30T02:09:36.603 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.603+0000 7f5e87fff700 1 --2- 172.21.15.59:0/1034612387 >> v2:172.21.15.110:6800/115870 conn(0x7f5e74068960 0x7f5e7406ae10 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f5e78009490 tx=0x7f5e78016000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:09:36.708 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.708+0000 7f5e6ffff700 1 -- 172.21.15.59:0/1034612387 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f5e8000ce40 con 0x7f5e881420c0 2023-09-30T02:09:36.708 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.708+0000 7f5e6ffff700 10 client.4583.objecter ms_dispatch 0x7f5e88001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:09:36.709 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.708+0000 7f5e6ffff700 3 client.4583.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:09:36.709 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.708+0000 7f5e6ffff700 3 client.4583.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:09:36.709 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.708+0000 7f5e6ffff700 20 client.4583.objecter dump_active .. 0 homeless 2023-09-30T02:09:36.894 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.894+0000 7f5e6ffff700 1 -- 172.21.15.59:0/1034612387 <== mon.2 v2:172.21.15.59:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5e80051b20 con 0x7f5e881420c0 2023-09-30T02:09:36.894 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.894+0000 7f5e6ffff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:09:36.895 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:36.894+0000 7f5e6ffff700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:09:37.164 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:37.164+0000 7f5e8e8c1700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:09:37.164 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:37.164+0000 7f5e8e8c1700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:09:37.164 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:37.164+0000 7f5e8e8c1700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:09:37.164 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:37.164+0000 7f5e8e8c1700 1 -- 172.21.15.59:0/1034612387 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f5e8814e1d0 con 0x7f5e881420c0 2023-09-30T02:09:37.169 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:37.169+0000 7f5e6ffff700 1 -- 172.21.15.59:0/1034612387 <== mon.2 v2:172.21.15.59:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f5e8814e1d0 con 0x7f5e881420c0 2023-09-30T02:09:37.170 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:37.169+0000 7f5e6ffff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:09:37.170 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:37.169+0000 7f5e6ffff700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:09:37.170 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:09:37.170 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:09:37.173 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:37.171+0000 7f5e8e8c1700 10 librados: watch_flush enter 2023-09-30T02:09:37.173 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:37.171+0000 7f5e8e8c1700 10 librados: watch_flush exit 2023-09-30T02:09:37.173 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:37.171+0000 7f5e8e8c1700 20 client.4583.objecter shutdown clearing up homeless session... 2023-09-30T02:09:37.173 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:37.171+0000 7f5e8e8c1700 10 client.4583.objecter successfully canceled tick 2023-09-30T02:09:37.173 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:37.172+0000 7f5e8e8c1700 1 -- 172.21.15.59:0/1034612387 >> v2:172.21.15.110:6800/115870 conn(0x7f5e74068960 msgr2=0x7f5e7406ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:37.173 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:37.172+0000 7f5e8e8c1700 1 --2- 172.21.15.59:0/1034612387 >> v2:172.21.15.110:6800/115870 conn(0x7f5e74068960 0x7f5e7406ae10 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f5e78009490 tx=0x7f5e78016000 comp rx=0 tx=0).stop 2023-09-30T02:09:37.173 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:37.172+0000 7f5e8e8c1700 10 monclient: shutdown 2023-09-30T02:09:37.173 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:37.172+0000 7f5e8e8c1700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:09:37.173 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:37.172+0000 7f5e8e8c1700 1 -- 172.21.15.59:0/1034612387 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5e881420c0 msgr2=0x7f5e880a7930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:37.173 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:37.172+0000 7f5e8e8c1700 1 --2- 172.21.15.59:0/1034612387 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5e881420c0 0x7f5e880a7930 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5e8000bb00 tx=0x7f5e80009780 comp rx=0 tx=0).stop 2023-09-30T02:09:37.174 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:37.172+0000 7f5e8e8c1700 1 -- 172.21.15.59:0/1034612387 shutdown_connections 2023-09-30T02:09:37.174 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:37.172+0000 7f5e8e8c1700 1 --2- 172.21.15.59:0/1034612387 >> v2:172.21.15.110:6800/115870 conn(0x7f5e74068960 0x7f5e7406ae10 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:37.174 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:37.172+0000 7f5e8e8c1700 1 --2- 172.21.15.59:0/1034612387 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5e880a5b70 0x7f5e8814e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:37.174 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:37.172+0000 7f5e8e8c1700 1 --2- 172.21.15.59:0/1034612387 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5e881420c0 0x7f5e880a7930 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:37.174 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:37.172+0000 7f5e8e8c1700 1 --2- 172.21.15.59:0/1034612387 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5e880a5f40 0x7f5e8814e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:37.174 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:37.172+0000 7f5e8e8c1700 1 -- 172.21.15.59:0/1034612387 >> 172.21.15.59:0/1034612387 conn(0x7f5e880631f0 msgr2=0x7f5e88138d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:09:37.174 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:37.172+0000 7f5e8e8c1700 1 -- 172.21.15.59:0/1034612387 shutdown_connections 2023-09-30T02:09:37.174 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:37.172+0000 7f5e8e8c1700 1 -- 172.21.15.59:0/1034612387 wait complete. 2023-09-30T02:09:37.174 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:37.172+0000 7f5e8e8c1700 1 librados: shutdown 2023-09-30T02:09:40.187 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:09:40.319 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.318+0000 7ff156672700 10 monclient: get_monmap_and_config 2023-09-30T02:09:40.319 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.318+0000 7ff156672700 10 monclient: build_initial_monmap 2023-09-30T02:09:40.319 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.318+0000 7ff156672700 10 monclient: monmap: 2023-09-30T02:09:40.319 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:09:40.319 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:40.320 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:09:40.319948+0000 2023-09-30T02:09:40.320 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:09:40.319948+0000 2023-09-30T02:09:40.320 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:09:40.320 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:40.320 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:09:40.320 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:09:40.321 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:09:40.321 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:09:40.321 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:09:40.321 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:40.322 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.319+0000 7ff156672700 1 Processor -- start 2023-09-30T02:09:40.322 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.319+0000 7ff156672700 1 -- start start 2023-09-30T02:09:40.323 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.319+0000 7ff156672700 10 monclient: init 2023-09-30T02:09:40.323 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.320+0000 7ff156672700 10 monclient: _reopen_session rank -1 2023-09-30T02:09:40.323 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.320+0000 7ff156672700 10 monclient: _add_conns ranks=[1,0,2,4,3] 2023-09-30T02:09:40.323 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.320+0000 7ff156672700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff1500a5b70 addr v1:172.21.15.110:6789/0 2023-09-30T02:09:40.323 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.320+0000 7ff156672700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff150142290 addr v1:172.21.15.59:6789/0 2023-09-30T02:09:40.323 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.320+0000 7ff156672700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff1500b1500 0x7ff1500a5f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:40.323 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.320+0000 7ff156672700 10 monclient(hunting): picked mon.noname-b con 0x7ff1500b1500 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:09:40.323 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.320+0000 7ff156672700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1500a75a0 con 0x7ff150142290 2023-09-30T02:09:40.323 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.320+0000 7ff156672700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1500af3f0 con 0x7ff1500a5b70 2023-09-30T02:09:40.323 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.320+0000 7ff156672700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:40.324 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.320+0000 7ff156672700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1500b1940 con 0x7ff1500b1500 2023-09-30T02:09:40.324 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.320+0000 7ff156672700 10 monclient(hunting): _renew_subs 2023-09-30T02:09:40.324 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.320+0000 7ff156672700 10 monclient(hunting): authenticate will time out at 2025.623098s 2023-09-30T02:09:40.324 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.320+0000 7ff14ffff700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7ff1500a5b70 0x7ff150141b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.59:52576/0 (socket says 172.21.15.59:52576) 2023-09-30T02:09:40.324 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.320+0000 7ff14ffff700 1 -- 172.21.15.59:0/163845242 learned_addr learned my addr 172.21.15.59:0/163845242 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:09:40.324 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.320+0000 7ff154c0f700 1 --2- 172.21.15.59:0/163845242 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff1500b1500 0x7ff1500a5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:40.324 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.321+0000 7ff154c0f700 10 monclient(hunting): get_auth_request con 0x7ff1500b1500 auth_method 0 2023-09-30T02:09:40.324 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.321+0000 7ff154c0f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:09:40.325 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.321+0000 7ff154c0f700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:40.325 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.321+0000 7ff154c0f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:09:40.325 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.322+0000 7ff154c0f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:09:40.325 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.322+0000 7ff154c0f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:09:40.325 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.322+0000 7ff154c0f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:09:40.327 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.324+0000 7ff14effd700 1 -- 172.21.15.59:0/163845242 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2604159811 0 0) 0x7ff1500a75a0 con 0x7ff150142290 2023-09-30T02:09:40.327 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.324+0000 7ff14effd700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:40.327 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.324+0000 7ff14effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:09:40.327 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.324+0000 7ff14effd700 10 monclient(hunting): my global_id is 4659 2023-09-30T02:09:40.328 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.324+0000 7ff14effd700 1 -- 172.21.15.59:0/163845242 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff138003570 con 0x7ff150142290 2023-09-30T02:09:40.328 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.324+0000 7ff14effd700 1 -- 172.21.15.59:0/163845242 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 406359203 0 0) 0x7ff1500af3f0 con 0x7ff1500a5b70 2023-09-30T02:09:40.328 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.324+0000 7ff14effd700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:40.328 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.324+0000 7ff14effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:09:40.328 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.324+0000 7ff14effd700 10 monclient(hunting): my global_id is 4681 2023-09-30T02:09:40.328 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.324+0000 7ff14effd700 1 -- 172.21.15.59:0/163845242 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1500a75a0 con 0x7ff1500a5b70 2023-09-30T02:09:40.329 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.325+0000 7ff154c0f700 10 monclient(hunting): handle_auth_done global_id 4586 payload 274 2023-09-30T02:09:40.329 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.325+0000 7ff154c0f700 1 -- 172.21.15.59:0/163845242 >> v1:172.21.15.110:6789/0 conn(0x7ff1500a5b70 legacy=0x7ff150141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:40.329 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.325+0000 7ff154c0f700 1 -- 172.21.15.59:0/163845242 >> v1:172.21.15.59:6789/0 conn(0x7ff150142290 legacy=0x7ff15013af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:40.329 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.325+0000 7ff154c0f700 10 monclient: _finish_hunting 0 2023-09-30T02:09:40.329 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.325+0000 7ff154c0f700 1 monclient: found mon.noname-b 2023-09-30T02:09:40.329 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.325+0000 7ff154c0f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:09:40.330 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.325+0000 7ff154c0f700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.59:3301/0 2023-09-30T02:09:40.330 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.325+0000 7ff154c0f700 1 -- 172.21.15.59:0/163845242 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff15013b640 con 0x7ff1500b1500 2023-09-30T02:09:40.331 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.327+0000 7ff154c0f700 1 --2- 172.21.15.59:0/163845242 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff1500b1500 0x7ff1500a5f40 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff148009e90 tx=0x7ff148009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e3b9c32e6296b105 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:09:40.331 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.330+0000 7ff14effd700 1 -- 172.21.15.59:0/163845242 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff14800e070 con 0x7ff1500b1500 2023-09-30T02:09:40.332 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.330+0000 7ff14effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:09:40.332 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.330+0000 7ff14effd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:09:40.332 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.330+0000 7ff14effd700 10 monclient: dump: 2023-09-30T02:09:40.332 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:09:40.332 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:40.333 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:40.333 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:40.333 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:09:40.333 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:40.333 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:09:40.333 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:09:40.333 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:09:40.334 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:40.334 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.330+0000 7ff14effd700 10 monclient: _finish_auth 0 2023-09-30T02:09:40.334 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.330+0000 7ff14effd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:09:40.334 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.330+0000 7ff156672700 5 monclient: authenticate success, global_id 4586 2023-09-30T02:09:40.334 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.330+0000 7ff156672700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:09:40.337 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.332+0000 7ff14effd700 1 -- 172.21.15.59:0/163845242 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff14800bb80 con 0x7ff1500b1500 2023-09-30T02:09:40.337 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.332+0000 7ff14effd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:09:40.337 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.332+0000 7ff14effd700 1 -- 172.21.15.59:0/163845242 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff148012430 con 0x7ff1500b1500 2023-09-30T02:09:40.338 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.332+0000 7ff156672700 10 monclient: get_monmap_and_config success 2023-09-30T02:09:40.338 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.332+0000 7ff156672700 4 set_mon_vals no callback set 2023-09-30T02:09:40.338 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.332+0000 7ff14effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:09:40.338 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.332+0000 7ff14effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:09:40.338 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.332+0000 7ff14effd700 10 monclient: dump: 2023-09-30T02:09:40.338 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:09:40.338 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:40.338 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:40.338 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:40.338 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:09:40.339 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:40.339 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:09:40.339 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:09:40.339 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:09:40.339 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:40.339 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.332+0000 7ff156672700 10 monclient: shutdown 2023-09-30T02:09:40.339 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.332+0000 7ff156672700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:09:40.339 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.332+0000 7ff156672700 1 -- 172.21.15.59:0/163845242 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff1500b1500 msgr2=0x7ff1500a5f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:40.339 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.333+0000 7ff156672700 1 --2- 172.21.15.59:0/163845242 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff1500b1500 0x7ff1500a5f40 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff148009e90 tx=0x7ff148009780 comp rx=0 tx=0).stop 2023-09-30T02:09:40.339 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.333+0000 7ff156672700 1 -- 172.21.15.59:0/163845242 shutdown_connections 2023-09-30T02:09:40.339 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.333+0000 7ff156672700 1 --2- 172.21.15.59:0/163845242 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff1500b1500 0x7ff1500a5f40 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:40.340 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.333+0000 7ff156672700 1 -- 172.21.15.59:0/163845242 >> 172.21.15.59:0/163845242 conn(0x7ff1500631f0 msgr2=0x7ff1500635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:09:40.340 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.333+0000 7ff156672700 1 -- 172.21.15.59:0/163845242 shutdown_connections 2023-09-30T02:09:40.340 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.333+0000 7ff156672700 1 -- 172.21.15.59:0/163845242 wait complete. 2023-09-30T02:09:40.340 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.333+0000 7ff156672700 10 monclient: build_initial_monmap 2023-09-30T02:09:40.340 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.333+0000 7ff156672700 10 monclient: monmap: 2023-09-30T02:09:40.340 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:09:40.340 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:09:40.340 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:09:40.340 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:09:40.340 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:09:40.341 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:09:40.341 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:09:40.341 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:09:40.341 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:09:40.341 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:40.341 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.334+0000 7ff156672700 1 librados: starting msgr at 2023-09-30T02:09:40.341 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.334+0000 7ff156672700 1 librados: starting objecter 2023-09-30T02:09:40.341 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.334+0000 7ff156672700 1 Processor -- start 2023-09-30T02:09:40.341 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.334+0000 7ff156672700 1 -- start start 2023-09-30T02:09:40.341 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.334+0000 7ff156672700 1 librados: setting wanted keys 2023-09-30T02:09:40.341 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.334+0000 7ff156672700 1 librados: calling monclient init 2023-09-30T02:09:40.341 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.334+0000 7ff156672700 10 monclient: init 2023-09-30T02:09:40.342 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.335+0000 7ff156672700 10 monclient: _reopen_session rank -1 2023-09-30T02:09:40.342 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.335+0000 7ff156672700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T02:09:40.342 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.335+0000 7ff156672700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff1500a5b70 0x7ff1501407a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:40.342 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.335+0000 7ff156672700 10 monclient(hunting): picked mon.noname-b con 0x7ff1500a5b70 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:09:40.342 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.335+0000 7ff156672700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff1500b1500 0x7ff15013c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:40.342 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.335+0000 7ff156672700 10 monclient(hunting): picked mon.noname-c con 0x7ff1500b1500 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:09:40.342 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.335+0000 7ff156672700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff150142290 0x7ff15013c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:40.342 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.335+0000 7ff156672700 10 monclient(hunting): picked mon.noname-a con 0x7ff150142290 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:09:40.342 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.335+0000 7ff156672700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:40.342 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.335+0000 7ff156672700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff15013cab0 con 0x7ff150142290 2023-09-30T02:09:40.342 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.335+0000 7ff156672700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:40.343 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.335+0000 7ff156672700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff15013ef00 con 0x7ff1500a5b70 2023-09-30T02:09:40.343 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.335+0000 7ff156672700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:40.343 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.335+0000 7ff156672700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff15013e9e0 con 0x7ff1500b1500 2023-09-30T02:09:40.343 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.335+0000 7ff156672700 10 monclient(hunting): _renew_subs 2023-09-30T02:09:40.343 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.335+0000 7ff156672700 10 monclient(hunting): authenticate will time out at 2025.638299s 2023-09-30T02:09:40.343 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.335+0000 7ff14f7fe700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff1500b1500 0x7ff15013c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:40.343 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.335+0000 7ff14ffff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff1500a5b70 0x7ff1501407a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:40.343 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.336+0000 7ff154c0f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff150142290 0x7ff15013c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:40.343 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.336+0000 7ff14f7fe700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff1500b1500 0x7ff15013c030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:43568/0 (socket says 172.21.15.59:43568) 2023-09-30T02:09:40.343 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.336+0000 7ff14f7fe700 1 -- 172.21.15.59:0/725637433 learned_addr learned my addr 172.21.15.59:0/725637433 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:09:40.343 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.336+0000 7ff14f7fe700 10 monclient(hunting): get_auth_request con 0x7ff1500b1500 auth_method 0 2023-09-30T02:09:40.344 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.336+0000 7ff14f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:09:40.344 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.336+0000 7ff14f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:40.344 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.336+0000 7ff14f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:09:40.344 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.336+0000 7ff14ffff700 10 monclient(hunting): get_auth_request con 0x7ff1500a5b70 auth_method 0 2023-09-30T02:09:40.344 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.336+0000 7ff14ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:09:40.344 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.336+0000 7ff14ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:40.344 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.336+0000 7ff14ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:09:40.344 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.336+0000 7ff154c0f700 10 monclient(hunting): get_auth_request con 0x7ff150142290 auth_method 0 2023-09-30T02:09:40.344 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.336+0000 7ff154c0f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:09:40.344 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.336+0000 7ff154c0f700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:40.344 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.336+0000 7ff154c0f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:09:40.344 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.336+0000 7ff14f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:09:40.345 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.336+0000 7ff14f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:09:40.345 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.336+0000 7ff14f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:09:40.345 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.336+0000 7ff14ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:09:40.345 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.336+0000 7ff14ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:09:40.345 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.336+0000 7ff14ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:09:40.345 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.337+0000 7ff154c0f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:09:40.345 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.337+0000 7ff154c0f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:09:40.345 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.337+0000 7ff154c0f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:09:40.346 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.340+0000 7ff14f7fe700 10 monclient(hunting): handle_auth_done global_id 4589 payload 824 2023-09-30T02:09:40.346 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.340+0000 7ff14f7fe700 1 -- 172.21.15.59:0/725637433 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff1500a5b70 msgr2=0x7ff1501407a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:40.346 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.340+0000 7ff14f7fe700 1 --2- 172.21.15.59:0/725637433 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff1500a5b70 0x7ff1501407a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:40.346 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.340+0000 7ff14f7fe700 1 -- 172.21.15.59:0/725637433 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff150142290 msgr2=0x7ff15013c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:40.346 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.340+0000 7ff14f7fe700 1 --2- 172.21.15.59:0/725637433 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff150142290 0x7ff15013c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:40.346 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.340+0000 7ff14f7fe700 10 monclient: _finish_hunting 0 2023-09-30T02:09:40.346 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.340+0000 7ff14f7fe700 1 monclient: found mon.noname-c 2023-09-30T02:09:40.347 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.340+0000 7ff14f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:09:40.347 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.340+0000 7ff14f7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.59:3301/0 2023-09-30T02:09:40.347 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.340+0000 7ff14f7fe700 1 -- 172.21.15.59:0/725637433 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff148009a20 con 0x7ff1500b1500 2023-09-30T02:09:40.347 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.343+0000 7ff14f7fe700 1 --2- 172.21.15.59:0/725637433 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff1500b1500 0x7ff15013c030 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff14400c580 tx=0x7ff144007e30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:09:40.347 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.343+0000 7ff137fff700 10 client.?.objecter ms_handle_connect 0x7ff1500b1500 2023-09-30T02:09:40.347 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.343+0000 7ff137fff700 10 client.?.objecter resend_mon_ops 2023-09-30T02:09:40.348 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.347+0000 7ff137fff700 1 -- 172.21.15.59:0/725637433 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff144010510 con 0x7ff1500b1500 2023-09-30T02:09:40.348 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.347+0000 7ff137fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:09:40.348 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.347+0000 7ff137fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:09:40.348 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.347+0000 7ff137fff700 10 monclient: dump: 2023-09-30T02:09:40.348 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:09:40.348 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:40.348 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:40.349 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:40.349 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:09:40.349 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:40.349 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:09:40.349 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:09:40.349 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:09:40.349 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:40.349 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.347+0000 7ff137fff700 10 monclient: _finish_auth 0 2023-09-30T02:09:40.349 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.347+0000 7ff137fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:09:40.349 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.347+0000 7ff137fff700 1 -- 172.21.15.59:0/725637433 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff144010670 con 0x7ff1500b1500 2023-09-30T02:09:40.349 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.347+0000 7ff137fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:09:40.349 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.347+0000 7ff137fff700 1 -- 172.21.15.59:0/725637433 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff144010a30 con 0x7ff1500b1500 2023-09-30T02:09:40.350 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.347+0000 7ff137fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:09:40.350 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.347+0000 7ff137fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:09:40.350 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.347+0000 7ff137fff700 10 monclient: dump: 2023-09-30T02:09:40.350 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:09:40.350 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:40.350 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:40.350 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:40.350 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:09:40.350 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:40.350 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:09:40.351 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:09:40.351 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:09:40.351 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:40.351 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.347+0000 7ff156672700 5 monclient: authenticate success, global_id 4589 2023-09-30T02:09:40.351 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.347+0000 7ff156672700 10 monclient: _renew_subs 2023-09-30T02:09:40.351 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.347+0000 7ff156672700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:09:40.351 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.347+0000 7ff156672700 1 -- 172.21.15.59:0/725637433 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1501414a0 con 0x7ff1500b1500 2023-09-30T02:09:40.351 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.347+0000 7ff14e7fc700 4 set_mon_vals no callback set 2023-09-30T02:09:40.351 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.347+0000 7ff156672700 10 client.4589.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:09:40.351 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.347+0000 7ff156672700 10 monclient: _renew_subs 2023-09-30T02:09:40.351 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.347+0000 7ff156672700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:09:40.351 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.347+0000 7ff156672700 1 -- 172.21.15.59:0/725637433 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff150137620 con 0x7ff1500b1500 2023-09-30T02:09:40.352 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.347+0000 7ff156672700 1 librados: init done 2023-09-30T02:09:40.352 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.348+0000 7ff156672700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:09:40.352 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.348+0000 7ff156672700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:09:40.352 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.348+0000 7ff156672700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:09:40.352 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.348+0000 7ff156672700 1 -- 172.21.15.59:0/725637433 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff15013ec00 con 0x7ff1500b1500 2023-09-30T02:09:40.376 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.376+0000 7ff137fff700 1 -- 172.21.15.59:0/725637433 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7ff144025430 con 0x7ff1500b1500 2023-09-30T02:09:40.377 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.376+0000 7ff137fff700 1 --2- 172.21.15.59:0/725637433 >> v2:172.21.15.110:6800/115870 conn(0x7ff13c068960 0x7ff13c06ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:40.377 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.376+0000 7ff137fff700 1 -- 172.21.15.59:0/725637433 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7ff144084500 con 0x7ff1500b1500 2023-09-30T02:09:40.377 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.377+0000 7ff137fff700 10 client.4589.objecter ms_dispatch 0x7ff150001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:09:40.378 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.377+0000 7ff137fff700 3 client.4589.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:09:40.378 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.377+0000 7ff137fff700 3 client.4589.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:09:40.378 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.377+0000 7ff137fff700 20 client.4589.objecter dump_active .. 0 homeless 2023-09-30T02:09:40.378 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.377+0000 7ff14ffff700 1 --2- 172.21.15.59:0/725637433 >> v2:172.21.15.110:6800/115870 conn(0x7ff13c068960 0x7ff13c06ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:40.378 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.377+0000 7ff14ffff700 10 monclient: get_auth_request con 0x7ff13c068960 auth_method 0 2023-09-30T02:09:40.378 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.377+0000 7ff137fff700 10 client.4589.objecter ms_handle_connect 0x7ff13c068960 2023-09-30T02:09:40.378 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.377+0000 7ff14ffff700 1 --2- 172.21.15.59:0/725637433 >> v2:172.21.15.110:6800/115870 conn(0x7ff13c068960 0x7ff13c06ae10 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ff140007fd0 tx=0x7ff140009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:09:40.651 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.651+0000 7ff137fff700 1 -- 172.21.15.59:0/725637433 <== mon.2 v2:172.21.15.59:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff144051e20 con 0x7ff1500b1500 2023-09-30T02:09:40.651 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.651+0000 7ff137fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:09:40.652 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.651+0000 7ff137fff700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:09:40.902 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.902+0000 7ff156672700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:09:40.902 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.902+0000 7ff156672700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:09:40.903 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.902+0000 7ff156672700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:09:40.903 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.902+0000 7ff156672700 1 -- 172.21.15.59:0/725637433 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ff150064a00 con 0x7ff1500b1500 2023-09-30T02:09:40.910 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.909+0000 7ff137fff700 1 -- 172.21.15.59:0/725637433 <== mon.2 v2:172.21.15.59:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7ff144055440 con 0x7ff1500b1500 2023-09-30T02:09:40.910 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.909+0000 7ff137fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:09:40.910 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.909+0000 7ff137fff700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:09:40.910 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:09:40.910 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:09:40.913 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.912+0000 7ff156672700 10 librados: watch_flush enter 2023-09-30T02:09:40.913 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.912+0000 7ff156672700 10 librados: watch_flush exit 2023-09-30T02:09:40.913 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.912+0000 7ff156672700 20 client.4589.objecter shutdown clearing up homeless session... 2023-09-30T02:09:40.913 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.912+0000 7ff156672700 10 client.4589.objecter successfully canceled tick 2023-09-30T02:09:40.913 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.912+0000 7ff156672700 1 -- 172.21.15.59:0/725637433 >> v2:172.21.15.110:6800/115870 conn(0x7ff13c068960 msgr2=0x7ff13c06ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:40.913 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.912+0000 7ff156672700 1 --2- 172.21.15.59:0/725637433 >> v2:172.21.15.110:6800/115870 conn(0x7ff13c068960 0x7ff13c06ae10 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ff140007fd0 tx=0x7ff140009040 comp rx=0 tx=0).stop 2023-09-30T02:09:40.913 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.912+0000 7ff156672700 10 monclient: shutdown 2023-09-30T02:09:40.914 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.912+0000 7ff156672700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:09:40.914 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.912+0000 7ff156672700 1 -- 172.21.15.59:0/725637433 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff1500b1500 msgr2=0x7ff15013c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:40.914 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.912+0000 7ff156672700 1 --2- 172.21.15.59:0/725637433 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff1500b1500 0x7ff15013c030 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff14400c580 tx=0x7ff144007e30 comp rx=0 tx=0).stop 2023-09-30T02:09:40.914 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.912+0000 7ff156672700 1 -- 172.21.15.59:0/725637433 shutdown_connections 2023-09-30T02:09:40.914 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.912+0000 7ff156672700 1 --2- 172.21.15.59:0/725637433 >> v2:172.21.15.110:6800/115870 conn(0x7ff13c068960 0x7ff13c06ae10 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:40.914 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.912+0000 7ff156672700 1 --2- 172.21.15.59:0/725637433 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff1500a5b70 0x7ff1501407a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:40.914 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.912+0000 7ff156672700 1 --2- 172.21.15.59:0/725637433 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff150142290 0x7ff15013c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:40.914 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.912+0000 7ff156672700 1 --2- 172.21.15.59:0/725637433 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff1500b1500 0x7ff15013c030 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:40.914 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.912+0000 7ff156672700 1 -- 172.21.15.59:0/725637433 >> 172.21.15.59:0/725637433 conn(0x7ff1500631f0 msgr2=0x7ff1500b0be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:09:40.914 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.912+0000 7ff156672700 1 -- 172.21.15.59:0/725637433 shutdown_connections 2023-09-30T02:09:40.915 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.912+0000 7ff156672700 1 -- 172.21.15.59:0/725637433 wait complete. 2023-09-30T02:09:40.915 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:40.912+0000 7ff156672700 1 librados: shutdown 2023-09-30T02:09:43.929 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:09:44.078 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.078+0000 7f7879f97700 10 monclient: get_monmap_and_config 2023-09-30T02:09:44.078 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.078+0000 7f7879f97700 10 monclient: build_initial_monmap 2023-09-30T02:09:44.079 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.078+0000 7f7879f97700 10 monclient: monmap: 2023-09-30T02:09:44.079 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:09:44.079 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:44.079 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:09:44.079569+0000 2023-09-30T02:09:44.079 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:09:44.079569+0000 2023-09-30T02:09:44.079 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:09:44.079 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:44.079 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:09:44.080 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:09:44.080 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:09:44.080 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:09:44.080 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:09:44.080 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:44.081 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.079+0000 7f7879f97700 1 Processor -- start 2023-09-30T02:09:44.081 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.079+0000 7f7879f97700 1 -- start start 2023-09-30T02:09:44.082 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.079+0000 7f7879f97700 10 monclient: init 2023-09-30T02:09:44.082 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.079+0000 7f7879f97700 10 monclient: _reopen_session rank -1 2023-09-30T02:09:44.082 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.079+0000 7f7879f97700 10 monclient: _add_conns ranks=[0,3,4,2,1] 2023-09-30T02:09:44.082 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.079+0000 7f7879f97700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f78740a5b70 addr v1:172.21.15.59:6789/0 2023-09-30T02:09:44.082 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.079+0000 7f7879f97700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f7874142290 0x7f78740b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:44.082 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.079+0000 7f7879f97700 10 monclient(hunting): picked mon.noname-a con 0x7f7874142290 addr v2:172.21.15.59:3300/0 2023-09-30T02:09:44.082 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.079+0000 7f7879f97700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f78740a5f40 0x7f787413af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:44.082 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.079+0000 7f7879f97700 10 monclient(hunting): picked mon.noname-c con 0x7f78740a5f40 addr v2:172.21.15.110:3300/0 2023-09-30T02:09:44.082 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.079+0000 7f7879f97700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78740a75a0 con 0x7f78740a5b70 2023-09-30T02:09:44.082 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.079+0000 7f7879f97700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:44.082 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.079+0000 7f7879f97700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78740a6360 con 0x7f7874142290 2023-09-30T02:09:44.083 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.079+0000 7f7879f97700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:44.083 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.079+0000 7f7879f97700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78740af3f0 con 0x7f78740a5f40 2023-09-30T02:09:44.083 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.079+0000 7f7879f97700 10 monclient(hunting): _renew_subs 2023-09-30T02:09:44.083 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.079+0000 7f7879f97700 10 monclient(hunting): authenticate will time out at 2029.382784s 2023-09-30T02:09:44.083 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.080+0000 7f7873fff700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f78740a5f40 0x7f787413af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:44.083 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.081+0000 7f7873fff700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f78740a5f40 0x7f787413af30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:42662/0 (socket says 172.21.15.59:42662) 2023-09-30T02:09:44.083 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.081+0000 7f7873fff700 1 -- 172.21.15.59:0/2887471088 learned_addr learned my addr 172.21.15.59:0/2887471088 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:09:44.083 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.081+0000 7f7873fff700 10 monclient(hunting): get_auth_request con 0x7f78740a5f40 auth_method 0 2023-09-30T02:09:44.083 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.081+0000 7f7873fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:09:44.083 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.081+0000 7f7873fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:44.083 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.081+0000 7f7873fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:09:44.083 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.081+0000 7f7873fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:09:44.084 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.081+0000 7f7873fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:09:44.084 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.081+0000 7f7873fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:09:44.084 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.084+0000 7f7873fff700 10 monclient(hunting): handle_auth_done global_id 4687 payload 274 2023-09-30T02:09:44.084 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.084+0000 7f7873fff700 1 -- 172.21.15.59:0/2887471088 >> v2:172.21.15.59:3300/0 conn(0x7f7874142290 msgr2=0x7f78740b1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:09:44.085 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.084+0000 7f7873fff700 1 --2- 172.21.15.59:0/2887471088 >> v2:172.21.15.59:3300/0 conn(0x7f7874142290 0x7f78740b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:44.085 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.084+0000 7f7873fff700 1 -- 172.21.15.59:0/2887471088 >> v1:172.21.15.59:6789/0 conn(0x7f78740a5b70 legacy=0x7f7874141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:09:44.085 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.084+0000 7f7873fff700 10 monclient: _finish_hunting 0 2023-09-30T02:09:44.085 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.084+0000 7f7873fff700 1 monclient: found mon.noname-c 2023-09-30T02:09:44.085 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.084+0000 7f7873fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:09:44.085 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.084+0000 7f7873fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3300/0 2023-09-30T02:09:44.085 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.084+0000 7f7873fff700 1 -- 172.21.15.59:0/2887471088 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f787413b470 con 0x7f78740a5f40 2023-09-30T02:09:44.086 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.086+0000 7f7873fff700 1 --2- 172.21.15.59:0/2887471088 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f78740a5f40 0x7f787413af30 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f78740b0640 tx=0x7f786c009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=602e3c06761616a2 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:09:44.092 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.091+0000 7f78727fc700 1 -- 172.21.15.59:0/2887471088 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f786c00b820 con 0x7f78740a5f40 2023-09-30T02:09:44.092 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.091+0000 7f78727fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:09:44.092 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.091+0000 7f78727fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:09:44.092 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.091+0000 7f78727fc700 10 monclient: dump: 2023-09-30T02:09:44.092 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:09:44.092 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:44.092 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:44.093 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:44.093 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:09:44.093 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:44.093 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:09:44.093 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:09:44.094 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:09:44.094 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:44.094 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.091+0000 7f78727fc700 10 monclient: _finish_auth 0 2023-09-30T02:09:44.094 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.091+0000 7f78727fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:09:44.094 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.091+0000 7f78727fc700 1 -- 172.21.15.59:0/2887471088 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f786c00b980 con 0x7f78740a5f40 2023-09-30T02:09:44.094 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.091+0000 7f78727fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:09:44.094 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.091+0000 7f78727fc700 1 -- 172.21.15.59:0/2887471088 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f786c00bd20 con 0x7f78740a5f40 2023-09-30T02:09:44.094 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.091+0000 7f78727fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:09:44.094 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.091+0000 7f78727fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:09:44.094 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.091+0000 7f78727fc700 10 monclient: dump: 2023-09-30T02:09:44.094 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:09:44.095 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:44.095 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:44.095 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:44.095 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:09:44.095 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:44.095 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:09:44.095 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:09:44.095 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:09:44.095 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:44.096 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.091+0000 7f7879f97700 5 monclient: authenticate success, global_id 4687 2023-09-30T02:09:44.096 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.091+0000 7f7879f97700 10 monclient: get_monmap_and_config success 2023-09-30T02:09:44.096 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.091+0000 7f7879f97700 4 set_mon_vals no callback set 2023-09-30T02:09:44.099 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.092+0000 7f7879f97700 10 monclient: shutdown 2023-09-30T02:09:44.099 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.092+0000 7f7879f97700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:09:44.099 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.092+0000 7f7879f97700 1 -- 172.21.15.59:0/2887471088 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f78740a5f40 msgr2=0x7f787413af30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:44.099 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.092+0000 7f7879f97700 1 --2- 172.21.15.59:0/2887471088 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f78740a5f40 0x7f787413af30 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f78740b0640 tx=0x7f786c009250 comp rx=0 tx=0).stop 2023-09-30T02:09:44.099 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.092+0000 7f7879f97700 1 -- 172.21.15.59:0/2887471088 shutdown_connections 2023-09-30T02:09:44.099 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.092+0000 7f7879f97700 1 --2- 172.21.15.59:0/2887471088 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f78740a5f40 0x7f787413af30 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:44.099 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.092+0000 7f7879f97700 1 --2- 172.21.15.59:0/2887471088 >> v2:172.21.15.59:3300/0 conn(0x7f7874142290 0x7f78740b1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:44.100 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.092+0000 7f7879f97700 1 -- 172.21.15.59:0/2887471088 >> 172.21.15.59:0/2887471088 conn(0x7f78740631f0 msgr2=0x7f78740635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:09:44.100 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.092+0000 7f7879f97700 1 -- 172.21.15.59:0/2887471088 shutdown_connections 2023-09-30T02:09:44.100 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.092+0000 7f7879f97700 1 -- 172.21.15.59:0/2887471088 wait complete. 2023-09-30T02:09:44.100 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.093+0000 7f7879f97700 10 monclient: build_initial_monmap 2023-09-30T02:09:44.100 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.093+0000 7f7879f97700 10 monclient: monmap: 2023-09-30T02:09:44.100 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:09:44.100 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:09:44.100 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:09:44.100 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:09:44.101 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:09:44.101 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:09:44.101 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:09:44.101 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:09:44.101 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:09:44.101 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:44.101 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.093+0000 7f7879f97700 1 librados: starting msgr at 2023-09-30T02:09:44.101 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.093+0000 7f7879f97700 1 librados: starting objecter 2023-09-30T02:09:44.101 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.093+0000 7f7879f97700 1 Processor -- start 2023-09-30T02:09:44.101 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.093+0000 7f7879f97700 1 -- start start 2023-09-30T02:09:44.101 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.093+0000 7f7879f97700 1 librados: setting wanted keys 2023-09-30T02:09:44.102 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.093+0000 7f7879f97700 1 librados: calling monclient init 2023-09-30T02:09:44.102 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.093+0000 7f7879f97700 10 monclient: init 2023-09-30T02:09:44.102 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.094+0000 7f7879f97700 10 monclient: _reopen_session rank -1 2023-09-30T02:09:44.102 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.094+0000 7f7879f97700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T02:09:44.102 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.094+0000 7f7879f97700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f78740a5b70 0x7f78740a7d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:44.102 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.094+0000 7f7879f97700 10 monclient(hunting): picked mon.noname-b con 0x7f78740a5b70 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:09:44.102 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.094+0000 7f7879f97700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f78740a5f40 0x7f787413e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:44.102 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.094+0000 7f7879f97700 10 monclient(hunting): picked mon.noname-c con 0x7f78740a5f40 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:09:44.102 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.094+0000 7f7879f97700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7874142290 0x7f7874137880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:44.102 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.094+0000 7f7879f97700 10 monclient(hunting): picked mon.noname-a con 0x7f7874142290 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:09:44.102 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.094+0000 7f7879f97700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:44.102 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.094+0000 7f7879f97700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78740a91c0 con 0x7f7874142290 2023-09-30T02:09:44.103 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.094+0000 7f7879f97700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:44.103 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.094+0000 7f7879f97700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f787413c030 con 0x7f78740a5b70 2023-09-30T02:09:44.103 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.094+0000 7f7879f97700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:44.103 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.094+0000 7f7879f97700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78740a82b0 con 0x7f78740a5f40 2023-09-30T02:09:44.103 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.094+0000 7f7879f97700 10 monclient(hunting): _renew_subs 2023-09-30T02:09:44.103 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.094+0000 7f7879f97700 10 monclient(hunting): authenticate will time out at 2029.397555s 2023-09-30T02:09:44.103 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.095+0000 7f78737fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f78740a5b70 0x7f78740a7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:44.103 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.095+0000 7f7872ffd700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f78740a5f40 0x7f787413e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:44.104 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.095+0000 7f78737fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f78740a5b70 0x7f78740a7d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:42676/0 (socket says 172.21.15.59:42676) 2023-09-30T02:09:44.104 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.095+0000 7f78737fe700 1 -- 172.21.15.59:0/1476175935 learned_addr learned my addr 172.21.15.59:0/1476175935 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:09:44.104 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.095+0000 7f78737fe700 10 monclient(hunting): get_auth_request con 0x7f78740a5b70 auth_method 0 2023-09-30T02:09:44.104 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.095+0000 7f78737fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:09:44.104 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.095+0000 7f78737fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:44.104 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.095+0000 7f78737fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:09:44.104 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.095+0000 7f7872ffd700 10 monclient(hunting): get_auth_request con 0x7f78740a5f40 auth_method 0 2023-09-30T02:09:44.104 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.095+0000 7f7872ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:09:44.104 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.095+0000 7f7872ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:44.104 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.095+0000 7f7872ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:09:44.104 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.096+0000 7f78737fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:09:44.104 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.096+0000 7f78737fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:09:44.105 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.096+0000 7f78737fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:09:44.105 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.096+0000 7f7872ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:09:44.105 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.096+0000 7f7872ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:09:44.105 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.096+0000 7f7872ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:09:44.106 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.100+0000 7f78737fe700 10 monclient(hunting): handle_auth_done global_id 4690 payload 824 2023-09-30T02:09:44.106 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.100+0000 7f78737fe700 1 -- 172.21.15.59:0/1476175935 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f78740a5f40 msgr2=0x7f787413e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:44.106 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.100+0000 7f78737fe700 1 --2- 172.21.15.59:0/1476175935 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f78740a5f40 0x7f787413e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:44.106 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.100+0000 7f78737fe700 1 -- 172.21.15.59:0/1476175935 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7874142290 msgr2=0x7f7874137880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:44.106 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.100+0000 7f78737fe700 1 --2- 172.21.15.59:0/1476175935 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7874142290 0x7f7874137880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:44.106 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.100+0000 7f78737fe700 10 monclient: _finish_hunting 0 2023-09-30T02:09:44.106 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.100+0000 7f78737fe700 1 monclient: found mon.noname-b 2023-09-30T02:09:44.106 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.100+0000 7f78737fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:09:44.107 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.100+0000 7f78737fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:09:44.107 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.100+0000 7f78737fe700 1 -- 172.21.15.59:0/1476175935 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f786c0094f0 con 0x7f78740a5b70 2023-09-30T02:09:44.107 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.103+0000 7f78737fe700 1 --2- 172.21.15.59:0/1476175935 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f78740a5b70 0x7f78740a7d70 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f786400bbb0 tx=0x7f7864009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:09:44.107 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.103+0000 7f785b7fe700 10 client.?.objecter ms_handle_connect 0x7f78740a5b70 2023-09-30T02:09:44.107 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.103+0000 7f785b7fe700 10 client.?.objecter resend_mon_ops 2023-09-30T02:09:44.108 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.107+0000 7f785b7fe700 1 -- 172.21.15.59:0/1476175935 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78640092e0 con 0x7f78740a5b70 2023-09-30T02:09:44.108 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.107+0000 7f785b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:09:44.108 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.107+0000 7f785b7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:09:44.109 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.107+0000 7f785b7fe700 10 monclient: dump: 2023-09-30T02:09:44.109 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:09:44.109 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:44.109 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:44.109 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:44.109 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:09:44.109 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:44.109 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:09:44.109 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:09:44.109 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:09:44.109 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:44.109 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.107+0000 7f785b7fe700 10 monclient: _finish_auth 0 2023-09-30T02:09:44.110 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.107+0000 7f785b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:09:44.110 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.107+0000 7f785b7fe700 1 -- 172.21.15.59:0/1476175935 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f786400c6f0 con 0x7f78740a5b70 2023-09-30T02:09:44.110 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.107+0000 7f785b7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:09:44.110 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.107+0000 7f785b7fe700 1 -- 172.21.15.59:0/1476175935 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f786400ca90 con 0x7f78740a5b70 2023-09-30T02:09:44.110 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.107+0000 7f7879f97700 5 monclient: authenticate success, global_id 4690 2023-09-30T02:09:44.110 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.107+0000 7f7879f97700 10 monclient: _renew_subs 2023-09-30T02:09:44.110 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.107+0000 7f7879f97700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:09:44.110 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.107+0000 7f7879f97700 1 -- 172.21.15.59:0/1476175935 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f787413edb0 con 0x7f78740a5b70 2023-09-30T02:09:44.110 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.107+0000 7f785b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:09:44.110 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.107+0000 7f785b7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:09:44.110 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.107+0000 7f7871ffb700 4 set_mon_vals no callback set 2023-09-30T02:09:44.110 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.107+0000 7f785b7fe700 10 monclient: dump: 2023-09-30T02:09:44.111 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:09:44.111 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:44.111 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:44.111 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:44.111 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:09:44.111 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:44.111 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:09:44.111 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:09:44.111 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:09:44.111 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:44.111 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.107+0000 7f7879f97700 10 client.4690.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:09:44.111 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.107+0000 7f7879f97700 10 monclient: _renew_subs 2023-09-30T02:09:44.112 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.107+0000 7f7879f97700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:09:44.112 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.107+0000 7f7879f97700 1 -- 172.21.15.59:0/1476175935 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f787413f0e0 con 0x7f78740a5b70 2023-09-30T02:09:44.112 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.107+0000 7f7879f97700 1 librados: init done 2023-09-30T02:09:44.112 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.108+0000 7f7879f97700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:09:44.112 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.108+0000 7f7879f97700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:09:44.112 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.108+0000 7f7879f97700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:09:44.112 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.108+0000 7f7879f97700 1 -- 172.21.15.59:0/1476175935 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f787413c270 con 0x7f78740a5b70 2023-09-30T02:09:44.137 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.136+0000 7f785b7fe700 1 -- 172.21.15.59:0/1476175935 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f7864012070 con 0x7f78740a5b70 2023-09-30T02:09:44.137 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.136+0000 7f785b7fe700 1 --2- 172.21.15.59:0/1476175935 >> v2:172.21.15.110:6800/115870 conn(0x7f7860068910 0x7f786006adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:44.137 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.137+0000 7f785b7fe700 1 -- 172.21.15.59:0/1476175935 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f7864083d30 con 0x7f78740a5b70 2023-09-30T02:09:44.137 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.137+0000 7f785b7fe700 10 client.4690.objecter ms_dispatch 0x7f7874001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:09:44.138 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.137+0000 7f785b7fe700 3 client.4690.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:09:44.138 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.137+0000 7f785b7fe700 3 client.4690.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:09:44.138 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.137+0000 7f785b7fe700 20 client.4690.objecter dump_active .. 0 homeless 2023-09-30T02:09:44.138 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.137+0000 7f7872ffd700 1 --2- 172.21.15.59:0/1476175935 >> v2:172.21.15.110:6800/115870 conn(0x7f7860068910 0x7f786006adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:44.138 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.137+0000 7f7872ffd700 10 monclient: get_auth_request con 0x7f7860068910 auth_method 0 2023-09-30T02:09:44.138 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.137+0000 7f7872ffd700 1 --2- 172.21.15.59:0/1476175935 >> v2:172.21.15.110:6800/115870 conn(0x7f7860068910 0x7f786006adc0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f7868009b40 tx=0x7f78680090f0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:09:44.138 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.137+0000 7f785b7fe700 10 client.4690.objecter ms_handle_connect 0x7f7860068910 2023-09-30T02:09:44.433 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.433+0000 7f785b7fe700 1 -- 172.21.15.59:0/1476175935 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f786404d5a0 con 0x7f78740a5b70 2023-09-30T02:09:44.433 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.433+0000 7f785b7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:09:44.434 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.433+0000 7f785b7fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:09:44.685 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.685+0000 7f7879f97700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:09:44.685 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.685+0000 7f7879f97700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:09:44.685 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.685+0000 7f7879f97700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:09:44.685 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.685+0000 7f7879f97700 1 -- 172.21.15.59:0/1476175935 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f7874064a00 con 0x7f78740a5b70 2023-09-30T02:09:44.690 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.690+0000 7f785b7fe700 1 -- 172.21.15.59:0/1476175935 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f786401a100 con 0x7f78740a5b70 2023-09-30T02:09:44.690 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.690+0000 7f785b7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:09:44.690 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.690+0000 7f785b7fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:09:44.690 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:09:44.691 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:09:44.693 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.692+0000 7f7879f97700 10 librados: watch_flush enter 2023-09-30T02:09:44.693 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.692+0000 7f7879f97700 10 librados: watch_flush exit 2023-09-30T02:09:44.693 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.692+0000 7f7879f97700 20 client.4690.objecter shutdown clearing up homeless session... 2023-09-30T02:09:44.693 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.692+0000 7f7879f97700 10 client.4690.objecter successfully canceled tick 2023-09-30T02:09:44.693 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.692+0000 7f7879f97700 1 -- 172.21.15.59:0/1476175935 >> v2:172.21.15.110:6800/115870 conn(0x7f7860068910 msgr2=0x7f786006adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:44.693 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.692+0000 7f7879f97700 1 --2- 172.21.15.59:0/1476175935 >> v2:172.21.15.110:6800/115870 conn(0x7f7860068910 0x7f786006adc0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f7868009b40 tx=0x7f78680090f0 comp rx=0 tx=0).stop 2023-09-30T02:09:44.693 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.692+0000 7f7879f97700 10 monclient: shutdown 2023-09-30T02:09:44.693 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.692+0000 7f7879f97700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:09:44.694 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.692+0000 7f7879f97700 1 -- 172.21.15.59:0/1476175935 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f78740a5b70 msgr2=0x7f78740a7d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:44.694 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.692+0000 7f7879f97700 1 --2- 172.21.15.59:0/1476175935 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f78740a5b70 0x7f78740a7d70 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f786400bbb0 tx=0x7f7864009840 comp rx=0 tx=0).stop 2023-09-30T02:09:44.694 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.692+0000 7f7879f97700 1 -- 172.21.15.59:0/1476175935 shutdown_connections 2023-09-30T02:09:44.694 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.692+0000 7f7879f97700 1 --2- 172.21.15.59:0/1476175935 >> v2:172.21.15.110:6800/115870 conn(0x7f7860068910 0x7f786006adc0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:44.694 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.692+0000 7f7879f97700 1 --2- 172.21.15.59:0/1476175935 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f78740a5b70 0x7f78740a7d70 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:44.694 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.692+0000 7f7879f97700 1 --2- 172.21.15.59:0/1476175935 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7874142290 0x7f7874137880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:44.694 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.692+0000 7f7879f97700 1 --2- 172.21.15.59:0/1476175935 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f78740a5f40 0x7f787413e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:44.694 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.692+0000 7f7879f97700 1 -- 172.21.15.59:0/1476175935 >> 172.21.15.59:0/1476175935 conn(0x7f78740631f0 msgr2=0x7f78740b3360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:09:44.694 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.692+0000 7f7879f97700 1 -- 172.21.15.59:0/1476175935 shutdown_connections 2023-09-30T02:09:44.694 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.692+0000 7f7879f97700 1 -- 172.21.15.59:0/1476175935 wait complete. 2023-09-30T02:09:44.694 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:44.692+0000 7f7879f97700 1 librados: shutdown 2023-09-30T02:09:47.708 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:09:47.851 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.849+0000 7f8bd055c700 10 monclient: get_monmap_and_config 2023-09-30T02:09:47.851 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.849+0000 7f8bd055c700 10 monclient: build_initial_monmap 2023-09-30T02:09:47.851 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.849+0000 7f8bd055c700 10 monclient: monmap: 2023-09-30T02:09:47.851 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:09:47.851 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:47.851 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:09:47.851232+0000 2023-09-30T02:09:47.851 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:09:47.851232+0000 2023-09-30T02:09:47.851 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:09:47.851 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:47.852 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:09:47.852 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:09:47.852 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:09:47.852 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:09:47.852 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:09:47.852 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:47.852 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.850+0000 7f8bd055c700 1 Processor -- start 2023-09-30T02:09:47.852 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.850+0000 7f8bd055c700 1 -- start start 2023-09-30T02:09:47.852 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.850+0000 7f8bd055c700 10 monclient: init 2023-09-30T02:09:47.854 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.851+0000 7f8bd055c700 10 monclient: _reopen_session rank -1 2023-09-30T02:09:47.854 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.851+0000 7f8bd055c700 10 monclient: _add_conns ranks=[3,4,2,1,0] 2023-09-30T02:09:47.854 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.851+0000 7f8bd055c700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f8bc80a5b70 0x7f8bc80b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:47.855 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.851+0000 7f8bd055c700 10 monclient(hunting): picked mon.noname-a con 0x7f8bc80a5b70 addr v2:172.21.15.59:3300/0 2023-09-30T02:09:47.855 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.851+0000 7f8bd055c700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f8bc80a5f40 0x7f8bc8141b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:47.855 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.851+0000 7f8bd055c700 10 monclient(hunting): picked mon.noname-c con 0x7f8bc80a5f40 addr v2:172.21.15.110:3300/0 2023-09-30T02:09:47.855 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.851+0000 7f8bd055c700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8bc81420c0 0x7f8bc813e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:47.855 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.851+0000 7f8bd055c700 10 monclient(hunting): picked mon.noname-b con 0x7f8bc81420c0 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:09:47.856 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.851+0000 7f8bd055c700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:47.856 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.851+0000 7f8bd055c700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bc80a63e0 con 0x7f8bc80a5b70 2023-09-30T02:09:47.856 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.851+0000 7f8bd055c700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:47.856 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.851+0000 7f8bd055c700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bc80a75a0 con 0x7f8bc80a5f40 2023-09-30T02:09:47.856 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.851+0000 7f8bd055c700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:47.856 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.851+0000 7f8bd055c700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc80af3f0 con 0x7f8bc81420c0 2023-09-30T02:09:47.857 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.851+0000 7f8bd055c700 10 monclient(hunting): _renew_subs 2023-09-30T02:09:47.857 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.851+0000 7f8bd055c700 10 monclient(hunting): authenticate will time out at 2033.154469s 2023-09-30T02:09:47.857 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.852+0000 7f8bceaf9700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8bc81420c0 0x7f8bc813e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:47.857 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.852+0000 7f8bcdaf7700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f8bc80a5f40 0x7f8bc8141b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:47.857 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.852+0000 7f8bce2f8700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f8bc80a5b70 0x7f8bc80b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:47.857 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.852+0000 7f8bceaf9700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8bc81420c0 0x7f8bc813e230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:42724/0 (socket says 172.21.15.59:42724) 2023-09-30T02:09:47.857 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.852+0000 7f8bceaf9700 1 -- 172.21.15.59:0/4127612898 learned_addr learned my addr 172.21.15.59:0/4127612898 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:09:47.857 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.852+0000 7f8bceaf9700 10 monclient(hunting): get_auth_request con 0x7f8bc81420c0 auth_method 0 2023-09-30T02:09:47.857 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.852+0000 7f8bceaf9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:09:47.857 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.852+0000 7f8bceaf9700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:47.857 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.852+0000 7f8bceaf9700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:09:47.858 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.852+0000 7f8bcdaf7700 10 monclient(hunting): get_auth_request con 0x7f8bc80a5f40 auth_method 0 2023-09-30T02:09:47.858 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.852+0000 7f8bcdaf7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:09:47.858 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.852+0000 7f8bcdaf7700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:47.858 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.852+0000 7f8bcdaf7700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:09:47.858 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.852+0000 7f8bce2f8700 10 monclient(hunting): get_auth_request con 0x7f8bc80a5b70 auth_method 0 2023-09-30T02:09:47.858 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.852+0000 7f8bce2f8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:09:47.858 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.852+0000 7f8bce2f8700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:47.858 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.852+0000 7f8bce2f8700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:09:47.858 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.853+0000 7f8bceaf9700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:09:47.858 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.853+0000 7f8bceaf9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:09:47.859 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.853+0000 7f8bceaf9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:09:47.859 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.853+0000 7f8bcdaf7700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:09:47.859 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.853+0000 7f8bcdaf7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:09:47.859 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.853+0000 7f8bcdaf7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:09:47.859 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.853+0000 7f8bce2f8700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:09:47.859 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.853+0000 7f8bce2f8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:09:47.859 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.853+0000 7f8bce2f8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:09:47.860 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.855+0000 7f8bceaf9700 10 monclient(hunting): handle_auth_done global_id 4595 payload 274 2023-09-30T02:09:47.860 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.855+0000 7f8bceaf9700 1 -- 172.21.15.59:0/4127612898 >> v2:172.21.15.110:3300/0 conn(0x7f8bc80a5f40 msgr2=0x7f8bc8141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:09:47.860 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.855+0000 7f8bceaf9700 1 --2- 172.21.15.59:0/4127612898 >> v2:172.21.15.110:3300/0 conn(0x7f8bc80a5f40 0x7f8bc8141b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:47.860 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.855+0000 7f8bceaf9700 1 -- 172.21.15.59:0/4127612898 >> v2:172.21.15.59:3300/0 conn(0x7f8bc80a5b70 msgr2=0x7f8bc80b1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:09:47.860 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.855+0000 7f8bceaf9700 1 --2- 172.21.15.59:0/4127612898 >> v2:172.21.15.59:3300/0 conn(0x7f8bc80a5b70 0x7f8bc80b1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:47.860 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.855+0000 7f8bceaf9700 10 monclient: _finish_hunting 0 2023-09-30T02:09:47.860 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.855+0000 7f8bceaf9700 1 monclient: found mon.noname-b 2023-09-30T02:09:47.860 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.855+0000 7f8bceaf9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:09:47.861 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.855+0000 7f8bceaf9700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.59:3301/0 2023-09-30T02:09:47.861 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.855+0000 7f8bceaf9700 1 -- 172.21.15.59:0/4127612898 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bc8142500 con 0x7f8bc81420c0 2023-09-30T02:09:47.861 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.858+0000 7f8bceaf9700 1 --2- 172.21.15.59:0/4127612898 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8bc81420c0 0x7f8bc813e230 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f8bc80b1fc0 tx=0x7f8bc0009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f6281447aa1da55 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:09:47.863 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.862+0000 7f8bcd2f6700 1 -- 172.21.15.59:0/4127612898 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bc000e070 con 0x7f8bc81420c0 2023-09-30T02:09:47.863 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.862+0000 7f8bcd2f6700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:09:47.863 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.862+0000 7f8bcd2f6700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:09:47.863 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.862+0000 7f8bcd2f6700 10 monclient: dump: 2023-09-30T02:09:47.863 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:09:47.863 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:47.863 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:47.863 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:47.864 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:09:47.864 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:47.864 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:09:47.864 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:09:47.864 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:09:47.864 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:47.864 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.862+0000 7f8bcd2f6700 10 monclient: _finish_auth 0 2023-09-30T02:09:47.864 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.862+0000 7f8bcd2f6700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:09:47.864 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.862+0000 7f8bcd2f6700 1 -- 172.21.15.59:0/4127612898 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bc000b970 con 0x7f8bc81420c0 2023-09-30T02:09:47.864 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.862+0000 7f8bd055c700 5 monclient: authenticate success, global_id 4595 2023-09-30T02:09:47.864 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.862+0000 7f8bd055c700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:09:47.865 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.862+0000 7f8bcd2f6700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:09:47.865 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.862+0000 7f8bd055c700 10 monclient: get_monmap_and_config success 2023-09-30T02:09:47.865 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.862+0000 7f8bd055c700 4 set_mon_vals no callback set 2023-09-30T02:09:47.865 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.862+0000 7f8bcd2f6700 1 -- 172.21.15.59:0/4127612898 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bc00092e0 con 0x7f8bc81420c0 2023-09-30T02:09:47.865 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.862+0000 7f8bcd2f6700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:09:47.865 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.862+0000 7f8bcd2f6700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:09:47.865 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.862+0000 7f8bcd2f6700 10 monclient: dump: 2023-09-30T02:09:47.865 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:09:47.865 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:47.865 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:47.865 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:47.866 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:09:47.866 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:47.866 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:09:47.866 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:09:47.866 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:09:47.866 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:47.870 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.863+0000 7f8bd055c700 10 monclient: shutdown 2023-09-30T02:09:47.870 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.863+0000 7f8bd055c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:09:47.870 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.863+0000 7f8bd055c700 1 -- 172.21.15.59:0/4127612898 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8bc81420c0 msgr2=0x7f8bc813e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:47.870 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.863+0000 7f8bd055c700 1 --2- 172.21.15.59:0/4127612898 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8bc81420c0 0x7f8bc813e230 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f8bc80b1fc0 tx=0x7f8bc0009840 comp rx=0 tx=0).stop 2023-09-30T02:09:47.870 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.863+0000 7f8bd055c700 1 -- 172.21.15.59:0/4127612898 shutdown_connections 2023-09-30T02:09:47.870 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.863+0000 7f8bd055c700 1 --2- 172.21.15.59:0/4127612898 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8bc81420c0 0x7f8bc813e230 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:47.871 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.863+0000 7f8bd055c700 1 --2- 172.21.15.59:0/4127612898 >> v2:172.21.15.59:3300/0 conn(0x7f8bc80a5b70 0x7f8bc80b1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:47.871 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.863+0000 7f8bd055c700 1 --2- 172.21.15.59:0/4127612898 >> v2:172.21.15.110:3300/0 conn(0x7f8bc80a5f40 0x7f8bc8141b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:47.871 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.863+0000 7f8bd055c700 1 -- 172.21.15.59:0/4127612898 >> 172.21.15.59:0/4127612898 conn(0x7f8bc80631f0 msgr2=0x7f8bc80635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:09:47.871 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.863+0000 7f8bd055c700 1 -- 172.21.15.59:0/4127612898 shutdown_connections 2023-09-30T02:09:47.871 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.863+0000 7f8bd055c700 1 -- 172.21.15.59:0/4127612898 wait complete. 2023-09-30T02:09:47.871 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.864+0000 7f8bd055c700 10 monclient: build_initial_monmap 2023-09-30T02:09:47.871 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.864+0000 7f8bd055c700 10 monclient: monmap: 2023-09-30T02:09:47.871 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:09:47.871 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:09:47.871 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:09:47.872 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:09:47.872 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:09:47.872 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:09:47.872 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:09:47.872 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:09:47.872 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:09:47.872 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:47.872 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.864+0000 7f8bd055c700 1 librados: starting msgr at 2023-09-30T02:09:47.872 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.864+0000 7f8bd055c700 1 librados: starting objecter 2023-09-30T02:09:47.872 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.864+0000 7f8bd055c700 1 Processor -- start 2023-09-30T02:09:47.873 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.864+0000 7f8bd055c700 1 -- start start 2023-09-30T02:09:47.873 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.864+0000 7f8bd055c700 1 librados: setting wanted keys 2023-09-30T02:09:47.873 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.864+0000 7f8bd055c700 1 librados: calling monclient init 2023-09-30T02:09:47.873 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.864+0000 7f8bd055c700 10 monclient: init 2023-09-30T02:09:47.873 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.865+0000 7f8bd055c700 10 monclient: _reopen_session rank -1 2023-09-30T02:09:47.873 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.865+0000 7f8bd055c700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T02:09:47.873 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.865+0000 7f8bd055c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8bc80a5b70 0x7f8bc813b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:47.873 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.865+0000 7f8bd055c700 10 monclient(hunting): picked mon.noname-a con 0x7f8bc80a5b70 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:09:47.873 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.865+0000 7f8bd055c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8bc80a5f40 0x7f8bc80aa040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:47.874 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.865+0000 7f8bd055c700 10 monclient(hunting): picked mon.noname-b con 0x7f8bc80a5f40 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:09:47.874 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.865+0000 7f8bd055c700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8bc81420c0 0x7f8bc80aa580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:47.874 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.865+0000 7f8bd055c700 10 monclient(hunting): picked mon.noname-c con 0x7f8bc81420c0 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:09:47.874 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.865+0000 7f8bd055c700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:47.874 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.865+0000 7f8bd055c700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc80a98d0 con 0x7f8bc80a5b70 2023-09-30T02:09:47.874 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.865+0000 7f8bd055c700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:47.874 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.865+0000 7f8bd055c700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc80a9bb0 con 0x7f8bc80a5f40 2023-09-30T02:09:47.874 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.865+0000 7f8bd055c700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:47.874 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.865+0000 7f8bd055c700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc80abab0 con 0x7f8bc81420c0 2023-09-30T02:09:47.875 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.865+0000 7f8bd055c700 10 monclient(hunting): _renew_subs 2023-09-30T02:09:47.875 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.865+0000 7f8bd055c700 10 monclient(hunting): authenticate will time out at 2033.168659s 2023-09-30T02:09:47.875 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.866+0000 7f8bcdaf7700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8bc80a5f40 0x7f8bc80aa040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:47.875 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.866+0000 7f8bce2f8700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8bc80a5b70 0x7f8bc813b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:47.875 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.866+0000 7f8bcdaf7700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8bc80a5f40 0x7f8bc80aa040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:49732/0 (socket says 172.21.15.59:49732) 2023-09-30T02:09:47.875 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.866+0000 7f8bcdaf7700 1 -- 172.21.15.59:0/3580378227 learned_addr learned my addr 172.21.15.59:0/3580378227 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:09:47.875 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.866+0000 7f8bcdaf7700 10 monclient(hunting): get_auth_request con 0x7f8bc80a5f40 auth_method 0 2023-09-30T02:09:47.875 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.866+0000 7f8bcdaf7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:09:47.875 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.866+0000 7f8bcdaf7700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:47.875 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.866+0000 7f8bcdaf7700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:09:47.875 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.866+0000 7f8bceaf9700 1 --2- 172.21.15.59:0/3580378227 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8bc81420c0 0x7f8bc80aa580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:47.876 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.866+0000 7f8bce2f8700 10 monclient(hunting): get_auth_request con 0x7f8bc80a5b70 auth_method 0 2023-09-30T02:09:47.876 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.866+0000 7f8bce2f8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:09:47.876 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.866+0000 7f8bce2f8700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:47.876 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.866+0000 7f8bce2f8700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:09:47.876 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.867+0000 7f8bcdaf7700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:09:47.876 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.867+0000 7f8bcdaf7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:09:47.876 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.867+0000 7f8bcdaf7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:09:47.876 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.867+0000 7f8bceaf9700 10 monclient(hunting): get_auth_request con 0x7f8bc81420c0 auth_method 0 2023-09-30T02:09:47.876 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.867+0000 7f8bceaf9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:09:47.877 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.867+0000 7f8bceaf9700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:47.877 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.867+0000 7f8bceaf9700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:09:47.877 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.867+0000 7f8bce2f8700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:09:47.877 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.867+0000 7f8bce2f8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:09:47.877 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.867+0000 7f8bce2f8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:09:47.877 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.868+0000 7f8bceaf9700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:09:47.878 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.868+0000 7f8bceaf9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:09:47.878 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.868+0000 7f8bceaf9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:09:47.880 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.871+0000 7f8bcdaf7700 10 monclient(hunting): handle_auth_done global_id 4696 payload 824 2023-09-30T02:09:47.880 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.871+0000 7f8bcdaf7700 1 -- 172.21.15.59:0/3580378227 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8bc81420c0 msgr2=0x7f8bc80aa580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:47.880 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.871+0000 7f8bcdaf7700 1 --2- 172.21.15.59:0/3580378227 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8bc81420c0 0x7f8bc80aa580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:47.880 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.871+0000 7f8bcdaf7700 1 -- 172.21.15.59:0/3580378227 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8bc80a5b70 msgr2=0x7f8bc813b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:47.880 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.871+0000 7f8bcdaf7700 1 --2- 172.21.15.59:0/3580378227 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8bc80a5b70 0x7f8bc813b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:47.880 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.871+0000 7f8bcdaf7700 10 monclient: _finish_hunting 0 2023-09-30T02:09:47.881 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.871+0000 7f8bcdaf7700 1 monclient: found mon.noname-b 2023-09-30T02:09:47.881 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.871+0000 7f8bcdaf7700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:09:47.881 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.871+0000 7f8bcdaf7700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:09:47.881 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.871+0000 7f8bcdaf7700 1 -- 172.21.15.59:0/3580378227 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bc0009ae0 con 0x7f8bc80a5f40 2023-09-30T02:09:47.881 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.874+0000 7f8bcdaf7700 1 --2- 172.21.15.59:0/3580378227 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8bc80a5f40 0x7f8bc80aa040 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f8bbc00ec80 tx=0x7f8bbc0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:09:47.881 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.874+0000 7f8bba7fc700 10 client.?.objecter ms_handle_connect 0x7f8bc80a5f40 2023-09-30T02:09:47.881 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.874+0000 7f8bba7fc700 10 client.?.objecter resend_mon_ops 2023-09-30T02:09:47.881 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.878+0000 7f8bba7fc700 1 -- 172.21.15.59:0/3580378227 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bbc00c850 con 0x7f8bc80a5f40 2023-09-30T02:09:47.881 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.878+0000 7f8bba7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:09:47.882 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.878+0000 7f8bba7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:09:47.882 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.878+0000 7f8bba7fc700 10 monclient: dump: 2023-09-30T02:09:47.882 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:09:47.882 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:47.882 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:47.882 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:47.882 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:09:47.882 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:47.882 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:09:47.883 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:09:47.883 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:09:47.883 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:47.883 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.878+0000 7f8bba7fc700 10 monclient: _finish_auth 0 2023-09-30T02:09:47.883 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.878+0000 7f8bba7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:09:47.883 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.878+0000 7f8bba7fc700 1 -- 172.21.15.59:0/3580378227 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bbc00c9b0 con 0x7f8bc80a5f40 2023-09-30T02:09:47.883 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.878+0000 7f8bba7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:09:47.883 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.878+0000 7f8bba7fc700 1 -- 172.21.15.59:0/3580378227 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bbc00cd50 con 0x7f8bc80a5f40 2023-09-30T02:09:47.883 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.878+0000 7f8bba7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:09:47.883 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.878+0000 7f8bba7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:09:47.884 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.878+0000 7f8bba7fc700 10 monclient: dump: 2023-09-30T02:09:47.884 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:09:47.884 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:47.884 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:47.884 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:47.884 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:09:47.884 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:47.884 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:09:47.884 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:09:47.885 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:09:47.885 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:47.885 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.878+0000 7f8bd055c700 5 monclient: authenticate success, global_id 4696 2023-09-30T02:09:47.885 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.878+0000 7f8bccaf5700 4 set_mon_vals no callback set 2023-09-30T02:09:47.885 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.878+0000 7f8bd055c700 10 monclient: _renew_subs 2023-09-30T02:09:47.885 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.878+0000 7f8bd055c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:09:47.885 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.878+0000 7f8bd055c700 1 -- 172.21.15.59:0/3580378227 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bc80a8810 con 0x7f8bc80a5f40 2023-09-30T02:09:47.885 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.878+0000 7f8bd055c700 10 client.4696.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:09:47.885 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.878+0000 7f8bd055c700 10 monclient: _renew_subs 2023-09-30T02:09:47.886 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.878+0000 7f8bd055c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:09:47.886 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.878+0000 7f8bd055c700 1 -- 172.21.15.59:0/3580378227 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bc80a9450 con 0x7f8bc80a5f40 2023-09-30T02:09:47.886 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.878+0000 7f8bd055c700 1 librados: init done 2023-09-30T02:09:47.886 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.879+0000 7f8bd055c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:09:47.886 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.879+0000 7f8bd055c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:09:47.886 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.879+0000 7f8bd055c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:09:47.886 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:47.879+0000 7f8bd055c700 1 -- 172.21.15.59:0/3580378227 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bc80a96d0 con 0x7f8bc80a5f40 2023-09-30T02:09:48.106 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:48.106+0000 7f8bba7fc700 1 -- 172.21.15.59:0/3580378227 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f8bbc013680 con 0x7f8bc80a5f40 2023-09-30T02:09:48.107 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:48.106+0000 7f8bba7fc700 1 --2- 172.21.15.59:0/3580378227 >> v2:172.21.15.110:6800/115870 conn(0x7f8bac0689e0 0x7f8bac06ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:48.107 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:48.106+0000 7f8bba7fc700 1 -- 172.21.15.59:0/3580378227 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f8bbc0848a0 con 0x7f8bc80a5f40 2023-09-30T02:09:48.107 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:48.106+0000 7f8bba7fc700 10 client.4696.objecter ms_dispatch 0x7f8bc8001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:09:48.107 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:48.106+0000 7f8bba7fc700 3 client.4696.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:09:48.108 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:48.106+0000 7f8bba7fc700 3 client.4696.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:09:48.108 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:48.106+0000 7f8bba7fc700 20 client.4696.objecter dump_active .. 0 homeless 2023-09-30T02:09:48.108 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:48.106+0000 7f8bce2f8700 1 --2- 172.21.15.59:0/3580378227 >> v2:172.21.15.110:6800/115870 conn(0x7f8bac0689e0 0x7f8bac06ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:48.108 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:48.106+0000 7f8bce2f8700 10 monclient: get_auth_request con 0x7f8bac0689e0 auth_method 0 2023-09-30T02:09:48.108 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:48.107+0000 7f8bce2f8700 1 --2- 172.21.15.59:0/3580378227 >> v2:172.21.15.110:6800/115870 conn(0x7f8bac0689e0 0x7f8bac06ae90 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f8bc4009db0 tx=0x7f8bc40091a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:09:48.108 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:48.107+0000 7f8bba7fc700 10 client.4696.objecter ms_handle_connect 0x7f8bac0689e0 2023-09-30T02:09:48.143 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:48.143+0000 7f8bba7fc700 1 -- 172.21.15.59:0/3580378227 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8bbc0521c0 con 0x7f8bc80a5f40 2023-09-30T02:09:48.143 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:48.143+0000 7f8bba7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:09:48.143 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:48.143+0000 7f8bba7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:09:48.400 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:48.400+0000 7f8bd055c700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:09:48.401 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:48.400+0000 7f8bd055c700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:09:48.401 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:48.400+0000 7f8bd055c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:09:48.401 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:48.400+0000 7f8bd055c700 1 -- 172.21.15.59:0/3580378227 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f8bc8064a00 con 0x7f8bc80a5f40 2023-09-30T02:09:48.406 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:48.405+0000 7f8bba7fc700 1 -- 172.21.15.59:0/3580378227 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f8bbc0557e0 con 0x7f8bc80a5f40 2023-09-30T02:09:48.406 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:48.405+0000 7f8bba7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:09:48.406 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:48.405+0000 7f8bba7fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:09:48.406 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:09:48.406 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:09:48.408 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:48.407+0000 7f8bd055c700 10 librados: watch_flush enter 2023-09-30T02:09:48.409 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:48.407+0000 7f8bd055c700 10 librados: watch_flush exit 2023-09-30T02:09:48.409 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:48.407+0000 7f8bd055c700 20 client.4696.objecter shutdown clearing up homeless session... 2023-09-30T02:09:48.409 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:48.407+0000 7f8bd055c700 10 client.4696.objecter successfully canceled tick 2023-09-30T02:09:48.409 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:48.408+0000 7f8bd055c700 1 -- 172.21.15.59:0/3580378227 >> v2:172.21.15.110:6800/115870 conn(0x7f8bac0689e0 msgr2=0x7f8bac06ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:48.409 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:48.408+0000 7f8bd055c700 1 --2- 172.21.15.59:0/3580378227 >> v2:172.21.15.110:6800/115870 conn(0x7f8bac0689e0 0x7f8bac06ae90 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f8bc4009db0 tx=0x7f8bc40091a0 comp rx=0 tx=0).stop 2023-09-30T02:09:48.409 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:48.408+0000 7f8bd055c700 10 monclient: shutdown 2023-09-30T02:09:48.409 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:48.408+0000 7f8bd055c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:09:48.409 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:48.408+0000 7f8bd055c700 1 -- 172.21.15.59:0/3580378227 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8bc80a5f40 msgr2=0x7f8bc80aa040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:48.409 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:48.408+0000 7f8bd055c700 1 --2- 172.21.15.59:0/3580378227 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8bc80a5f40 0x7f8bc80aa040 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f8bbc00ec80 tx=0x7f8bbc0091a0 comp rx=0 tx=0).stop 2023-09-30T02:09:48.409 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:48.408+0000 7f8bd055c700 1 -- 172.21.15.59:0/3580378227 shutdown_connections 2023-09-30T02:09:48.409 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:48.408+0000 7f8bd055c700 1 --2- 172.21.15.59:0/3580378227 >> v2:172.21.15.110:6800/115870 conn(0x7f8bac0689e0 0x7f8bac06ae90 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:48.409 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:48.408+0000 7f8bd055c700 1 --2- 172.21.15.59:0/3580378227 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8bc81420c0 0x7f8bc80aa580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:48.410 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:48.408+0000 7f8bd055c700 1 --2- 172.21.15.59:0/3580378227 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8bc80a5b70 0x7f8bc813b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:48.410 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:48.408+0000 7f8bd055c700 1 --2- 172.21.15.59:0/3580378227 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8bc80a5f40 0x7f8bc80aa040 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:48.410 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:48.408+0000 7f8bd055c700 1 -- 172.21.15.59:0/3580378227 >> 172.21.15.59:0/3580378227 conn(0x7f8bc80631f0 msgr2=0x7f8bc80b0be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:09:48.410 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:48.408+0000 7f8bd055c700 1 -- 172.21.15.59:0/3580378227 shutdown_connections 2023-09-30T02:09:48.410 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:48.408+0000 7f8bd055c700 1 -- 172.21.15.59:0/3580378227 wait complete. 2023-09-30T02:09:48.410 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:48.408+0000 7f8bd055c700 1 librados: shutdown 2023-09-30T02:09:51.423 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:09:51.560 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.560+0000 7fbb16573700 10 monclient: get_monmap_and_config 2023-09-30T02:09:51.561 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.560+0000 7fbb16573700 10 monclient: build_initial_monmap 2023-09-30T02:09:51.561 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.560+0000 7fbb16573700 10 monclient: monmap: 2023-09-30T02:09:51.561 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:09:51.561 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:51.561 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:09:51.561866+0000 2023-09-30T02:09:51.561 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:09:51.561866+0000 2023-09-30T02:09:51.561 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:09:51.561 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:51.561 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:09:51.561 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:09:51.561 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:09:51.562 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:09:51.562 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:09:51.562 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:51.564 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.561+0000 7fbb16573700 1 Processor -- start 2023-09-30T02:09:51.564 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.561+0000 7fbb16573700 1 -- start start 2023-09-30T02:09:51.565 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.561+0000 7fbb16573700 10 monclient: init 2023-09-30T02:09:51.565 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.561+0000 7fbb16573700 10 monclient: _reopen_session rank -1 2023-09-30T02:09:51.565 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.561+0000 7fbb16573700 10 monclient: _add_conns ranks=[2,4,3,0,1] 2023-09-30T02:09:51.565 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.561+0000 7fbb16573700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fbb100a5b70 0x7fbb100b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:51.565 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.561+0000 7fbb16573700 10 monclient(hunting): picked mon.noname-b con 0x7fbb100a5b70 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:09:51.565 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.561+0000 7fbb16573700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fbb100a5f40 0x7fbb10141b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:51.566 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.561+0000 7fbb16573700 10 monclient(hunting): picked mon.noname-c con 0x7fbb100a5f40 addr v2:172.21.15.110:3300/0 2023-09-30T02:09:51.566 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.562+0000 7fbb16573700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fbb101420c0 0x7fbb1013e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:51.566 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.562+0000 7fbb16573700 10 monclient(hunting): picked mon.noname-a con 0x7fbb101420c0 addr v2:172.21.15.59:3300/0 2023-09-30T02:09:51.566 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.562+0000 7fbb16573700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:51.566 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.562+0000 7fbb16573700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb100a63b0 con 0x7fbb101420c0 2023-09-30T02:09:51.566 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.562+0000 7fbb16573700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:51.566 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.562+0000 7fbb16573700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb100a75a0 con 0x7fbb100a5f40 2023-09-30T02:09:51.566 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.562+0000 7fbb16573700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:51.566 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.562+0000 7fbb16573700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb100af3f0 con 0x7fbb100a5b70 2023-09-30T02:09:51.566 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.562+0000 7fbb16573700 10 monclient(hunting): _renew_subs 2023-09-30T02:09:51.566 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.562+0000 7fbb16573700 10 monclient(hunting): authenticate will time out at 2036.865031s 2023-09-30T02:09:51.567 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.562+0000 7fbb0ffff700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fbb100a5b70 0x7fbb100b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:51.567 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.562+0000 7fbb14b10700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fbb101420c0 0x7fbb1013e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:51.567 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.563+0000 7fbb0f7fe700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fbb100a5f40 0x7fbb10141b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:51.567 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.563+0000 7fbb0ffff700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fbb100a5b70 0x7fbb100b1500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:42750/0 (socket says 172.21.15.59:42750) 2023-09-30T02:09:51.567 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.563+0000 7fbb0ffff700 1 -- 172.21.15.59:0/4008313667 learned_addr learned my addr 172.21.15.59:0/4008313667 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:09:51.567 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.563+0000 7fbb0ffff700 10 monclient(hunting): get_auth_request con 0x7fbb100a5b70 auth_method 0 2023-09-30T02:09:51.567 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.563+0000 7fbb0ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:09:51.567 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.563+0000 7fbb0ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:51.567 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.563+0000 7fbb0ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:09:51.567 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.563+0000 7fbb14b10700 10 monclient(hunting): get_auth_request con 0x7fbb101420c0 auth_method 0 2023-09-30T02:09:51.567 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.563+0000 7fbb14b10700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:09:51.567 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.563+0000 7fbb14b10700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:51.567 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.563+0000 7fbb14b10700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:09:51.568 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.563+0000 7fbb0f7fe700 10 monclient(hunting): get_auth_request con 0x7fbb100a5f40 auth_method 0 2023-09-30T02:09:51.568 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.563+0000 7fbb0f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:09:51.568 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.563+0000 7fbb0f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:51.568 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.563+0000 7fbb0f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:09:51.568 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.563+0000 7fbb0ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:09:51.568 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.563+0000 7fbb0ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:09:51.568 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.563+0000 7fbb0ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:09:51.568 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.564+0000 7fbb14b10700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:09:51.568 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.564+0000 7fbb14b10700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:09:51.568 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.564+0000 7fbb14b10700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:09:51.568 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.564+0000 7fbb0f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:09:51.568 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.564+0000 7fbb0f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:09:51.569 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.564+0000 7fbb0f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:09:51.569 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.566+0000 7fbb0ffff700 10 monclient(hunting): handle_auth_done global_id 4601 payload 274 2023-09-30T02:09:51.569 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.566+0000 7fbb0ffff700 1 -- 172.21.15.59:0/4008313667 >> v2:172.21.15.110:3300/0 conn(0x7fbb100a5f40 msgr2=0x7fbb10141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:09:51.570 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.566+0000 7fbb0ffff700 1 --2- 172.21.15.59:0/4008313667 >> v2:172.21.15.110:3300/0 conn(0x7fbb100a5f40 0x7fbb10141b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:51.570 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.566+0000 7fbb0ffff700 1 -- 172.21.15.59:0/4008313667 >> v2:172.21.15.59:3300/0 conn(0x7fbb101420c0 msgr2=0x7fbb1013e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:09:51.570 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.566+0000 7fbb0ffff700 1 --2- 172.21.15.59:0/4008313667 >> v2:172.21.15.59:3300/0 conn(0x7fbb101420c0 0x7fbb1013e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:51.570 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.566+0000 7fbb0ffff700 10 monclient: _finish_hunting 0 2023-09-30T02:09:51.570 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.566+0000 7fbb0ffff700 1 monclient: found mon.noname-b 2023-09-30T02:09:51.570 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.566+0000 7fbb0ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:09:51.570 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.566+0000 7fbb0ffff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.59:3301/0 2023-09-30T02:09:51.570 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.566+0000 7fbb0ffff700 1 -- 172.21.15.59:0/4008313667 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb10142500 con 0x7fbb100a5b70 2023-09-30T02:09:51.570 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.568+0000 7fbb0ffff700 1 --2- 172.21.15.59:0/4008313667 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fbb100a5b70 0x7fbb100b1500 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fbb10069af0 tx=0x7fbb00009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad69b84eb9e23142 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:09:51.573 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.572+0000 7fbb0effd700 1 -- 172.21.15.59:0/4008313667 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb0000b640 con 0x7fbb100a5b70 2023-09-30T02:09:51.573 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.572+0000 7fbb0effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:09:51.573 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.572+0000 7fbb0effd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:09:51.573 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.572+0000 7fbb0effd700 10 monclient: dump: 2023-09-30T02:09:51.573 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:09:51.573 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:51.574 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:51.574 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:51.574 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:09:51.574 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:51.574 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:09:51.574 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:09:51.574 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:09:51.574 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:51.574 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.572+0000 7fbb0effd700 10 monclient: _finish_auth 0 2023-09-30T02:09:51.574 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.572+0000 7fbb0effd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:09:51.575 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.572+0000 7fbb0effd700 1 -- 172.21.15.59:0/4008313667 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb0000b7a0 con 0x7fbb100a5b70 2023-09-30T02:09:51.575 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.572+0000 7fbb0effd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:09:51.575 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.572+0000 7fbb0effd700 1 -- 172.21.15.59:0/4008313667 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb0000bb60 con 0x7fbb100a5b70 2023-09-30T02:09:51.575 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.572+0000 7fbb16573700 5 monclient: authenticate success, global_id 4601 2023-09-30T02:09:51.575 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.572+0000 7fbb16573700 10 monclient: get_monmap_and_config success 2023-09-30T02:09:51.575 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.572+0000 7fbb16573700 4 set_mon_vals no callback set 2023-09-30T02:09:51.575 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.572+0000 7fbb0effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:09:51.575 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.572+0000 7fbb0effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:09:51.575 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.572+0000 7fbb0effd700 10 monclient: dump: 2023-09-30T02:09:51.575 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:09:51.575 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:51.576 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:51.576 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:51.576 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:09:51.576 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:51.576 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:09:51.576 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:09:51.576 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:09:51.576 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:51.580 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.573+0000 7fbb16573700 10 monclient: shutdown 2023-09-30T02:09:51.580 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.573+0000 7fbb16573700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:09:51.580 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.573+0000 7fbb16573700 1 -- 172.21.15.59:0/4008313667 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fbb100a5b70 msgr2=0x7fbb100b1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:51.580 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.573+0000 7fbb16573700 1 --2- 172.21.15.59:0/4008313667 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fbb100a5b70 0x7fbb100b1500 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fbb10069af0 tx=0x7fbb00009250 comp rx=0 tx=0).stop 2023-09-30T02:09:51.580 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.573+0000 7fbb16573700 1 -- 172.21.15.59:0/4008313667 shutdown_connections 2023-09-30T02:09:51.581 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.573+0000 7fbb16573700 1 --2- 172.21.15.59:0/4008313667 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fbb100a5b70 0x7fbb100b1500 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:51.581 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.573+0000 7fbb16573700 1 --2- 172.21.15.59:0/4008313667 >> v2:172.21.15.59:3300/0 conn(0x7fbb101420c0 0x7fbb1013e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:51.581 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.573+0000 7fbb16573700 1 --2- 172.21.15.59:0/4008313667 >> v2:172.21.15.110:3300/0 conn(0x7fbb100a5f40 0x7fbb10141b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:51.581 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.573+0000 7fbb16573700 1 -- 172.21.15.59:0/4008313667 >> 172.21.15.59:0/4008313667 conn(0x7fbb100631f0 msgr2=0x7fbb100635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:09:51.581 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.573+0000 7fbb16573700 1 -- 172.21.15.59:0/4008313667 shutdown_connections 2023-09-30T02:09:51.581 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.573+0000 7fbb16573700 1 -- 172.21.15.59:0/4008313667 wait complete. 2023-09-30T02:09:51.582 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.574+0000 7fbb16573700 10 monclient: build_initial_monmap 2023-09-30T02:09:51.582 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.574+0000 7fbb16573700 10 monclient: monmap: 2023-09-30T02:09:51.582 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:09:51.582 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:09:51.582 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:09:51.582 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:09:51.583 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:09:51.583 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:09:51.583 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:09:51.583 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:09:51.583 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:09:51.583 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:51.583 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.574+0000 7fbb16573700 1 librados: starting msgr at 2023-09-30T02:09:51.583 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.574+0000 7fbb16573700 1 librados: starting objecter 2023-09-30T02:09:51.583 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.575+0000 7fbb16573700 1 Processor -- start 2023-09-30T02:09:51.583 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.575+0000 7fbb16573700 1 -- start start 2023-09-30T02:09:51.583 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.575+0000 7fbb16573700 1 librados: setting wanted keys 2023-09-30T02:09:51.583 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.575+0000 7fbb16573700 1 librados: calling monclient init 2023-09-30T02:09:51.584 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.575+0000 7fbb16573700 10 monclient: init 2023-09-30T02:09:51.584 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.575+0000 7fbb16573700 10 monclient: _reopen_session rank -1 2023-09-30T02:09:51.584 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.575+0000 7fbb16573700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T02:09:51.584 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.575+0000 7fbb16573700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fbb100a5b70 0x7fbb100a8120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:51.584 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.575+0000 7fbb16573700 10 monclient(hunting): picked mon.noname-c con 0x7fbb100a5b70 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:09:51.584 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.576+0000 7fbb16573700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb100a5f40 0x7fbb100a8bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:51.584 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.576+0000 7fbb16573700 10 monclient(hunting): picked mon.noname-b con 0x7fbb100a5f40 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:09:51.584 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.576+0000 7fbb16573700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbb101420c0 0x7fbb100a9110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:51.584 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.576+0000 7fbb16573700 10 monclient(hunting): picked mon.noname-a con 0x7fbb101420c0 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:09:51.584 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.576+0000 7fbb16573700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:51.584 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.576+0000 7fbb16573700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb1013bbc0 con 0x7fbb101420c0 2023-09-30T02:09:51.584 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.576+0000 7fbb16573700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:51.585 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.576+0000 7fbb16573700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb100a8660 con 0x7fbb100a5f40 2023-09-30T02:09:51.585 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.576+0000 7fbb16573700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:51.585 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.576+0000 7fbb16573700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb100a9650 con 0x7fbb100a5b70 2023-09-30T02:09:51.585 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.576+0000 7fbb16573700 10 monclient(hunting): _renew_subs 2023-09-30T02:09:51.585 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.576+0000 7fbb16573700 10 monclient(hunting): authenticate will time out at 2036.879092s 2023-09-30T02:09:51.585 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.576+0000 7fbb0ffff700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fbb100a5b70 0x7fbb100a8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:51.585 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.576+0000 7fbb14b10700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbb101420c0 0x7fbb100a9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:51.585 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.576+0000 7fbb0f7fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb100a5f40 0x7fbb100a8bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:51.585 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.576+0000 7fbb0ffff700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fbb100a5b70 0x7fbb100a8120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:42762/0 (socket says 172.21.15.59:42762) 2023-09-30T02:09:51.585 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.576+0000 7fbb0ffff700 1 -- 172.21.15.59:0/3843714068 learned_addr learned my addr 172.21.15.59:0/3843714068 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:09:51.585 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.576+0000 7fbb0ffff700 10 monclient(hunting): get_auth_request con 0x7fbb100a5b70 auth_method 0 2023-09-30T02:09:51.585 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.576+0000 7fbb0ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:09:51.586 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.576+0000 7fbb0ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:51.586 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.576+0000 7fbb0ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:09:51.586 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.577+0000 7fbb0f7fe700 10 monclient(hunting): get_auth_request con 0x7fbb100a5f40 auth_method 0 2023-09-30T02:09:51.586 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.577+0000 7fbb0f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:09:51.586 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.577+0000 7fbb0f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:51.586 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.577+0000 7fbb0f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:09:51.586 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.577+0000 7fbb14b10700 10 monclient(hunting): get_auth_request con 0x7fbb101420c0 auth_method 0 2023-09-30T02:09:51.586 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.577+0000 7fbb14b10700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:09:51.586 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.577+0000 7fbb14b10700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:51.586 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.577+0000 7fbb14b10700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:09:51.586 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.577+0000 7fbb0ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:09:51.587 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.577+0000 7fbb0ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:09:51.587 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.577+0000 7fbb0ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:09:51.587 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.577+0000 7fbb0f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:09:51.587 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.577+0000 7fbb0f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:09:51.587 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.577+0000 7fbb0f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:09:51.587 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.577+0000 7fbb14b10700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:09:51.587 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.577+0000 7fbb14b10700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:09:51.587 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.577+0000 7fbb14b10700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:09:51.589 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.581+0000 7fbb0ffff700 10 monclient(hunting): handle_auth_done global_id 4604 payload 824 2023-09-30T02:09:51.590 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.581+0000 7fbb0ffff700 1 -- 172.21.15.59:0/3843714068 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb100a5f40 msgr2=0x7fbb100a8bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:51.590 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.581+0000 7fbb0ffff700 1 --2- 172.21.15.59:0/3843714068 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb100a5f40 0x7fbb100a8bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:51.590 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.581+0000 7fbb0ffff700 1 -- 172.21.15.59:0/3843714068 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbb101420c0 msgr2=0x7fbb100a9110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:51.590 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.581+0000 7fbb0ffff700 1 --2- 172.21.15.59:0/3843714068 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbb101420c0 0x7fbb100a9110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:51.590 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.581+0000 7fbb0ffff700 10 monclient: _finish_hunting 0 2023-09-30T02:09:51.590 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.581+0000 7fbb0ffff700 1 monclient: found mon.noname-c 2023-09-30T02:09:51.590 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.581+0000 7fbb0ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:09:51.590 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.581+0000 7fbb0ffff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.59:3301/0 2023-09-30T02:09:51.590 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.581+0000 7fbb0ffff700 1 -- 172.21.15.59:0/3843714068 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb000094f0 con 0x7fbb100a5b70 2023-09-30T02:09:51.590 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.584+0000 7fbb0ffff700 1 --2- 172.21.15.59:0/3843714068 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fbb100a5b70 0x7fbb100a8120 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fbb100af530 tx=0x7fbb00017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:09:51.591 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.584+0000 7fbb07fff700 10 client.?.objecter ms_handle_connect 0x7fbb100a5b70 2023-09-30T02:09:51.591 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.584+0000 7fbb07fff700 10 client.?.objecter resend_mon_ops 2023-09-30T02:09:51.591 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.588+0000 7fbb07fff700 1 -- 172.21.15.59:0/3843714068 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb0000b910 con 0x7fbb100a5b70 2023-09-30T02:09:51.591 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.588+0000 7fbb07fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:09:51.591 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.588+0000 7fbb07fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:09:51.591 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.588+0000 7fbb07fff700 10 monclient: dump: 2023-09-30T02:09:51.591 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:09:51.591 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:51.591 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:51.591 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:51.591 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:09:51.591 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:51.592 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:09:51.592 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:09:51.592 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:09:51.592 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:51.592 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.588+0000 7fbb07fff700 10 monclient: _finish_auth 0 2023-09-30T02:09:51.592 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.588+0000 7fbb07fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:09:51.592 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.588+0000 7fbb07fff700 1 -- 172.21.15.59:0/3843714068 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb0000ba70 con 0x7fbb100a5b70 2023-09-30T02:09:51.592 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.588+0000 7fbb16573700 5 monclient: authenticate success, global_id 4604 2023-09-30T02:09:51.592 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.588+0000 7fbb16573700 10 monclient: _renew_subs 2023-09-30T02:09:51.592 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.588+0000 7fbb16573700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:09:51.592 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.588+0000 7fbb16573700 1 -- 172.21.15.59:0/3843714068 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb100aaba0 con 0x7fbb100a5b70 2023-09-30T02:09:51.592 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.588+0000 7fbb07fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:09:51.593 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.588+0000 7fbb07fff700 1 -- 172.21.15.59:0/3843714068 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb00011630 con 0x7fbb100a5b70 2023-09-30T02:09:51.593 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.588+0000 7fbb07fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:09:51.593 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.588+0000 7fbb07fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:09:51.593 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.588+0000 7fbb07fff700 10 monclient: dump: 2023-09-30T02:09:51.593 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:09:51.593 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:51.593 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:51.593 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:51.593 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:09:51.593 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:51.593 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:09:51.593 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:09:51.593 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:09:51.594 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:51.594 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.588+0000 7fbb0e7fc700 4 set_mon_vals no callback set 2023-09-30T02:09:51.594 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.588+0000 7fbb16573700 10 client.4604.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:09:51.594 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.588+0000 7fbb16573700 10 monclient: _renew_subs 2023-09-30T02:09:51.594 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.588+0000 7fbb16573700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:09:51.594 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.588+0000 7fbb16573700 1 -- 172.21.15.59:0/3843714068 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb100ae8e0 con 0x7fbb100a5b70 2023-09-30T02:09:51.594 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.588+0000 7fbb16573700 1 librados: init done 2023-09-30T02:09:51.594 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.589+0000 7fbb16573700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:09:51.594 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.589+0000 7fbb16573700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:09:51.594 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.589+0000 7fbb16573700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:09:51.594 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.589+0000 7fbb16573700 1 -- 172.21.15.59:0/3843714068 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb100a89d0 con 0x7fbb100a5b70 2023-09-30T02:09:51.817 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.816+0000 7fbb07fff700 1 -- 172.21.15.59:0/3843714068 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fbb000119a0 con 0x7fbb100a5b70 2023-09-30T02:09:51.817 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.817+0000 7fbb07fff700 1 --2- 172.21.15.59:0/3843714068 >> v2:172.21.15.110:6800/115870 conn(0x7fbaf0068960 0x7fbaf006ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:51.817 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.817+0000 7fbb0f7fe700 1 --2- 172.21.15.59:0/3843714068 >> v2:172.21.15.110:6800/115870 conn(0x7fbaf0068960 0x7fbaf006ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:51.817 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.817+0000 7fbb0f7fe700 10 monclient: get_auth_request con 0x7fbaf0068960 auth_method 0 2023-09-30T02:09:51.818 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.818+0000 7fbb0f7fe700 1 --2- 172.21.15.59:0/3843714068 >> v2:172.21.15.110:6800/115870 conn(0x7fbaf0068960 0x7fbaf006ae10 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fbb10138100 tx=0x7fbaf80091a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:09:51.818 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.818+0000 7fbb07fff700 10 client.4604.objecter ms_handle_connect 0x7fbaf0068960 2023-09-30T02:09:51.888 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.887+0000 7fbb07fff700 1 -- 172.21.15.59:0/3843714068 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fbb00056ba0 con 0x7fbb100a5b70 2023-09-30T02:09:51.888 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.887+0000 7fbb07fff700 10 client.4604.objecter ms_dispatch 0x7fbb10001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:09:51.888 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.887+0000 7fbb07fff700 3 client.4604.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:09:51.888 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.887+0000 7fbb07fff700 3 client.4604.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:09:51.888 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.887+0000 7fbb07fff700 20 client.4604.objecter dump_active .. 0 homeless 2023-09-30T02:09:51.888 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.887+0000 7fbb07fff700 1 -- 172.21.15.59:0/3843714068 <== mon.2 v2:172.21.15.59:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbb00052ee0 con 0x7fbb100a5b70 2023-09-30T02:09:51.888 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.887+0000 7fbb07fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:09:51.888 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:51.887+0000 7fbb07fff700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:09:52.170 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:52.170+0000 7fbb16573700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:09:52.170 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:52.170+0000 7fbb16573700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:09:52.171 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:52.170+0000 7fbb16573700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:09:52.171 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:52.170+0000 7fbb16573700 1 -- 172.21.15.59:0/3843714068 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fbb100ae8e0 con 0x7fbb100a5b70 2023-09-30T02:09:52.177 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:52.177+0000 7fbb07fff700 1 -- 172.21.15.59:0/3843714068 <== mon.2 v2:172.21.15.59:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fbb100ae8e0 con 0x7fbb100a5b70 2023-09-30T02:09:52.177 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:52.177+0000 7fbb07fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:09:52.177 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:52.177+0000 7fbb07fff700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:09:52.178 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:09:52.178 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:09:52.181 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:52.179+0000 7fbb16573700 10 librados: watch_flush enter 2023-09-30T02:09:52.181 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:52.179+0000 7fbb16573700 10 librados: watch_flush exit 2023-09-30T02:09:52.181 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:52.179+0000 7fbb16573700 20 client.4604.objecter shutdown clearing up homeless session... 2023-09-30T02:09:52.181 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:52.179+0000 7fbb16573700 10 client.4604.objecter successfully canceled tick 2023-09-30T02:09:52.181 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:52.180+0000 7fbb16573700 1 -- 172.21.15.59:0/3843714068 >> v2:172.21.15.110:6800/115870 conn(0x7fbaf0068960 msgr2=0x7fbaf006ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:52.181 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:52.180+0000 7fbb16573700 1 --2- 172.21.15.59:0/3843714068 >> v2:172.21.15.110:6800/115870 conn(0x7fbaf0068960 0x7fbaf006ae10 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fbb10138100 tx=0x7fbaf80091a0 comp rx=0 tx=0).stop 2023-09-30T02:09:52.181 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:52.180+0000 7fbb16573700 10 monclient: shutdown 2023-09-30T02:09:52.181 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:52.180+0000 7fbb16573700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:09:52.181 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:52.180+0000 7fbb16573700 1 -- 172.21.15.59:0/3843714068 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fbb100a5b70 msgr2=0x7fbb100a8120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:52.181 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:52.180+0000 7fbb16573700 1 --2- 172.21.15.59:0/3843714068 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fbb100a5b70 0x7fbb100a8120 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fbb100af530 tx=0x7fbb00017000 comp rx=0 tx=0).stop 2023-09-30T02:09:52.181 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:52.180+0000 7fbb16573700 1 -- 172.21.15.59:0/3843714068 shutdown_connections 2023-09-30T02:09:52.182 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:52.180+0000 7fbb16573700 1 --2- 172.21.15.59:0/3843714068 >> v2:172.21.15.110:6800/115870 conn(0x7fbaf0068960 0x7fbaf006ae10 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:52.182 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:52.180+0000 7fbb16573700 1 --2- 172.21.15.59:0/3843714068 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbb101420c0 0x7fbb100a9110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:52.182 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:52.180+0000 7fbb16573700 1 --2- 172.21.15.59:0/3843714068 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fbb100a5b70 0x7fbb100a8120 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:52.182 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:52.180+0000 7fbb16573700 1 --2- 172.21.15.59:0/3843714068 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fbb100a5f40 0x7fbb100a8bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:52.182 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:52.180+0000 7fbb16573700 1 -- 172.21.15.59:0/3843714068 >> 172.21.15.59:0/3843714068 conn(0x7fbb100631f0 msgr2=0x7fbb100af560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:09:52.182 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:52.180+0000 7fbb16573700 1 -- 172.21.15.59:0/3843714068 shutdown_connections 2023-09-30T02:09:52.182 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:52.180+0000 7fbb16573700 1 -- 172.21.15.59:0/3843714068 wait complete. 2023-09-30T02:09:52.182 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:52.180+0000 7fbb16573700 1 librados: shutdown 2023-09-30T02:09:54.077 INFO:tasks.ceph.mon.a.smithi059.stderr:==00:00:05:03.942 116256== Warning: unimplemented fcntl command: 1036 2023-09-30T02:09:54.141 INFO:tasks.ceph.mon.a.smithi059.stderr:==00:00:05:04.006 116256== Warning: unimplemented fcntl command: 1036 2023-09-30T02:09:54.177 INFO:tasks.ceph.mon.a.smithi059.stderr:==00:00:05:04.043 116256== Warning: unimplemented fcntl command: 1036 2023-09-30T02:09:54.216 INFO:tasks.ceph.mon.b.smithi110.stderr:==00:00:05:04.063 115869== Warning: unimplemented fcntl command: 1036 2023-09-30T02:09:54.218 INFO:tasks.ceph.mon.b.smithi110.stderr:==00:00:05:04.066 115869== Warning: unimplemented fcntl command: 1036 2023-09-30T02:09:54.239 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:05:04.105 116257== Warning: unimplemented fcntl command: 1036 2023-09-30T02:09:54.242 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:05:04.107 116257== Warning: unimplemented fcntl command: 1036 2023-09-30T02:09:54.245 INFO:tasks.ceph.mon.b.smithi110.stderr:==00:00:05:04.093 115869== Warning: unimplemented fcntl command: 1036 2023-09-30T02:09:54.268 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:05:04.133 116257== Warning: unimplemented fcntl command: 1036 2023-09-30T02:09:55.194 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:09:55.339 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.338+0000 7ff0a2c85700 10 monclient: get_monmap_and_config 2023-09-30T02:09:55.339 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.338+0000 7ff0a2c85700 10 monclient: build_initial_monmap 2023-09-30T02:09:55.339 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.338+0000 7ff0a2c85700 10 monclient: monmap: 2023-09-30T02:09:55.339 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:09:55.339 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:55.339 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:09:55.339838+0000 2023-09-30T02:09:55.339 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:09:55.339838+0000 2023-09-30T02:09:55.339 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:09:55.340 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:55.340 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:09:55.340 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:09:55.340 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:09:55.340 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:09:55.340 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:09:55.340 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:55.342 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.339+0000 7ff0a2c85700 1 Processor -- start 2023-09-30T02:09:55.342 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.339+0000 7ff0a2c85700 1 -- start start 2023-09-30T02:09:55.342 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.339+0000 7ff0a2c85700 10 monclient: init 2023-09-30T02:09:55.342 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.340+0000 7ff0a2c85700 10 monclient: _reopen_session rank -1 2023-09-30T02:09:55.342 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.340+0000 7ff0a2c85700 10 monclient: _add_conns ranks=[0,1,4,3,2] 2023-09-30T02:09:55.342 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.340+0000 7ff0a2c85700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff09c0a5b70 addr v1:172.21.15.59:6789/0 2023-09-30T02:09:55.342 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.340+0000 7ff0a2c85700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff09c142290 addr v1:172.21.15.110:6789/0 2023-09-30T02:09:55.342 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.340+0000 7ff0a2c85700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff09c0b1500 0x7ff09c0a5f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:55.343 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.340+0000 7ff0a2c85700 10 monclient(hunting): picked mon.noname-c con 0x7ff09c0b1500 addr v2:172.21.15.110:3300/0 2023-09-30T02:09:55.343 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.340+0000 7ff0a2c85700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff09c0b1920 con 0x7ff09c0a5b70 2023-09-30T02:09:55.343 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.340+0000 7ff0a2c85700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff09c0a75a0 con 0x7ff09c142290 2023-09-30T02:09:55.343 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.340+0000 7ff0a2c85700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:55.343 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.340+0000 7ff0a2c85700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff09c0af3f0 con 0x7ff09c0b1500 2023-09-30T02:09:55.343 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.340+0000 7ff0a2c85700 10 monclient(hunting): _renew_subs 2023-09-30T02:09:55.343 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.340+0000 7ff0a2c85700 10 monclient(hunting): authenticate will time out at 2040.643526s 2023-09-30T02:09:55.343 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.341+0000 7ff0a1222700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7ff09c0b1500 0x7ff09c0a5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:55.343 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.341+0000 7ff09bfff700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7ff09c142290 0x7ff09c13af30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.59:44226/0 (socket says 172.21.15.59:44226) 2023-09-30T02:09:55.344 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.341+0000 7ff09bfff700 1 -- 172.21.15.59:0/3237546619 learned_addr learned my addr 172.21.15.59:0/3237546619 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:09:55.344 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.341+0000 7ff0a1222700 10 monclient(hunting): get_auth_request con 0x7ff09c0b1500 auth_method 0 2023-09-30T02:09:55.344 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.341+0000 7ff0a1222700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:09:55.344 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.341+0000 7ff0a1222700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:55.344 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.341+0000 7ff0a1222700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:09:55.344 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.343+0000 7ff0a1222700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:09:55.345 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.343+0000 7ff0a1222700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:09:55.345 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.343+0000 7ff0a1222700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:09:55.346 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.345+0000 7ff0a1222700 10 monclient(hunting): handle_auth_done global_id 4705 payload 274 2023-09-30T02:09:55.346 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.345+0000 7ff0a1222700 1 -- 172.21.15.59:0/3237546619 >> v1:172.21.15.110:6789/0 conn(0x7ff09c142290 legacy=0x7ff09c13af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:55.346 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.345+0000 7ff0a1222700 1 -- 172.21.15.59:0/3237546619 >> v1:172.21.15.59:6789/0 conn(0x7ff09c0a5b70 legacy=0x7ff09c141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:09:55.346 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.345+0000 7ff0a1222700 10 monclient: _finish_hunting 0 2023-09-30T02:09:55.346 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.345+0000 7ff0a1222700 1 monclient: found mon.noname-c 2023-09-30T02:09:55.346 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.345+0000 7ff0a1222700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:09:55.346 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.345+0000 7ff0a1222700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3300/0 2023-09-30T02:09:55.346 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.345+0000 7ff0a1222700 1 -- 172.21.15.59:0/3237546619 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff09c13b640 con 0x7ff09c0b1500 2023-09-30T02:09:55.348 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.348+0000 7ff0a1222700 1 --2- 172.21.15.59:0/3237546619 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff09c0b1500 0x7ff09c0a5f40 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ff094009e60 tx=0x7ff094009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8e160936591afe6 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:09:55.360 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.358+0000 7ff09b7fe700 1 -- 172.21.15.59:0/3237546619 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff09400e070 con 0x7ff09c0b1500 2023-09-30T02:09:55.360 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.358+0000 7ff09b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:09:55.360 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.358+0000 7ff09b7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:09:55.360 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.358+0000 7ff09b7fe700 10 monclient: dump: 2023-09-30T02:09:55.360 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:09:55.360 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:55.360 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:55.360 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:55.361 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:09:55.361 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:55.361 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:09:55.361 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:09:55.361 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:09:55.361 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:55.361 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.358+0000 7ff09b7fe700 10 monclient: _finish_auth 0 2023-09-30T02:09:55.361 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.358+0000 7ff09b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:09:55.361 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.358+0000 7ff09b7fe700 1 -- 172.21.15.59:0/3237546619 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff09400bbf0 con 0x7ff09c0b1500 2023-09-30T02:09:55.361 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.358+0000 7ff0a2c85700 5 monclient: authenticate success, global_id 4705 2023-09-30T02:09:55.362 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.358+0000 7ff09b7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:09:55.362 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.358+0000 7ff09b7fe700 1 -- 172.21.15.59:0/3237546619 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff094012430 con 0x7ff09c0b1500 2023-09-30T02:09:55.362 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.358+0000 7ff0a2c85700 10 monclient: get_monmap_and_config success 2023-09-30T02:09:55.362 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.358+0000 7ff0a2c85700 4 set_mon_vals no callback set 2023-09-30T02:09:55.362 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.358+0000 7ff09b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:09:55.362 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.358+0000 7ff09b7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:09:55.362 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.358+0000 7ff09b7fe700 10 monclient: dump: 2023-09-30T02:09:55.362 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:09:55.363 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:55.363 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:55.363 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:55.363 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:09:55.363 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:55.363 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:09:55.363 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:09:55.363 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:09:55.364 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:55.364 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.359+0000 7ff0a2c85700 10 monclient: shutdown 2023-09-30T02:09:55.364 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.359+0000 7ff0a2c85700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:09:55.364 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.359+0000 7ff0a2c85700 1 -- 172.21.15.59:0/3237546619 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff09c0b1500 msgr2=0x7ff09c0a5f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:55.364 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.359+0000 7ff0a2c85700 1 --2- 172.21.15.59:0/3237546619 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff09c0b1500 0x7ff09c0a5f40 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ff094009e60 tx=0x7ff094009780 comp rx=0 tx=0).stop 2023-09-30T02:09:55.364 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.359+0000 7ff0a2c85700 1 -- 172.21.15.59:0/3237546619 shutdown_connections 2023-09-30T02:09:55.365 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.359+0000 7ff0a2c85700 1 --2- 172.21.15.59:0/3237546619 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff09c0b1500 0x7ff09c0a5f40 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:55.365 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.359+0000 7ff0a2c85700 1 -- 172.21.15.59:0/3237546619 >> 172.21.15.59:0/3237546619 conn(0x7ff09c0631f0 msgr2=0x7ff09c0635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:09:55.365 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.359+0000 7ff0a2c85700 1 -- 172.21.15.59:0/3237546619 shutdown_connections 2023-09-30T02:09:55.365 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.359+0000 7ff0a2c85700 1 -- 172.21.15.59:0/3237546619 wait complete. 2023-09-30T02:09:55.369 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.360+0000 7ff0a2c85700 10 monclient: build_initial_monmap 2023-09-30T02:09:55.369 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.360+0000 7ff0a2c85700 10 monclient: monmap: 2023-09-30T02:09:55.369 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:09:55.369 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:09:55.369 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:09:55.370 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:09:55.370 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:09:55.370 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:09:55.370 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:09:55.370 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:09:55.370 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:09:55.370 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:55.370 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.360+0000 7ff0a2c85700 1 librados: starting msgr at 2023-09-30T02:09:55.370 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.360+0000 7ff0a2c85700 1 librados: starting objecter 2023-09-30T02:09:55.371 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.360+0000 7ff0a2c85700 1 Processor -- start 2023-09-30T02:09:55.371 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.361+0000 7ff0a2c85700 1 -- start start 2023-09-30T02:09:55.371 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.361+0000 7ff0a2c85700 1 librados: setting wanted keys 2023-09-30T02:09:55.371 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.361+0000 7ff0a2c85700 1 librados: calling monclient init 2023-09-30T02:09:55.371 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.361+0000 7ff0a2c85700 10 monclient: init 2023-09-30T02:09:55.371 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.361+0000 7ff0a2c85700 10 monclient: _reopen_session rank -1 2023-09-30T02:09:55.371 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.361+0000 7ff0a2c85700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T02:09:55.372 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.361+0000 7ff0a2c85700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff09c0a5b70 0x7ff09c144f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:55.372 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.361+0000 7ff0a2c85700 10 monclient(hunting): picked mon.noname-b con 0x7ff09c0a5b70 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:09:55.372 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.361+0000 7ff0a2c85700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff09c0b1500 0x7ff09c13e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:55.372 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.361+0000 7ff0a2c85700 10 monclient(hunting): picked mon.noname-a con 0x7ff09c0b1500 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:09:55.372 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.361+0000 7ff0a2c85700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff09c142290 0x7ff09c0ae000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:55.372 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.361+0000 7ff0a2c85700 10 monclient(hunting): picked mon.noname-c con 0x7ff09c142290 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:09:55.372 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.361+0000 7ff0a2c85700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:55.372 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.361+0000 7ff0a2c85700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff09c0a8b00 con 0x7ff09c0b1500 2023-09-30T02:09:55.373 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.361+0000 7ff0a2c85700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:55.373 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.361+0000 7ff0a2c85700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff09c13d210 con 0x7ff09c0a5b70 2023-09-30T02:09:55.373 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.361+0000 7ff0a2c85700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:55.373 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.361+0000 7ff0a2c85700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff09c0a78a0 con 0x7ff09c142290 2023-09-30T02:09:55.373 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.361+0000 7ff0a2c85700 10 monclient(hunting): _renew_subs 2023-09-30T02:09:55.373 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.361+0000 7ff0a2c85700 10 monclient(hunting): authenticate will time out at 2040.664754s 2023-09-30T02:09:55.374 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.362+0000 7ff0a0a21700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff09c0a5b70 0x7ff09c144f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:55.374 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.362+0000 7ff09bfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff09c0b1500 0x7ff09c13e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:55.374 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.362+0000 7ff0a1222700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff09c142290 0x7ff09c0ae000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:55.374 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.362+0000 7ff0a0a21700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff09c0a5b70 0x7ff09c144f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:49776/0 (socket says 172.21.15.59:49776) 2023-09-30T02:09:55.374 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.362+0000 7ff0a0a21700 1 -- 172.21.15.59:0/174483757 learned_addr learned my addr 172.21.15.59:0/174483757 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:09:55.374 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.362+0000 7ff0a0a21700 10 monclient(hunting): get_auth_request con 0x7ff09c0a5b70 auth_method 0 2023-09-30T02:09:55.374 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.362+0000 7ff0a0a21700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:09:55.374 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.362+0000 7ff0a0a21700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:55.374 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.362+0000 7ff0a0a21700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:09:55.375 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.362+0000 7ff0a1222700 10 monclient(hunting): get_auth_request con 0x7ff09c142290 auth_method 0 2023-09-30T02:09:55.375 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.362+0000 7ff0a1222700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:09:55.375 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.362+0000 7ff0a1222700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:55.375 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.362+0000 7ff0a1222700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:09:55.375 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.362+0000 7ff09bfff700 10 monclient(hunting): get_auth_request con 0x7ff09c0b1500 auth_method 0 2023-09-30T02:09:55.375 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.362+0000 7ff09bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:09:55.375 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.362+0000 7ff09bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:55.375 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.362+0000 7ff09bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:09:55.376 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.363+0000 7ff0a0a21700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:09:55.376 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.363+0000 7ff0a0a21700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:09:55.376 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.363+0000 7ff0a0a21700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:09:55.376 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.363+0000 7ff09bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:09:55.376 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.363+0000 7ff09bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:09:55.376 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.363+0000 7ff09bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:09:55.376 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.363+0000 7ff0a1222700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:09:55.376 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.363+0000 7ff0a1222700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:09:55.377 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.363+0000 7ff0a1222700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:09:55.377 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.366+0000 7ff0a0a21700 10 monclient(hunting): handle_auth_done global_id 4708 payload 824 2023-09-30T02:09:55.377 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.366+0000 7ff0a0a21700 1 -- 172.21.15.59:0/174483757 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff09c142290 msgr2=0x7ff09c0ae000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:55.377 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.366+0000 7ff0a0a21700 1 --2- 172.21.15.59:0/174483757 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff09c142290 0x7ff09c0ae000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:55.377 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.366+0000 7ff0a0a21700 1 -- 172.21.15.59:0/174483757 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff09c0b1500 msgr2=0x7ff09c13e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:55.377 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.366+0000 7ff0a0a21700 1 --2- 172.21.15.59:0/174483757 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff09c0b1500 0x7ff09c13e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:55.377 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.366+0000 7ff0a0a21700 10 monclient: _finish_hunting 0 2023-09-30T02:09:55.377 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.366+0000 7ff0a0a21700 1 monclient: found mon.noname-b 2023-09-30T02:09:55.377 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.366+0000 7ff0a0a21700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:09:55.377 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.366+0000 7ff0a0a21700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:09:55.378 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.366+0000 7ff0a0a21700 1 -- 172.21.15.59:0/174483757 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff094009a20 con 0x7ff09c0a5b70 2023-09-30T02:09:55.378 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.368+0000 7ff0a0a21700 1 --2- 172.21.15.59:0/174483757 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff09c0a5b70 0x7ff09c144f70 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ff09000b600 tx=0x7ff0900091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:09:55.378 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.368+0000 7ff098ff9700 10 client.?.objecter ms_handle_connect 0x7ff09c0a5b70 2023-09-30T02:09:55.378 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.368+0000 7ff098ff9700 10 client.?.objecter resend_mon_ops 2023-09-30T02:09:55.379 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.372+0000 7ff098ff9700 1 -- 172.21.15.59:0/174483757 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff09000f510 con 0x7ff09c0a5b70 2023-09-30T02:09:55.379 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.372+0000 7ff098ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:09:55.379 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.372+0000 7ff098ff9700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:09:55.380 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.372+0000 7ff098ff9700 10 monclient: dump: 2023-09-30T02:09:55.380 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:09:55.380 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:55.380 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:55.380 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:55.380 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:09:55.380 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:55.380 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:09:55.380 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:09:55.380 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:09:55.381 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:55.381 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.372+0000 7ff098ff9700 10 monclient: _finish_auth 0 2023-09-30T02:09:55.381 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.372+0000 7ff098ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:09:55.381 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.372+0000 7ff098ff9700 1 -- 172.21.15.59:0/174483757 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff09000f690 con 0x7ff09c0a5b70 2023-09-30T02:09:55.381 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.372+0000 7ff098ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:09:55.381 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.372+0000 7ff098ff9700 1 -- 172.21.15.59:0/174483757 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff09000fa50 con 0x7ff09c0a5b70 2023-09-30T02:09:55.381 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.372+0000 7ff0a2c85700 5 monclient: authenticate success, global_id 4708 2023-09-30T02:09:55.381 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.372+0000 7ff0a2c85700 10 monclient: _renew_subs 2023-09-30T02:09:55.381 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.372+0000 7ff0a2c85700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:09:55.381 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.372+0000 7ff0a2c85700 1 -- 172.21.15.59:0/174483757 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff09c0a7ac0 con 0x7ff09c0a5b70 2023-09-30T02:09:55.381 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.372+0000 7ff09affd700 4 set_mon_vals no callback set 2023-09-30T02:09:55.382 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.372+0000 7ff098ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:09:55.382 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.372+0000 7ff098ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:09:55.382 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.372+0000 7ff098ff9700 10 monclient: dump: 2023-09-30T02:09:55.382 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:09:55.382 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:55.382 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:55.382 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:55.382 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:09:55.382 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:55.382 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:09:55.383 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:09:55.383 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:09:55.383 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:55.383 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.372+0000 7ff0a2c85700 10 client.4708.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:09:55.383 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.372+0000 7ff0a2c85700 10 monclient: _renew_subs 2023-09-30T02:09:55.383 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.372+0000 7ff0a2c85700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:09:55.383 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.372+0000 7ff0a2c85700 1 -- 172.21.15.59:0/174483757 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff09c13e070 con 0x7ff09c0a5b70 2023-09-30T02:09:55.383 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.372+0000 7ff0a2c85700 1 librados: init done 2023-09-30T02:09:55.383 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.373+0000 7ff0a2c85700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:09:55.383 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.373+0000 7ff0a2c85700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:09:55.384 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.373+0000 7ff0a2c85700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:09:55.384 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.373+0000 7ff0a2c85700 1 -- 172.21.15.59:0/174483757 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff09c064a00 con 0x7ff09c0a5b70 2023-09-30T02:09:55.633 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.631+0000 7ff098ff9700 1 -- 172.21.15.59:0/174483757 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7ff090024430 con 0x7ff09c0a5b70 2023-09-30T02:09:55.633 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.632+0000 7ff098ff9700 1 --2- 172.21.15.59:0/174483757 >> v2:172.21.15.110:6800/115870 conn(0x7ff080068890 0x7ff08006ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:55.633 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.632+0000 7ff098ff9700 1 -- 172.21.15.59:0/174483757 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7ff090084300 con 0x7ff09c0a5b70 2023-09-30T02:09:55.633 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.632+0000 7ff098ff9700 10 client.4708.objecter ms_dispatch 0x7ff09c001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:09:55.633 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.632+0000 7ff098ff9700 3 client.4708.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:09:55.633 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.632+0000 7ff098ff9700 3 client.4708.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:09:55.633 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.632+0000 7ff098ff9700 20 client.4708.objecter dump_active .. 0 homeless 2023-09-30T02:09:55.633 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.632+0000 7ff09bfff700 1 --2- 172.21.15.59:0/174483757 >> v2:172.21.15.110:6800/115870 conn(0x7ff080068890 0x7ff08006ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:55.633 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.632+0000 7ff09bfff700 10 monclient: get_auth_request con 0x7ff080068890 auth_method 0 2023-09-30T02:09:55.634 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.633+0000 7ff09bfff700 1 --2- 172.21.15.59:0/174483757 >> v2:172.21.15.110:6800/115870 conn(0x7ff080068890 0x7ff08006ad40 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ff09c1373c0 tx=0x7ff088009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:09:55.634 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.633+0000 7ff098ff9700 10 client.4708.objecter ms_handle_connect 0x7ff080068890 2023-09-30T02:09:55.653 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.653+0000 7ff098ff9700 1 -- 172.21.15.59:0/174483757 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff090051c20 con 0x7ff09c0a5b70 2023-09-30T02:09:55.653 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.653+0000 7ff098ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:09:55.653 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.653+0000 7ff098ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:09:55.915 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.914+0000 7ff0a2c85700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:09:55.915 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.914+0000 7ff0a2c85700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:09:55.915 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.914+0000 7ff0a2c85700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:09:55.915 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.914+0000 7ff0a2c85700 1 -- 172.21.15.59:0/174483757 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ff09c06a720 con 0x7ff09c0a5b70 2023-09-30T02:09:55.920 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.920+0000 7ff098ff9700 1 -- 172.21.15.59:0/174483757 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7ff090055240 con 0x7ff09c0a5b70 2023-09-30T02:09:55.920 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.920+0000 7ff098ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:09:55.920 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.920+0000 7ff098ff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:09:55.920 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:09:55.920 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:09:55.923 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.922+0000 7ff0a2c85700 10 librados: watch_flush enter 2023-09-30T02:09:55.923 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.922+0000 7ff0a2c85700 10 librados: watch_flush exit 2023-09-30T02:09:55.923 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.922+0000 7ff0a2c85700 20 client.4708.objecter shutdown clearing up homeless session... 2023-09-30T02:09:55.923 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.922+0000 7ff0a2c85700 10 client.4708.objecter successfully canceled tick 2023-09-30T02:09:55.923 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.922+0000 7ff0a2c85700 1 -- 172.21.15.59:0/174483757 >> v2:172.21.15.110:6800/115870 conn(0x7ff080068890 msgr2=0x7ff08006ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:55.923 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.922+0000 7ff0a2c85700 1 --2- 172.21.15.59:0/174483757 >> v2:172.21.15.110:6800/115870 conn(0x7ff080068890 0x7ff08006ad40 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7ff09c1373c0 tx=0x7ff088009040 comp rx=0 tx=0).stop 2023-09-30T02:09:55.923 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.922+0000 7ff0a2c85700 10 monclient: shutdown 2023-09-30T02:09:55.924 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.922+0000 7ff0a2c85700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:09:55.924 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.922+0000 7ff0a2c85700 1 -- 172.21.15.59:0/174483757 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff09c0a5b70 msgr2=0x7ff09c144f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:55.924 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.922+0000 7ff0a2c85700 1 --2- 172.21.15.59:0/174483757 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff09c0a5b70 0x7ff09c144f70 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ff09000b600 tx=0x7ff0900091a0 comp rx=0 tx=0).stop 2023-09-30T02:09:55.924 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.922+0000 7ff0a2c85700 1 -- 172.21.15.59:0/174483757 shutdown_connections 2023-09-30T02:09:55.924 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.922+0000 7ff0a2c85700 1 --2- 172.21.15.59:0/174483757 >> v2:172.21.15.110:6800/115870 conn(0x7ff080068890 0x7ff08006ad40 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:55.924 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.922+0000 7ff0a2c85700 1 --2- 172.21.15.59:0/174483757 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff09c0a5b70 0x7ff09c144f70 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:55.924 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.922+0000 7ff0a2c85700 1 --2- 172.21.15.59:0/174483757 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff09c142290 0x7ff09c0ae000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:55.924 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.922+0000 7ff0a2c85700 1 --2- 172.21.15.59:0/174483757 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff09c0b1500 0x7ff09c13e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:55.924 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.922+0000 7ff0a2c85700 1 -- 172.21.15.59:0/174483757 >> 172.21.15.59:0/174483757 conn(0x7ff09c0631f0 msgr2=0x7ff09c070f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:09:55.924 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.922+0000 7ff0a2c85700 1 -- 172.21.15.59:0/174483757 shutdown_connections 2023-09-30T02:09:55.924 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.922+0000 7ff0a2c85700 1 -- 172.21.15.59:0/174483757 wait complete. 2023-09-30T02:09:55.924 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:55.922+0000 7ff0a2c85700 1 librados: shutdown 2023-09-30T02:09:58.938 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:09:59.077 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.076+0000 7f324a362700 10 monclient: get_monmap_and_config 2023-09-30T02:09:59.077 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.076+0000 7f324a362700 10 monclient: build_initial_monmap 2023-09-30T02:09:59.077 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.076+0000 7f324a362700 10 monclient: monmap: 2023-09-30T02:09:59.077 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:09:59.078 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:59.078 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:09:59.077905+0000 2023-09-30T02:09:59.078 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:09:59.077905+0000 2023-09-30T02:09:59.078 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:09:59.078 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:59.078 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:09:59.078 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:09:59.079 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:09:59.079 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:09:59.079 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:09:59.079 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:59.081 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.077+0000 7f324a362700 1 Processor -- start 2023-09-30T02:09:59.082 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.077+0000 7f324a362700 1 -- start start 2023-09-30T02:09:59.082 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.077+0000 7f324a362700 10 monclient: init 2023-09-30T02:09:59.082 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.078+0000 7f324a362700 10 monclient: _reopen_session rank -1 2023-09-30T02:09:59.082 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.078+0000 7f324a362700 10 monclient: _add_conns ranks=[3,2,1,0,4] 2023-09-30T02:09:59.082 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.078+0000 7f324a362700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f32440a5850 0x7f32440a5c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:59.082 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.078+0000 7f324a362700 10 monclient(hunting): picked mon.noname-a con 0x7f32440a5850 addr v2:172.21.15.59:3300/0 2023-09-30T02:09:59.082 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.078+0000 7f324a362700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f324413e180 0x7f324413e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:59.082 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.078+0000 7f324a362700 10 monclient(hunting): picked mon.noname-b con 0x7f324413e180 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:09:59.082 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.078+0000 7f324a362700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f324413eb10 addr v1:172.21.15.110:6789/0 2023-09-30T02:09:59.082 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.078+0000 7f324a362700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32440a8010 con 0x7f324413eb10 2023-09-30T02:09:59.083 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.078+0000 7f324a362700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:59.083 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.078+0000 7f324a362700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32440ab7a0 con 0x7f32440a5850 2023-09-30T02:09:59.083 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.078+0000 7f324a362700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:59.083 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.078+0000 7f324a362700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32440ab8e0 con 0x7f324413e180 2023-09-30T02:09:59.083 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.078+0000 7f324a362700 10 monclient(hunting): _renew_subs 2023-09-30T02:09:59.083 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.078+0000 7f324a362700 10 monclient(hunting): authenticate will time out at 2044.381126s 2023-09-30T02:09:59.083 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.078+0000 7f32437fe700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f324413e180 0x7f324413e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:59.083 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.078+0000 7f32488ff700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f324413eb10 0x7f3244141710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.59:59680/0 (socket says 172.21.15.59:59680) 2023-09-30T02:09:59.083 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.078+0000 7f32488ff700 1 -- 172.21.15.59:0/1260564441 learned_addr learned my addr 172.21.15.59:0/1260564441 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:09:59.083 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.079+0000 7f32437fe700 10 monclient(hunting): get_auth_request con 0x7f324413e180 auth_method 0 2023-09-30T02:09:59.084 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.079+0000 7f32437fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:09:59.084 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.079+0000 7f32437fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:59.084 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.079+0000 7f32437fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:09:59.084 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.080+0000 7f32437fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:09:59.084 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.080+0000 7f32437fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:09:59.084 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.080+0000 7f32437fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:09:59.085 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.082+0000 7f3242ffd700 1 -- 172.21.15.59:0/1260564441 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3386108054 0 0) 0x7f32440a8010 con 0x7f324413eb10 2023-09-30T02:09:59.085 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.082+0000 7f3242ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:59.085 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.082+0000 7f3242ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:09:59.085 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.082+0000 7f3242ffd700 10 monclient(hunting): my global_id is 4711 2023-09-30T02:09:59.085 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.082+0000 7f3242ffd700 1 -- 172.21.15.59:0/1260564441 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f322c003570 con 0x7f324413eb10 2023-09-30T02:09:59.085 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.083+0000 7f32437fe700 10 monclient(hunting): handle_auth_done global_id 4610 payload 274 2023-09-30T02:09:59.086 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.083+0000 7f32437fe700 1 -- 172.21.15.59:0/1260564441 >> v2:172.21.15.59:3300/0 conn(0x7f32440a5850 msgr2=0x7f32440a5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:09:59.086 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.083+0000 7f32437fe700 1 --2- 172.21.15.59:0/1260564441 >> v2:172.21.15.59:3300/0 conn(0x7f32440a5850 0x7f32440a5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:59.086 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.083+0000 7f32437fe700 1 -- 172.21.15.59:0/1260564441 >> v1:172.21.15.110:6789/0 conn(0x7f324413eb10 legacy=0x7f3244141710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:59.086 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.083+0000 7f32437fe700 10 monclient: _finish_hunting 0 2023-09-30T02:09:59.086 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.083+0000 7f32437fe700 1 monclient: found mon.noname-b 2023-09-30T02:09:59.086 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.083+0000 7f32437fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:09:59.086 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.083+0000 7f32437fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.59:3301/0 2023-09-30T02:09:59.087 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.083+0000 7f32437fe700 1 -- 172.21.15.59:0/1260564441 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32440a7bd0 con 0x7f324413e180 2023-09-30T02:09:59.087 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.086+0000 7f32437fe700 1 --2- 172.21.15.59:0/1260564441 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f324413e180 0x7f324413e5d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f32380098e0 tx=0x7f32380091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=975429694f875bc4 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:09:59.093 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.091+0000 7f3242ffd700 1 -- 172.21.15.59:0/1260564441 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f323800b640 con 0x7f324413e180 2023-09-30T02:09:59.093 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.091+0000 7f3242ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:09:59.093 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.091+0000 7f3242ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:09:59.093 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.091+0000 7f3242ffd700 10 monclient: dump: 2023-09-30T02:09:59.094 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:09:59.094 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:59.094 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:59.094 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:59.094 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:09:59.094 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:59.094 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:09:59.094 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:09:59.095 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:09:59.095 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:59.095 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.091+0000 7f3242ffd700 10 monclient: _finish_auth 0 2023-09-30T02:09:59.095 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.091+0000 7f3242ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:09:59.095 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.091+0000 7f324a362700 5 monclient: authenticate success, global_id 4610 2023-09-30T02:09:59.095 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.091+0000 7f3242ffd700 1 -- 172.21.15.59:0/1260564441 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f323800b7a0 con 0x7f324413e180 2023-09-30T02:09:59.095 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.091+0000 7f324a362700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:09:59.095 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.091+0000 7f3242ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:09:59.095 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.091+0000 7f324a362700 10 monclient: get_monmap_and_config success 2023-09-30T02:09:59.095 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.091+0000 7f324a362700 4 set_mon_vals no callback set 2023-09-30T02:09:59.096 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.091+0000 7f3242ffd700 1 -- 172.21.15.59:0/1260564441 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f323800bb40 con 0x7f324413e180 2023-09-30T02:09:59.096 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.091+0000 7f3242ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:09:59.096 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.091+0000 7f3242ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:09:59.096 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.091+0000 7f3242ffd700 10 monclient: dump: 2023-09-30T02:09:59.096 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:09:59.096 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:59.096 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:59.096 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:59.096 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:09:59.097 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:59.097 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:09:59.097 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:09:59.097 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:09:59.097 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:59.097 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.092+0000 7f324a362700 10 monclient: shutdown 2023-09-30T02:09:59.097 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.092+0000 7f324a362700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:09:59.097 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.092+0000 7f324a362700 1 -- 172.21.15.59:0/1260564441 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f324413e180 msgr2=0x7f324413e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:59.097 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.092+0000 7f324a362700 1 --2- 172.21.15.59:0/1260564441 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f324413e180 0x7f324413e5d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f32380098e0 tx=0x7f32380091a0 comp rx=0 tx=0).stop 2023-09-30T02:09:59.098 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.092+0000 7f324a362700 1 -- 172.21.15.59:0/1260564441 shutdown_connections 2023-09-30T02:09:59.098 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.092+0000 7f324a362700 1 --2- 172.21.15.59:0/1260564441 >> v2:172.21.15.59:3300/0 conn(0x7f32440a5850 0x7f32440a5c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:59.098 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.092+0000 7f324a362700 1 --2- 172.21.15.59:0/1260564441 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f324413e180 0x7f324413e5d0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:59.098 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.092+0000 7f324a362700 1 -- 172.21.15.59:0/1260564441 >> 172.21.15.59:0/1260564441 conn(0x7f324406f130 msgr2=0x7f32440631f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:09:59.098 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.092+0000 7f324a362700 1 -- 172.21.15.59:0/1260564441 shutdown_connections 2023-09-30T02:09:59.098 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.092+0000 7f324a362700 1 -- 172.21.15.59:0/1260564441 wait complete. 2023-09-30T02:09:59.098 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.093+0000 7f324a362700 10 monclient: build_initial_monmap 2023-09-30T02:09:59.098 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.093+0000 7f324a362700 10 monclient: monmap: 2023-09-30T02:09:59.098 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:09:59.098 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:09:59.099 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:09:59.099 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:09:59.099 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:09:59.099 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:09:59.099 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:09:59.099 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:09:59.099 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:09:59.099 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:59.106 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.093+0000 7f324a362700 1 librados: starting msgr at 2023-09-30T02:09:59.106 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.093+0000 7f324a362700 1 librados: starting objecter 2023-09-30T02:09:59.106 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.093+0000 7f324a362700 1 Processor -- start 2023-09-30T02:09:59.106 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.094+0000 7f324a362700 1 -- start start 2023-09-30T02:09:59.107 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.094+0000 7f324a362700 1 librados: setting wanted keys 2023-09-30T02:09:59.107 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.094+0000 7f324a362700 1 librados: calling monclient init 2023-09-30T02:09:59.107 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.094+0000 7f324a362700 10 monclient: init 2023-09-30T02:09:59.107 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.094+0000 7f324a362700 10 monclient: _reopen_session rank -1 2023-09-30T02:09:59.107 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.094+0000 7f324a362700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T02:09:59.107 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.094+0000 7f324a362700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f32440a5850 0x7f3244140590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:59.108 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.094+0000 7f324a362700 10 monclient(hunting): picked mon.noname-c con 0x7f32440a5850 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:09:59.108 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.094+0000 7f324a362700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f324413e180 0x7f3244140ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:59.108 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.094+0000 7f324a362700 10 monclient(hunting): picked mon.noname-a con 0x7f324413e180 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:09:59.108 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.094+0000 7f324a362700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f324413eb10 0x7f324414e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:59.109 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.094+0000 7f324a362700 10 monclient(hunting): picked mon.noname-b con 0x7f324413eb10 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:09:59.109 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.094+0000 7f324a362700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:59.109 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.094+0000 7f324a362700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3244141160 con 0x7f324413e180 2023-09-30T02:09:59.109 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.094+0000 7f324a362700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:59.109 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.094+0000 7f324a362700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3244138bb0 con 0x7f324413eb10 2023-09-30T02:09:59.110 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.094+0000 7f324a362700 10 monclient(hunting): start opening mon connection 2023-09-30T02:09:59.110 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.094+0000 7f324a362700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f324413aaa0 con 0x7f32440a5850 2023-09-30T02:09:59.110 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.094+0000 7f324a362700 10 monclient(hunting): _renew_subs 2023-09-30T02:09:59.110 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.094+0000 7f324a362700 10 monclient(hunting): authenticate will time out at 2044.397776s 2023-09-30T02:09:59.110 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.095+0000 7f3243fff700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f32440a5850 0x7f3244140590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:59.111 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.095+0000 7f32437fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f324413e180 0x7f3244140ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:59.111 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.096+0000 7f32437fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f324413e180 0x7f3244140ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:43020/0 (socket says 172.21.15.59:43020) 2023-09-30T02:09:59.111 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.096+0000 7f32437fe700 1 -- 172.21.15.59:0/2267259605 learned_addr learned my addr 172.21.15.59:0/2267259605 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:09:59.111 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.096+0000 7f32437fe700 10 monclient(hunting): get_auth_request con 0x7f324413e180 auth_method 0 2023-09-30T02:09:59.111 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.096+0000 7f32437fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:09:59.111 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.096+0000 7f32437fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:59.112 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.096+0000 7f32437fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:09:59.112 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.097+0000 7f32437fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:09:59.112 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.097+0000 7f32437fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:09:59.112 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.097+0000 7f32437fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:09:59.113 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.097+0000 7f32488ff700 1 --2- 172.21.15.59:0/2267259605 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f324413eb10 0x7f324414e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:59.113 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.097+0000 7f3243fff700 10 monclient(hunting): get_auth_request con 0x7f32440a5850 auth_method 0 2023-09-30T02:09:59.113 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.097+0000 7f3243fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:09:59.113 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.097+0000 7f3243fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:59.113 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.097+0000 7f3243fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:09:59.113 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.097+0000 7f32488ff700 10 monclient(hunting): get_auth_request con 0x7f324413eb10 auth_method 0 2023-09-30T02:09:59.114 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.097+0000 7f32488ff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:09:59.114 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.097+0000 7f32488ff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:09:59.114 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.097+0000 7f32488ff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:09:59.114 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.098+0000 7f32488ff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:09:59.114 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.098+0000 7f32488ff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:09:59.114 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.098+0000 7f32488ff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:09:59.115 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.099+0000 7f3243fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:09:59.115 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.099+0000 7f3243fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:09:59.115 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.099+0000 7f3243fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:09:59.115 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.101+0000 7f32437fe700 10 monclient(hunting): handle_auth_done global_id 4680 payload 824 2023-09-30T02:09:59.116 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.101+0000 7f32437fe700 1 -- 172.21.15.59:0/2267259605 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f32440a5850 msgr2=0x7f3244140590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:59.116 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.101+0000 7f32437fe700 1 --2- 172.21.15.59:0/2267259605 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f32440a5850 0x7f3244140590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:59.116 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.101+0000 7f32437fe700 1 -- 172.21.15.59:0/2267259605 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f324413eb10 msgr2=0x7f324414e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:59.116 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.101+0000 7f32437fe700 1 --2- 172.21.15.59:0/2267259605 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f324413eb10 0x7f324414e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:59.116 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.101+0000 7f32437fe700 10 monclient: _finish_hunting 0 2023-09-30T02:09:59.116 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.101+0000 7f32437fe700 1 monclient: found mon.noname-a 2023-09-30T02:09:59.116 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.101+0000 7f32437fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:09:59.117 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.101+0000 7f32437fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.59:3300/0 2023-09-30T02:09:59.117 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.101+0000 7f32437fe700 1 -- 172.21.15.59:0/2267259605 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3238009420 con 0x7f324413e180 2023-09-30T02:09:59.119 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.107+0000 7f32437fe700 1 --2- 172.21.15.59:0/2267259605 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f324413e180 0x7f3244140ad0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f32380093f0 tx=0x7f3238017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:09:59.119 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.107+0000 7f322bfff700 10 client.?.objecter ms_handle_connect 0x7f324413e180 2023-09-30T02:09:59.120 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.107+0000 7f322bfff700 10 client.?.objecter resend_mon_ops 2023-09-30T02:09:59.120 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.114+0000 7f322bfff700 1 -- 172.21.15.59:0/2267259605 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f323801bc00 con 0x7f324413e180 2023-09-30T02:09:59.120 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.114+0000 7f322bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:09:59.120 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.114+0000 7f322bfff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T02:09:59.120 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.114+0000 7f322bfff700 10 monclient: dump: 2023-09-30T02:09:59.120 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:09:59.120 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:59.120 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:59.120 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:59.121 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:09:59.121 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:59.121 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:09:59.121 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:09:59.121 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:09:59.121 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:59.121 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.114+0000 7f322bfff700 10 monclient: _finish_auth 0 2023-09-30T02:09:59.122 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.114+0000 7f322bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:09:59.122 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.114+0000 7f322bfff700 1 -- 172.21.15.59:0/2267259605 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f323801bd80 con 0x7f324413e180 2023-09-30T02:09:59.122 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.114+0000 7f324a362700 5 monclient: authenticate success, global_id 4680 2023-09-30T02:09:59.122 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.114+0000 7f324a362700 10 monclient: _renew_subs 2023-09-30T02:09:59.122 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.114+0000 7f324a362700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:09:59.122 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.114+0000 7f324a362700 1 -- 172.21.15.59:0/2267259605 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f324413fab0 con 0x7f324413e180 2023-09-30T02:09:59.122 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.114+0000 7f322bfff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:09:59.123 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.114+0000 7f322bfff700 1 -- 172.21.15.59:0/2267259605 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3238011670 con 0x7f324413e180 2023-09-30T02:09:59.123 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.114+0000 7f322bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:09:59.123 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.114+0000 7f322bfff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T02:09:59.123 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.114+0000 7f322bfff700 10 monclient: dump: 2023-09-30T02:09:59.123 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:09:59.123 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:09:59.123 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:59.123 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:09:59.123 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:09:59.123 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:09:59.123 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:09:59.124 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:09:59.124 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:09:59.124 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:09:59.124 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.114+0000 7f324a362700 10 client.4680.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:09:59.124 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.114+0000 7f32427fc700 4 set_mon_vals no callback set 2023-09-30T02:09:59.124 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.114+0000 7f324a362700 10 monclient: _renew_subs 2023-09-30T02:09:59.124 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.114+0000 7f324a362700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:09:59.124 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.114+0000 7f324a362700 1 -- 172.21.15.59:0/2267259605 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f324414eca0 con 0x7f324413e180 2023-09-30T02:09:59.124 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.114+0000 7f324a362700 1 librados: init done 2023-09-30T02:09:59.124 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.115+0000 7f324a362700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:09:59.125 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.115+0000 7f324a362700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:09:59.125 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.115+0000 7f324a362700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:09:59.125 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.115+0000 7f324a362700 1 -- 172.21.15.59:0/2267259605 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32440a6c20 con 0x7f324413e180 2023-09-30T02:09:59.198 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.197+0000 7f322bfff700 1 -- 172.21.15.59:0/2267259605 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f3238011aa0 con 0x7f324413e180 2023-09-30T02:09:59.198 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.197+0000 7f322bfff700 1 --2- 172.21.15.59:0/2267259605 >> v2:172.21.15.110:6800/115870 conn(0x7f3230068960 0x7f323006ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:09:59.199 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.198+0000 7f3243fff700 1 --2- 172.21.15.59:0/2267259605 >> v2:172.21.15.110:6800/115870 conn(0x7f3230068960 0x7f323006ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:09:59.199 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.198+0000 7f3243fff700 10 monclient: get_auth_request con 0x7f3230068960 auth_method 0 2023-09-30T02:09:59.200 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.198+0000 7f3243fff700 1 --2- 172.21.15.59:0/2267259605 >> v2:172.21.15.110:6800/115870 conn(0x7f3230068960 0x7f323006ae10 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f3234009bf0 tx=0x7f32340091a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:09:59.200 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.198+0000 7f322bfff700 10 client.4680.objecter ms_handle_connect 0x7f3230068960 2023-09-30T02:09:59.474 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.473+0000 7f322bfff700 1 -- 172.21.15.59:0/2267259605 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f3238057b80 con 0x7f324413e180 2023-09-30T02:09:59.474 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.473+0000 7f322bfff700 10 client.4680.objecter ms_dispatch 0x7f3244001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:09:59.474 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.473+0000 7f322bfff700 3 client.4680.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:09:59.474 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.473+0000 7f322bfff700 3 client.4680.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:09:59.475 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.473+0000 7f322bfff700 20 client.4680.objecter dump_active .. 0 homeless 2023-09-30T02:09:59.475 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.473+0000 7f322bfff700 1 -- 172.21.15.59:0/2267259605 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f32380537c0 con 0x7f324413e180 2023-09-30T02:09:59.475 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.473+0000 7f322bfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:09:59.475 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.473+0000 7f322bfff700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:09:59.736 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.735+0000 7f324a362700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:09:59.736 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.735+0000 7f324a362700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:09:59.736 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.735+0000 7f324a362700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:09:59.736 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.735+0000 7f324a362700 1 -- 172.21.15.59:0/2267259605 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f324413ef70 con 0x7f324413e180 2023-09-30T02:09:59.742 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.741+0000 7f322bfff700 1 -- 172.21.15.59:0/2267259605 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f3238033100 con 0x7f324413e180 2023-09-30T02:09:59.742 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.741+0000 7f322bfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:09:59.742 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.741+0000 7f322bfff700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:09:59.742 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:09:59.742 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:09:59.746 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.744+0000 7f324a362700 10 librados: watch_flush enter 2023-09-30T02:09:59.746 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.744+0000 7f324a362700 10 librados: watch_flush exit 2023-09-30T02:09:59.746 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.744+0000 7f324a362700 20 client.4680.objecter shutdown clearing up homeless session... 2023-09-30T02:09:59.746 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.744+0000 7f324a362700 10 client.4680.objecter successfully canceled tick 2023-09-30T02:09:59.746 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.744+0000 7f324a362700 1 -- 172.21.15.59:0/2267259605 >> v2:172.21.15.110:6800/115870 conn(0x7f3230068960 msgr2=0x7f323006ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:59.747 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.744+0000 7f324a362700 1 --2- 172.21.15.59:0/2267259605 >> v2:172.21.15.110:6800/115870 conn(0x7f3230068960 0x7f323006ae10 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f3234009bf0 tx=0x7f32340091a0 comp rx=0 tx=0).stop 2023-09-30T02:09:59.747 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.744+0000 7f324a362700 10 monclient: shutdown 2023-09-30T02:09:59.747 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.744+0000 7f324a362700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:09:59.747 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.744+0000 7f324a362700 1 -- 172.21.15.59:0/2267259605 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f324413e180 msgr2=0x7f3244140ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:09:59.747 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.744+0000 7f324a362700 1 --2- 172.21.15.59:0/2267259605 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f324413e180 0x7f3244140ad0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f32380093f0 tx=0x7f3238017000 comp rx=0 tx=0).stop 2023-09-30T02:09:59.747 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.744+0000 7f324a362700 1 -- 172.21.15.59:0/2267259605 shutdown_connections 2023-09-30T02:09:59.748 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.744+0000 7f324a362700 1 --2- 172.21.15.59:0/2267259605 >> v2:172.21.15.110:6800/115870 conn(0x7f3230068960 0x7f323006ae10 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:59.748 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.744+0000 7f324a362700 1 --2- 172.21.15.59:0/2267259605 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f324413eb10 0x7f324414e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:59.748 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.744+0000 7f324a362700 1 --2- 172.21.15.59:0/2267259605 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f32440a5850 0x7f3244140590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:59.748 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.744+0000 7f324a362700 1 --2- 172.21.15.59:0/2267259605 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f324413e180 0x7f3244140ad0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:09:59.748 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.744+0000 7f324a362700 1 -- 172.21.15.59:0/2267259605 >> 172.21.15.59:0/2267259605 conn(0x7f324406f130 msgr2=0x7f32440b5100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:09:59.748 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.744+0000 7f324a362700 1 -- 172.21.15.59:0/2267259605 shutdown_connections 2023-09-30T02:09:59.749 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.744+0000 7f324a362700 1 -- 172.21.15.59:0/2267259605 wait complete. 2023-09-30T02:09:59.749 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:09:59.745+0000 7f324a362700 1 librados: shutdown 2023-09-30T02:10:00.006 INFO:tasks.ceph.mon.a.smithi059.stderr:2023-09-30T02:10:00.004+0000 11417700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale' 2023-09-30T02:10:02.760 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:10:02.899 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.898+0000 7f01feaa3700 10 monclient: get_monmap_and_config 2023-09-30T02:10:02.900 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.898+0000 7f01feaa3700 10 monclient: build_initial_monmap 2023-09-30T02:10:02.900 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.898+0000 7f01feaa3700 10 monclient: monmap: 2023-09-30T02:10:02.900 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:10:02.900 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:02.900 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:10:02.900236+0000 2023-09-30T02:10:02.900 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:10:02.900236+0000 2023-09-30T02:10:02.901 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:10:02.901 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:02.901 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:10:02.901 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:10:02.901 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:10:02.901 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:10:02.901 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:10:02.902 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:02.902 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.899+0000 7f01feaa3700 1 Processor -- start 2023-09-30T02:10:02.902 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.899+0000 7f01feaa3700 1 -- start start 2023-09-30T02:10:02.902 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.899+0000 7f01feaa3700 10 monclient: init 2023-09-30T02:10:02.905 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.900+0000 7f01feaa3700 10 monclient: _reopen_session rank -1 2023-09-30T02:10:02.906 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.900+0000 7f01feaa3700 10 monclient: _add_conns ranks=[2,3,4,1,0] 2023-09-30T02:10:02.906 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.900+0000 7f01feaa3700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f01f806b810 0x7f01f806e000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:02.906 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.900+0000 7f01feaa3700 10 monclient(hunting): picked mon.noname-b con 0x7f01f806b810 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:10:02.906 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.900+0000 7f01feaa3700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f01f806e540 0x7f01f813a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:02.906 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.900+0000 7f01feaa3700 10 monclient(hunting): picked mon.noname-a con 0x7f01f806e540 addr v2:172.21.15.59:3300/0 2023-09-30T02:10:02.906 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.900+0000 7f01feaa3700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f01f813ad70 0x7f01f813b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:02.907 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.900+0000 7f01feaa3700 10 monclient(hunting): picked mon.noname-c con 0x7f01f813ad70 addr v2:172.21.15.110:3300/0 2023-09-30T02:10:02.907 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.900+0000 7f01feaa3700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:02.907 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.900+0000 7f01feaa3700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01f806bbe0 con 0x7f01f806e540 2023-09-30T02:10:02.907 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.900+0000 7f01feaa3700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:02.907 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.900+0000 7f01feaa3700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01f806c4e0 con 0x7f01f813ad70 2023-09-30T02:10:02.908 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.900+0000 7f01feaa3700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:02.908 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.900+0000 7f01feaa3700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01f806d900 con 0x7f01f806b810 2023-09-30T02:10:02.908 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.900+0000 7f01feaa3700 10 monclient(hunting): _renew_subs 2023-09-30T02:10:02.908 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.900+0000 7f01feaa3700 10 monclient(hunting): authenticate will time out at 2048.203533s 2023-09-30T02:10:02.908 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.901+0000 7f01fd040700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f01f813ad70 0x7f01f813b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:02.908 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.901+0000 7f01f7fff700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f01f806e540 0x7f01f813a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:02.908 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.901+0000 7f01fc83f700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f01f806b810 0x7f01f806e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:02.909 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.901+0000 7f01fd040700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f01f813ad70 0x7f01f813b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:40540/0 (socket says 172.21.15.59:40540) 2023-09-30T02:10:02.909 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.901+0000 7f01fd040700 1 -- 172.21.15.59:0/3273944783 learned_addr learned my addr 172.21.15.59:0/3273944783 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:10:02.909 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.901+0000 7f01fd040700 10 monclient(hunting): get_auth_request con 0x7f01f813ad70 auth_method 0 2023-09-30T02:10:02.909 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.901+0000 7f01fd040700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:10:02.909 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.901+0000 7f01fd040700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:02.909 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.901+0000 7f01fd040700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:02.909 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.901+0000 7f01f7fff700 10 monclient(hunting): get_auth_request con 0x7f01f806e540 auth_method 0 2023-09-30T02:10:02.909 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.901+0000 7f01f7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:10:02.909 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.901+0000 7f01f7fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:02.909 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.901+0000 7f01f7fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:02.909 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.901+0000 7f01fc83f700 10 monclient(hunting): get_auth_request con 0x7f01f806b810 auth_method 0 2023-09-30T02:10:02.910 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.901+0000 7f01fc83f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:10:02.910 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.901+0000 7f01fc83f700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:02.910 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.901+0000 7f01fc83f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:02.910 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.902+0000 7f01fd040700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:10:02.910 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.902+0000 7f01fd040700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:10:02.910 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.902+0000 7f01fd040700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:10:02.910 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.902+0000 7f01f7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:10:02.910 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.902+0000 7f01f7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:10:02.911 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.902+0000 7f01f7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:10:02.911 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.902+0000 7f01fc83f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:10:02.911 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.902+0000 7f01fc83f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:10:02.911 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.902+0000 7f01fc83f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:10:02.911 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.904+0000 7f01fd040700 10 monclient(hunting): handle_auth_done global_id 4717 payload 274 2023-09-30T02:10:02.911 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.904+0000 7f01fd040700 1 -- 172.21.15.59:0/3273944783 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f01f806b810 msgr2=0x7f01f806e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:10:02.911 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.904+0000 7f01fd040700 1 --2- 172.21.15.59:0/3273944783 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f01f806b810 0x7f01f806e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:02.911 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.904+0000 7f01fd040700 1 -- 172.21.15.59:0/3273944783 >> v2:172.21.15.59:3300/0 conn(0x7f01f806e540 msgr2=0x7f01f813a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:10:02.911 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.904+0000 7f01fd040700 1 --2- 172.21.15.59:0/3273944783 >> v2:172.21.15.59:3300/0 conn(0x7f01f806e540 0x7f01f813a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:02.911 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.904+0000 7f01fd040700 10 monclient: _finish_hunting 0 2023-09-30T02:10:02.912 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.904+0000 7f01fd040700 1 monclient: found mon.noname-c 2023-09-30T02:10:02.912 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.904+0000 7f01fd040700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:10:02.912 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.904+0000 7f01fd040700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3300/0 2023-09-30T02:10:02.912 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.904+0000 7f01fd040700 1 -- 172.21.15.59:0/3273944783 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01f806dbe0 con 0x7f01f813ad70 2023-09-30T02:10:02.914 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.907+0000 7f01fd040700 1 --2- 172.21.15.59:0/3273944783 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01f813ad70 0x7f01f813b1f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f01f0009f20 tx=0x7f01f0009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a5cd96442177d4 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:10:02.914 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.911+0000 7f01f77fe700 1 -- 172.21.15.59:0/3273944783 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01f000e070 con 0x7f01f813ad70 2023-09-30T02:10:02.914 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.911+0000 7f01f77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:10:02.914 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.911+0000 7f01f77fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:10:02.914 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.911+0000 7f01f77fe700 10 monclient: dump: 2023-09-30T02:10:02.915 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:10:02.915 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:02.915 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:02.915 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:02.915 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:10:02.916 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:02.916 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:10:02.916 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:10:02.916 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:10:02.916 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:02.916 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.911+0000 7f01f77fe700 10 monclient: _finish_auth 0 2023-09-30T02:10:02.917 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.911+0000 7f01f77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:10:02.917 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.911+0000 7f01f77fe700 1 -- 172.21.15.59:0/3273944783 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01f000b930 con 0x7f01f813ad70 2023-09-30T02:10:02.917 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.911+0000 7f01feaa3700 5 monclient: authenticate success, global_id 4717 2023-09-30T02:10:02.917 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.911+0000 7f01feaa3700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:10:02.917 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.911+0000 7f01f77fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:10:02.917 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.911+0000 7f01feaa3700 10 monclient: get_monmap_and_config success 2023-09-30T02:10:02.917 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.911+0000 7f01f77fe700 1 -- 172.21.15.59:0/3273944783 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01f00092e0 con 0x7f01f813ad70 2023-09-30T02:10:02.917 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.911+0000 7f01f77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:10:02.917 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.911+0000 7f01feaa3700 4 set_mon_vals no callback set 2023-09-30T02:10:02.917 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.911+0000 7f01f77fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:10:02.917 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.911+0000 7f01f77fe700 10 monclient: dump: 2023-09-30T02:10:02.917 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:10:02.918 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:02.918 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:02.918 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:02.918 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:10:02.918 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:02.918 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:10:02.918 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:10:02.918 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:10:02.918 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:02.918 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.912+0000 7f01feaa3700 10 monclient: shutdown 2023-09-30T02:10:02.918 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.912+0000 7f01feaa3700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:10:02.919 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.912+0000 7f01feaa3700 1 -- 172.21.15.59:0/3273944783 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01f813ad70 msgr2=0x7f01f813b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:02.919 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.912+0000 7f01feaa3700 1 --2- 172.21.15.59:0/3273944783 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01f813ad70 0x7f01f813b1f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f01f0009f20 tx=0x7f01f0009840 comp rx=0 tx=0).stop 2023-09-30T02:10:02.919 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.912+0000 7f01feaa3700 1 -- 172.21.15.59:0/3273944783 shutdown_connections 2023-09-30T02:10:02.919 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.912+0000 7f01feaa3700 1 --2- 172.21.15.59:0/3273944783 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f01f806b810 0x7f01f806e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:02.919 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.912+0000 7f01feaa3700 1 --2- 172.21.15.59:0/3273944783 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01f813ad70 0x7f01f813b1f0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:02.919 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.912+0000 7f01feaa3700 1 --2- 172.21.15.59:0/3273944783 >> v2:172.21.15.59:3300/0 conn(0x7f01f806e540 0x7f01f813a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:02.919 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.912+0000 7f01feaa3700 1 -- 172.21.15.59:0/3273944783 >> 172.21.15.59:0/3273944783 conn(0x7f01f8067620 msgr2=0x7f01f8067a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:10:02.920 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.912+0000 7f01feaa3700 1 -- 172.21.15.59:0/3273944783 shutdown_connections 2023-09-30T02:10:02.920 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.912+0000 7f01feaa3700 1 -- 172.21.15.59:0/3273944783 wait complete. 2023-09-30T02:10:02.920 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.913+0000 7f01feaa3700 10 monclient: build_initial_monmap 2023-09-30T02:10:02.920 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.913+0000 7f01feaa3700 10 monclient: monmap: 2023-09-30T02:10:02.920 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:10:02.920 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:10:02.921 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:10:02.921 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:10:02.921 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:10:02.921 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:10:02.921 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:10:02.921 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:10:02.921 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:10:02.922 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:02.922 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.913+0000 7f01feaa3700 1 librados: starting msgr at 2023-09-30T02:10:02.922 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.913+0000 7f01feaa3700 1 librados: starting objecter 2023-09-30T02:10:02.922 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.913+0000 7f01feaa3700 1 Processor -- start 2023-09-30T02:10:02.922 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.913+0000 7f01feaa3700 1 -- start start 2023-09-30T02:10:02.922 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.913+0000 7f01feaa3700 1 librados: setting wanted keys 2023-09-30T02:10:02.923 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.913+0000 7f01feaa3700 1 librados: calling monclient init 2023-09-30T02:10:02.923 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.913+0000 7f01feaa3700 10 monclient: init 2023-09-30T02:10:02.927 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.914+0000 7f01feaa3700 10 monclient: _reopen_session rank -1 2023-09-30T02:10:02.928 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.914+0000 7f01feaa3700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T02:10:02.928 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.914+0000 7f01feaa3700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f01f806b810 0x7f01f811e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:02.928 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.914+0000 7f01feaa3700 10 monclient(hunting): picked mon.noname-c con 0x7f01f806b810 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:10:02.928 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.914+0000 7f01feaa3700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01f806e540 0x7f01f811d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:02.928 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.914+0000 7f01feaa3700 10 monclient(hunting): picked mon.noname-b con 0x7f01f806e540 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:10:02.929 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.914+0000 7f01feaa3700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01f813ad70 0x7f01f811aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:02.929 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.914+0000 7f01feaa3700 10 monclient(hunting): picked mon.noname-a con 0x7f01f813ad70 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:10:02.929 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.914+0000 7f01feaa3700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:02.929 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.914+0000 7f01feaa3700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01f811f5a0 con 0x7f01f813ad70 2023-09-30T02:10:02.929 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.914+0000 7f01feaa3700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:02.930 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.914+0000 7f01feaa3700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01f811e830 con 0x7f01f806e540 2023-09-30T02:10:02.930 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.914+0000 7f01feaa3700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:02.930 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.914+0000 7f01feaa3700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01f811f810 con 0x7f01f806b810 2023-09-30T02:10:02.930 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.914+0000 7f01feaa3700 10 monclient(hunting): _renew_subs 2023-09-30T02:10:02.930 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.914+0000 7f01feaa3700 10 monclient(hunting): authenticate will time out at 2048.217570s 2023-09-30T02:10:02.930 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.915+0000 7f01fd040700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01f813ad70 0x7f01f811aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:02.931 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.916+0000 7f01f7fff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01f806e540 0x7f01f811d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:02.931 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.916+0000 7f01fd040700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01f813ad70 0x7f01f811aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:43056/0 (socket says 172.21.15.59:43056) 2023-09-30T02:10:02.931 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.916+0000 7f01fd040700 1 -- 172.21.15.59:0/1891057205 learned_addr learned my addr 172.21.15.59:0/1891057205 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:10:02.931 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.916+0000 7f01fd040700 10 monclient(hunting): get_auth_request con 0x7f01f813ad70 auth_method 0 2023-09-30T02:10:02.932 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.916+0000 7f01fd040700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:10:02.932 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.916+0000 7f01fd040700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:02.932 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.916+0000 7f01fd040700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:02.932 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.916+0000 7f01f7fff700 10 monclient(hunting): get_auth_request con 0x7f01f806e540 auth_method 0 2023-09-30T02:10:02.932 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.916+0000 7f01f7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:10:02.932 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.916+0000 7f01f7fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:02.933 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.916+0000 7f01f7fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:02.933 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.916+0000 7f01fd040700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:10:02.933 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.916+0000 7f01fd040700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:10:02.933 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.916+0000 7f01fd040700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:10:02.933 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.916+0000 7f01f7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:10:02.934 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.916+0000 7f01f7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:10:02.934 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.916+0000 7f01f7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:10:02.934 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.918+0000 7f01fc83f700 1 --2- 172.21.15.59:0/1891057205 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f01f806b810 0x7f01f811e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:02.934 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.918+0000 7f01fc83f700 10 monclient(hunting): get_auth_request con 0x7f01f806b810 auth_method 0 2023-09-30T02:10:02.934 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.918+0000 7f01fc83f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:10:02.935 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.918+0000 7f01fc83f700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:02.935 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.918+0000 7f01fc83f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:02.935 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.919+0000 7f01fc83f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:10:02.935 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.919+0000 7f01fc83f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:10:02.935 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.919+0000 7f01fc83f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:10:02.935 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.920+0000 7f01f7fff700 10 monclient(hunting): handle_auth_done global_id 4720 payload 824 2023-09-30T02:10:02.936 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.920+0000 7f01f7fff700 1 -- 172.21.15.59:0/1891057205 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f01f806b810 msgr2=0x7f01f811e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:02.936 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.920+0000 7f01f7fff700 1 --2- 172.21.15.59:0/1891057205 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f01f806b810 0x7f01f811e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:02.936 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.920+0000 7f01f7fff700 1 -- 172.21.15.59:0/1891057205 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01f813ad70 msgr2=0x7f01f811aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:02.936 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.920+0000 7f01f7fff700 1 --2- 172.21.15.59:0/1891057205 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01f813ad70 0x7f01f811aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:02.936 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.920+0000 7f01f7fff700 10 monclient: _finish_hunting 0 2023-09-30T02:10:02.937 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.920+0000 7f01f7fff700 1 monclient: found mon.noname-b 2023-09-30T02:10:02.937 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.920+0000 7f01f7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:10:02.937 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.920+0000 7f01f7fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:10:02.937 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.920+0000 7f01f7fff700 1 -- 172.21.15.59:0/1891057205 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01f0009ae0 con 0x7f01f806e540 2023-09-30T02:10:02.937 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.922+0000 7f01f7fff700 1 --2- 172.21.15.59:0/1891057205 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01f806e540 0x7f01f811d400 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f01e000e9a0 tx=0x7f01e000a330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:10:02.937 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.922+0000 7f01f4ff9700 10 client.?.objecter ms_handle_connect 0x7f01f806e540 2023-09-30T02:10:02.938 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.922+0000 7f01f4ff9700 10 client.?.objecter resend_mon_ops 2023-09-30T02:10:02.938 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.925+0000 7f01f4ff9700 1 -- 172.21.15.59:0/1891057205 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01e000c6f0 con 0x7f01f806e540 2023-09-30T02:10:02.938 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.925+0000 7f01f4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:10:02.938 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.925+0000 7f01f4ff9700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:10:02.939 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.925+0000 7f01f4ff9700 10 monclient: dump: 2023-09-30T02:10:02.939 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:10:02.939 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:02.939 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:02.939 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:02.940 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:10:02.940 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:02.940 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:10:02.940 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:10:02.940 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:10:02.940 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:02.941 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.925+0000 7f01f4ff9700 10 monclient: _finish_auth 0 2023-09-30T02:10:02.941 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.925+0000 7f01f4ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:10:02.941 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.925+0000 7f01feaa3700 5 monclient: authenticate success, global_id 4720 2023-09-30T02:10:02.941 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.925+0000 7f01feaa3700 10 monclient: _renew_subs 2023-09-30T02:10:02.941 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.925+0000 7f01feaa3700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:10:02.942 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.925+0000 7f01feaa3700 1 -- 172.21.15.59:0/1891057205 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01f811c570 con 0x7f01f806e540 2023-09-30T02:10:02.942 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.925+0000 7f01feaa3700 10 client.4720.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:10:02.942 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.925+0000 7f01feaa3700 10 monclient: _renew_subs 2023-09-30T02:10:02.942 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.925+0000 7f01feaa3700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:10:02.942 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.925+0000 7f01feaa3700 1 -- 172.21.15.59:0/1891057205 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01f811b2a0 con 0x7f01f806e540 2023-09-30T02:10:02.943 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.925+0000 7f01feaa3700 1 librados: init done 2023-09-30T02:10:02.943 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.926+0000 7f01f4ff9700 1 -- 172.21.15.59:0/1891057205 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01e000ca60 con 0x7f01f806e540 2023-09-30T02:10:02.943 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.926+0000 7f01f4ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:10:02.943 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.926+0000 7f01f4ff9700 1 -- 172.21.15.59:0/1891057205 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01e000cbd0 con 0x7f01f806e540 2023-09-30T02:10:02.943 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.926+0000 7f01f4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:10:02.943 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.926+0000 7f01f4ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:10:02.944 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.926+0000 7f01f4ff9700 10 monclient: dump: 2023-09-30T02:10:02.944 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:10:02.944 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:02.944 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:02.944 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:02.945 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:10:02.945 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:02.945 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:10:02.945 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:10:02.945 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:10:02.946 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:02.946 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.926+0000 7f01f6ffd700 4 set_mon_vals no callback set 2023-09-30T02:10:02.946 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.926+0000 7f01feaa3700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:10:02.946 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.927+0000 7f01feaa3700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:10:02.946 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.927+0000 7f01feaa3700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:10:02.947 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.927+0000 7f01feaa3700 1 -- 172.21.15.59:0/1891057205 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01dc0050b0 con 0x7f01f806e540 2023-09-30T02:10:02.953 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.951+0000 7f01f4ff9700 1 -- 172.21.15.59:0/1891057205 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f01e0005a00 con 0x7f01f806e540 2023-09-30T02:10:02.953 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.951+0000 7f01f4ff9700 1 --2- 172.21.15.59:0/1891057205 >> v2:172.21.15.110:6800/115870 conn(0x7f01d8068940 0x7f01d806adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:02.953 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.952+0000 7f01f4ff9700 1 -- 172.21.15.59:0/1891057205 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f01e0015030 con 0x7f01f806e540 2023-09-30T02:10:02.954 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.952+0000 7f01f4ff9700 10 client.4720.objecter ms_dispatch 0x7f01f8001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:10:02.954 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.952+0000 7f01f4ff9700 3 client.4720.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:10:02.954 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.952+0000 7f01f4ff9700 3 client.4720.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:10:02.954 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.952+0000 7f01f4ff9700 20 client.4720.objecter dump_active .. 0 homeless 2023-09-30T02:10:02.955 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.952+0000 7f01fc83f700 1 --2- 172.21.15.59:0/1891057205 >> v2:172.21.15.110:6800/115870 conn(0x7f01d8068940 0x7f01d806adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:02.955 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.952+0000 7f01fc83f700 10 monclient: get_auth_request con 0x7f01d8068940 auth_method 0 2023-09-30T02:10:02.955 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.952+0000 7f01f4ff9700 10 client.4720.objecter ms_handle_connect 0x7f01d8068940 2023-09-30T02:10:02.955 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:02.952+0000 7f01fc83f700 1 --2- 172.21.15.59:0/1891057205 >> v2:172.21.15.110:6800/115870 conn(0x7f01d8068940 0x7f01d806adf0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f01e8009db0 tx=0x7f01e80091a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:10:03.193 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:03.194+0000 7f01f4ff9700 1 -- 172.21.15.59:0/1891057205 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f01e0051770 con 0x7f01f806e540 2023-09-30T02:10:03.194 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:03.194+0000 7f01f4ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:10:03.194 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:03.194+0000 7f01f4ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:10:03.444 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:03.443+0000 7f01feaa3700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:10:03.445 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:03.443+0000 7f01feaa3700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:10:03.445 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:03.443+0000 7f01feaa3700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:10:03.445 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:03.443+0000 7f01feaa3700 1 -- 172.21.15.59:0/1891057205 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f01dc005630 con 0x7f01f806e540 2023-09-30T02:10:03.449 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:03.449+0000 7f01f4ff9700 1 -- 172.21.15.59:0/1891057205 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f01e001f080 con 0x7f01f806e540 2023-09-30T02:10:03.449 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:03.449+0000 7f01f4ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:10:03.449 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:03.449+0000 7f01f4ff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:10:03.450 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:10:03.450 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:10:03.453 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:03.451+0000 7f01feaa3700 10 librados: watch_flush enter 2023-09-30T02:10:03.453 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:03.451+0000 7f01feaa3700 10 librados: watch_flush exit 2023-09-30T02:10:03.453 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:03.451+0000 7f01feaa3700 20 client.4720.objecter shutdown clearing up homeless session... 2023-09-30T02:10:03.454 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:03.451+0000 7f01feaa3700 10 client.4720.objecter successfully canceled tick 2023-09-30T02:10:03.454 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:03.451+0000 7f01feaa3700 1 -- 172.21.15.59:0/1891057205 >> v2:172.21.15.110:6800/115870 conn(0x7f01d8068940 msgr2=0x7f01d806adf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:03.454 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:03.451+0000 7f01feaa3700 1 --2- 172.21.15.59:0/1891057205 >> v2:172.21.15.110:6800/115870 conn(0x7f01d8068940 0x7f01d806adf0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f01e8009db0 tx=0x7f01e80091a0 comp rx=0 tx=0).stop 2023-09-30T02:10:03.454 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:03.451+0000 7f01feaa3700 10 monclient: shutdown 2023-09-30T02:10:03.454 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:03.451+0000 7f01feaa3700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:10:03.454 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:03.451+0000 7f01feaa3700 1 -- 172.21.15.59:0/1891057205 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01f806e540 msgr2=0x7f01f811d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:03.454 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:03.451+0000 7f01feaa3700 1 --2- 172.21.15.59:0/1891057205 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01f806e540 0x7f01f811d400 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f01e000e9a0 tx=0x7f01e000a330 comp rx=0 tx=0).stop 2023-09-30T02:10:03.455 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:03.451+0000 7f01feaa3700 1 -- 172.21.15.59:0/1891057205 shutdown_connections 2023-09-30T02:10:03.455 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:03.451+0000 7f01feaa3700 1 --2- 172.21.15.59:0/1891057205 >> v2:172.21.15.110:6800/115870 conn(0x7f01d8068940 0x7f01d806adf0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:03.455 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:03.451+0000 7f01feaa3700 1 --2- 172.21.15.59:0/1891057205 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01f813ad70 0x7f01f811aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:03.455 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:03.451+0000 7f01feaa3700 1 --2- 172.21.15.59:0/1891057205 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f01f806b810 0x7f01f811e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:03.455 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:03.451+0000 7f01feaa3700 1 --2- 172.21.15.59:0/1891057205 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f01f806e540 0x7f01f811d400 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:03.455 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:03.451+0000 7f01feaa3700 1 -- 172.21.15.59:0/1891057205 >> 172.21.15.59:0/1891057205 conn(0x7f01f8067620 msgr2=0x7f01f80ad5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:10:03.456 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:03.451+0000 7f01feaa3700 1 -- 172.21.15.59:0/1891057205 shutdown_connections 2023-09-30T02:10:03.456 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:03.451+0000 7f01feaa3700 1 -- 172.21.15.59:0/1891057205 wait complete. 2023-09-30T02:10:03.456 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:03.451+0000 7f01feaa3700 1 librados: shutdown 2023-09-30T02:10:06.467 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:10:06.607 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.607+0000 7f7c2f543700 10 monclient: get_monmap_and_config 2023-09-30T02:10:06.607 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.607+0000 7f7c2f543700 10 monclient: build_initial_monmap 2023-09-30T02:10:06.607 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.607+0000 7f7c2f543700 10 monclient: monmap: 2023-09-30T02:10:06.607 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:10:06.607 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:06.607 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:10:06.608088+0000 2023-09-30T02:10:06.607 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:10:06.608088+0000 2023-09-30T02:10:06.608 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:10:06.608 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:06.608 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:10:06.608 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:10:06.608 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:10:06.608 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:10:06.608 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:10:06.608 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:06.610 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.608+0000 7f7c2f543700 1 Processor -- start 2023-09-30T02:10:06.610 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.608+0000 7f7c2f543700 1 -- start start 2023-09-30T02:10:06.610 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.608+0000 7f7c2f543700 10 monclient: init 2023-09-30T02:10:06.610 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.609+0000 7f7c2f543700 10 monclient: _reopen_session rank -1 2023-09-30T02:10:06.610 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.609+0000 7f7c2f543700 10 monclient: _add_conns ranks=[1,4,0,2,3] 2023-09-30T02:10:06.610 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.609+0000 7f7c2f543700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7c280a5b70 addr v1:172.21.15.110:6789/0 2023-09-30T02:10:06.610 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.609+0000 7f7c2f543700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f7c28142290 0x7f7c280b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:06.610 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.609+0000 7f7c2f543700 10 monclient(hunting): picked mon.noname-c con 0x7f7c28142290 addr v2:172.21.15.110:3300/0 2023-09-30T02:10:06.610 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.609+0000 7f7c2f543700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7c280a5f40 addr v1:172.21.15.59:6789/0 2023-09-30T02:10:06.610 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.609+0000 7f7c2f543700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c280a75a0 con 0x7f7c280a5f40 2023-09-30T02:10:06.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.609+0000 7f7c2f543700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c280a6360 con 0x7f7c280a5b70 2023-09-30T02:10:06.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.609+0000 7f7c2f543700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:06.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.609+0000 7f7c2f543700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c280af3f0 con 0x7f7c28142290 2023-09-30T02:10:06.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.609+0000 7f7c2f543700 10 monclient(hunting): _renew_subs 2023-09-30T02:10:06.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.609+0000 7f7c2f543700 10 monclient(hunting): authenticate will time out at 2051.911424s 2023-09-30T02:10:06.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.610+0000 7f7c2dae0700 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f7c280a5f40 0x7f7c2813af30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:52960/0 (socket says 172.21.15.59:52960) 2023-09-30T02:10:06.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.610+0000 7f7c2dae0700 1 -- 172.21.15.59:0/214037968 learned_addr learned my addr 172.21.15.59:0/214037968 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:10:06.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.610+0000 7f7c2cade700 1 --2- 172.21.15.59:0/214037968 >> v2:172.21.15.110:3300/0 conn(0x7f7c28142290 0x7f7c280b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:06.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.610+0000 7f7c2cade700 10 monclient(hunting): get_auth_request con 0x7f7c28142290 auth_method 0 2023-09-30T02:10:06.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.610+0000 7f7c2cade700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:10:06.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.610+0000 7f7c2cade700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:06.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.610+0000 7f7c2cade700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:06.612 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.611+0000 7f7c2cade700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:10:06.612 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.611+0000 7f7c2cade700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:10:06.612 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.611+0000 7f7c2cade700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:10:06.612 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.611+0000 7f7c1ffff700 1 -- 172.21.15.59:0/214037968 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1637938516 0 0) 0x7f7c280a75a0 con 0x7f7c280a5f40 2023-09-30T02:10:06.612 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.613+0000 7f7c1ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:06.613 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.613+0000 7f7c1ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:06.613 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.613+0000 7f7c1ffff700 10 monclient(hunting): my global_id is 4689 2023-09-30T02:10:06.613 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.613+0000 7f7c1ffff700 1 -- 172.21.15.59:0/214037968 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c14003570 con 0x7f7c280a5f40 2023-09-30T02:10:06.613 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.614+0000 7f7c2cade700 10 monclient(hunting): handle_auth_done global_id 4723 payload 274 2023-09-30T02:10:06.613 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.614+0000 7f7c2cade700 1 -- 172.21.15.59:0/214037968 >> v1:172.21.15.110:6789/0 conn(0x7f7c280a5b70 legacy=0x7f7c28141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:06.614 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.614+0000 7f7c2cade700 1 -- 172.21.15.59:0/214037968 >> v1:172.21.15.59:6789/0 conn(0x7f7c280a5f40 legacy=0x7f7c2813af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:06.614 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.614+0000 7f7c2cade700 10 monclient: _finish_hunting 0 2023-09-30T02:10:06.614 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.614+0000 7f7c2cade700 1 monclient: found mon.noname-c 2023-09-30T02:10:06.615 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.614+0000 7f7c2cade700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:10:06.615 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.614+0000 7f7c2cade700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3300/0 2023-09-30T02:10:06.615 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.614+0000 7f7c2cade700 1 -- 172.21.15.59:0/214037968 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c2813b640 con 0x7f7c28142290 2023-09-30T02:10:06.616 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.617+0000 7f7c2cade700 1 --2- 172.21.15.59:0/214037968 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7c28142290 0x7f7c280b1500 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f7c180098e0 tx=0x7f7c180091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=798423fa3e9d5acd server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:10:06.625 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.624+0000 7f7c1ffff700 1 -- 172.21.15.59:0/214037968 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c1800b640 con 0x7f7c28142290 2023-09-30T02:10:06.625 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.624+0000 7f7c1ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:10:06.625 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.624+0000 7f7c1ffff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:10:06.625 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.624+0000 7f7c1ffff700 10 monclient: dump: 2023-09-30T02:10:06.625 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:10:06.625 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:06.625 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:06.625 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:06.626 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:10:06.626 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:06.626 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:10:06.626 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:10:06.626 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:10:06.626 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:06.626 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.624+0000 7f7c1ffff700 10 monclient: _finish_auth 0 2023-09-30T02:10:06.626 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.624+0000 7f7c1ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:10:06.626 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.624+0000 7f7c1ffff700 1 -- 172.21.15.59:0/214037968 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c1800b7a0 con 0x7f7c28142290 2023-09-30T02:10:06.626 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.624+0000 7f7c2f543700 5 monclient: authenticate success, global_id 4723 2023-09-30T02:10:06.626 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.624+0000 7f7c2f543700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:10:06.626 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.624+0000 7f7c1ffff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:10:06.627 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.624+0000 7f7c1ffff700 1 -- 172.21.15.59:0/214037968 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c1800bb40 con 0x7f7c28142290 2023-09-30T02:10:06.627 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.624+0000 7f7c1ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:10:06.627 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.624+0000 7f7c1ffff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:10:06.627 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.624+0000 7f7c1ffff700 10 monclient: dump: 2023-09-30T02:10:06.627 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:10:06.627 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:06.627 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:06.627 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:06.627 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:10:06.627 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:06.627 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:10:06.627 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:10:06.627 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:10:06.628 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:06.628 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.624+0000 7f7c2f543700 10 monclient: get_monmap_and_config success 2023-09-30T02:10:06.628 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.624+0000 7f7c2f543700 4 set_mon_vals no callback set 2023-09-30T02:10:06.628 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.624+0000 7f7c2f543700 10 monclient: shutdown 2023-09-30T02:10:06.628 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.624+0000 7f7c2f543700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:10:06.628 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.624+0000 7f7c2f543700 1 -- 172.21.15.59:0/214037968 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7c28142290 msgr2=0x7f7c280b1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:06.628 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.624+0000 7f7c2f543700 1 --2- 172.21.15.59:0/214037968 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7c28142290 0x7f7c280b1500 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f7c180098e0 tx=0x7f7c180091a0 comp rx=0 tx=0).stop 2023-09-30T02:10:06.628 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.624+0000 7f7c2f543700 1 -- 172.21.15.59:0/214037968 shutdown_connections 2023-09-30T02:10:06.628 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.624+0000 7f7c2f543700 1 --2- 172.21.15.59:0/214037968 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7c28142290 0x7f7c280b1500 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:06.628 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.624+0000 7f7c2f543700 1 -- 172.21.15.59:0/214037968 >> 172.21.15.59:0/214037968 conn(0x7f7c280631f0 msgr2=0x7f7c280635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:10:06.628 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.624+0000 7f7c2f543700 1 -- 172.21.15.59:0/214037968 shutdown_connections 2023-09-30T02:10:06.629 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.624+0000 7f7c2f543700 1 -- 172.21.15.59:0/214037968 wait complete. 2023-09-30T02:10:06.631 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.626+0000 7f7c2f543700 10 monclient: build_initial_monmap 2023-09-30T02:10:06.631 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.626+0000 7f7c2f543700 10 monclient: monmap: 2023-09-30T02:10:06.631 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:10:06.631 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:10:06.631 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:10:06.631 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:10:06.631 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:10:06.632 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:10:06.632 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:10:06.632 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:10:06.632 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:10:06.632 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:06.632 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.626+0000 7f7c2f543700 1 librados: starting msgr at 2023-09-30T02:10:06.632 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.626+0000 7f7c2f543700 1 librados: starting objecter 2023-09-30T02:10:06.632 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.626+0000 7f7c2f543700 1 Processor -- start 2023-09-30T02:10:06.632 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.627+0000 7f7c2f543700 1 -- start start 2023-09-30T02:10:06.632 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.627+0000 7f7c2f543700 1 librados: setting wanted keys 2023-09-30T02:10:06.632 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.627+0000 7f7c2f543700 1 librados: calling monclient init 2023-09-30T02:10:06.632 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.627+0000 7f7c2f543700 10 monclient: init 2023-09-30T02:10:06.633 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.627+0000 7f7c2f543700 10 monclient: _reopen_session rank -1 2023-09-30T02:10:06.633 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.627+0000 7f7c2f543700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T02:10:06.633 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.627+0000 7f7c2f543700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7c280a5b70 0x7f7c28144ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:06.633 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.627+0000 7f7c2f543700 10 monclient(hunting): picked mon.noname-b con 0x7f7c280a5b70 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:10:06.633 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.627+0000 7f7c2f543700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7c280a5f40 0x7f7c2813c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:06.633 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.627+0000 7f7c2f543700 10 monclient(hunting): picked mon.noname-a con 0x7f7c280a5f40 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:10:06.633 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.627+0000 7f7c2f543700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7c28142290 0x7f7c2813ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:06.633 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.627+0000 7f7c2f543700 10 monclient(hunting): picked mon.noname-c con 0x7f7c28142290 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:10:06.633 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.627+0000 7f7c2f543700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:06.634 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.627+0000 7f7c2f543700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c2813d170 con 0x7f7c280a5f40 2023-09-30T02:10:06.634 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.627+0000 7f7c2f543700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:06.635 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.627+0000 7f7c2f543700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c2813cf80 con 0x7f7c280a5b70 2023-09-30T02:10:06.635 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.627+0000 7f7c2f543700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:06.635 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.627+0000 7f7c2f543700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c2813e020 con 0x7f7c28142290 2023-09-30T02:10:06.635 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.627+0000 7f7c2f543700 10 monclient(hunting): _renew_subs 2023-09-30T02:10:06.635 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.627+0000 7f7c2f543700 10 monclient(hunting): authenticate will time out at 2051.929873s 2023-09-30T02:10:06.635 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.628+0000 7f7c2cade700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7c280a5f40 0x7f7c2813c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:06.635 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.628+0000 7f7c2dae0700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7c28142290 0x7f7c2813ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:06.635 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.628+0000 7f7c2d2df700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7c280a5b70 0x7f7c28144ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:06.635 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.628+0000 7f7c2d2df700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7c280a5b70 0x7f7c28144ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:40566/0 (socket says 172.21.15.59:40566) 2023-09-30T02:10:06.635 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.628+0000 7f7c2d2df700 1 -- 172.21.15.59:0/2400390968 learned_addr learned my addr 172.21.15.59:0/2400390968 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:10:06.636 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.628+0000 7f7c2d2df700 10 monclient(hunting): get_auth_request con 0x7f7c280a5b70 auth_method 0 2023-09-30T02:10:06.636 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.628+0000 7f7c2d2df700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:10:06.636 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.628+0000 7f7c2d2df700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:06.636 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.628+0000 7f7c2d2df700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:06.636 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.628+0000 7f7c2cade700 10 monclient(hunting): get_auth_request con 0x7f7c280a5f40 auth_method 0 2023-09-30T02:10:06.636 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.628+0000 7f7c2cade700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:10:06.636 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.628+0000 7f7c2cade700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:06.636 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.628+0000 7f7c2cade700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:06.636 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.629+0000 7f7c2dae0700 10 monclient(hunting): get_auth_request con 0x7f7c28142290 auth_method 0 2023-09-30T02:10:06.636 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.629+0000 7f7c2dae0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:10:06.637 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.629+0000 7f7c2dae0700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:06.637 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.629+0000 7f7c2dae0700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:06.637 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.629+0000 7f7c2d2df700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:10:06.637 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.629+0000 7f7c2d2df700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:10:06.637 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.629+0000 7f7c2d2df700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:10:06.637 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.629+0000 7f7c2dae0700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:10:06.637 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.629+0000 7f7c2dae0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:10:06.637 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.629+0000 7f7c2dae0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:10:06.637 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.629+0000 7f7c2cade700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:10:06.637 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.629+0000 7f7c2cade700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:10:06.637 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.629+0000 7f7c2cade700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:10:06.640 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.632+0000 7f7c2d2df700 10 monclient(hunting): handle_auth_done global_id 4729 payload 824 2023-09-30T02:10:06.640 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.632+0000 7f7c2d2df700 1 -- 172.21.15.59:0/2400390968 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7c28142290 msgr2=0x7f7c2813ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:06.640 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.632+0000 7f7c2d2df700 1 --2- 172.21.15.59:0/2400390968 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7c28142290 0x7f7c2813ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:06.640 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.632+0000 7f7c2d2df700 1 -- 172.21.15.59:0/2400390968 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7c280a5f40 msgr2=0x7f7c2813c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:06.640 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.632+0000 7f7c2d2df700 1 --2- 172.21.15.59:0/2400390968 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7c280a5f40 0x7f7c2813c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:06.640 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.632+0000 7f7c2d2df700 10 monclient: _finish_hunting 0 2023-09-30T02:10:06.641 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.632+0000 7f7c2d2df700 1 monclient: found mon.noname-b 2023-09-30T02:10:06.641 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.632+0000 7f7c2d2df700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:10:06.641 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.632+0000 7f7c2d2df700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:10:06.641 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.632+0000 7f7c2d2df700 1 -- 172.21.15.59:0/2400390968 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c18009420 con 0x7f7c280a5b70 2023-09-30T02:10:06.641 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.635+0000 7f7c2d2df700 1 --2- 172.21.15.59:0/2400390968 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7c280a5b70 0x7f7c28144ff0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f7c2400c590 tx=0x7f7c2400c8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:10:06.641 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.635+0000 7f7c1d7fa700 10 client.?.objecter ms_handle_connect 0x7f7c280a5b70 2023-09-30T02:10:06.641 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.635+0000 7f7c1d7fa700 10 client.?.objecter resend_mon_ops 2023-09-30T02:10:06.642 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.639+0000 7f7c1d7fa700 1 -- 172.21.15.59:0/2400390968 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c24011590 con 0x7f7c280a5b70 2023-09-30T02:10:06.642 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.639+0000 7f7c1d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:10:06.642 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.639+0000 7f7c1d7fa700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:10:06.642 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.639+0000 7f7c1d7fa700 10 monclient: dump: 2023-09-30T02:10:06.642 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:10:06.642 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:06.642 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:06.642 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:06.643 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:10:06.643 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:06.643 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:10:06.643 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:10:06.643 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:10:06.643 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:06.643 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.639+0000 7f7c1d7fa700 10 monclient: _finish_auth 0 2023-09-30T02:10:06.643 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.639+0000 7f7c1d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:10:06.644 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.639+0000 7f7c1d7fa700 1 -- 172.21.15.59:0/2400390968 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c24011710 con 0x7f7c280a5b70 2023-09-30T02:10:06.644 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.639+0000 7f7c1d7fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:10:06.645 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.639+0000 7f7c1d7fa700 1 -- 172.21.15.59:0/2400390968 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c24011ad0 con 0x7f7c280a5b70 2023-09-30T02:10:06.645 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.639+0000 7f7c2f543700 5 monclient: authenticate success, global_id 4729 2023-09-30T02:10:06.645 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.639+0000 7f7c2f543700 10 monclient: _renew_subs 2023-09-30T02:10:06.645 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.639+0000 7f7c2f543700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:10:06.646 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.639+0000 7f7c2f543700 1 -- 172.21.15.59:0/2400390968 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c2813e300 con 0x7f7c280a5b70 2023-09-30T02:10:06.646 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.639+0000 7f7c1d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:10:06.646 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.639+0000 7f7c1d7fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:10:06.646 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.639+0000 7f7c1d7fa700 10 monclient: dump: 2023-09-30T02:10:06.646 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:10:06.647 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:06.647 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:06.647 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:06.647 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:10:06.647 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:06.648 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:10:06.648 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:10:06.648 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:10:06.648 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:06.648 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.639+0000 7f7c1f7fe700 4 set_mon_vals no callback set 2023-09-30T02:10:06.649 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.639+0000 7f7c2f543700 10 client.4729.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:10:06.649 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.639+0000 7f7c2f543700 10 monclient: _renew_subs 2023-09-30T02:10:06.649 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.639+0000 7f7c2f543700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:10:06.649 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.639+0000 7f7c2f543700 1 -- 172.21.15.59:0/2400390968 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c2813e7c0 con 0x7f7c280a5b70 2023-09-30T02:10:06.649 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.639+0000 7f7c2f543700 1 librados: init done 2023-09-30T02:10:06.649 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.640+0000 7f7c2f543700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:10:06.649 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.640+0000 7f7c2f543700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:10:06.649 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.640+0000 7f7c2f543700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:10:06.650 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.640+0000 7f7c2f543700 1 -- 172.21.15.59:0/2400390968 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c2813dd40 con 0x7f7c280a5b70 2023-09-30T02:10:06.894 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.894+0000 7f7c1d7fa700 1 -- 172.21.15.59:0/2400390968 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f7c2401b3b0 con 0x7f7c280a5b70 2023-09-30T02:10:06.895 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.895+0000 7f7c1d7fa700 1 --2- 172.21.15.59:0/2400390968 >> v2:172.21.15.110:6800/115870 conn(0x7f7c0c068960 0x7f7c0c06ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:06.895 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.895+0000 7f7c1d7fa700 1 -- 172.21.15.59:0/2400390968 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f7c240835e0 con 0x7f7c280a5b70 2023-09-30T02:10:06.895 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.895+0000 7f7c1d7fa700 10 client.4729.objecter ms_dispatch 0x7f7c28001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:10:06.895 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.895+0000 7f7c1d7fa700 3 client.4729.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:10:06.895 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.895+0000 7f7c1d7fa700 3 client.4729.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:10:06.895 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.895+0000 7f7c1d7fa700 20 client.4729.objecter dump_active .. 0 homeless 2023-09-30T02:10:06.895 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.895+0000 7f7c2cade700 1 --2- 172.21.15.59:0/2400390968 >> v2:172.21.15.110:6800/115870 conn(0x7f7c0c068960 0x7f7c0c06ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:06.895 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.895+0000 7f7c2cade700 10 monclient: get_auth_request con 0x7f7c0c068960 auth_method 0 2023-09-30T02:10:06.896 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.896+0000 7f7c1d7fa700 10 client.4729.objecter ms_handle_connect 0x7f7c0c068960 2023-09-30T02:10:06.896 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.896+0000 7f7c2cade700 1 --2- 172.21.15.59:0/2400390968 >> v2:172.21.15.110:6800/115870 conn(0x7f7c0c068960 0x7f7c0c06ae10 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f7c180093f0 tx=0x7f7c18017000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:10:06.903 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.903+0000 7f7c1d7fa700 1 -- 172.21.15.59:0/2400390968 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7c24050f00 con 0x7f7c280a5b70 2023-09-30T02:10:06.903 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.903+0000 7f7c1d7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:10:06.903 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:06.903+0000 7f7c1d7fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:10:07.160 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:07.161+0000 7f7c2f543700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:10:07.161 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:07.161+0000 7f7c2f543700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:10:07.161 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:07.161+0000 7f7c2f543700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:10:07.161 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:07.161+0000 7f7c2f543700 1 -- 172.21.15.59:0/2400390968 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f7c2813dd40 con 0x7f7c280a5b70 2023-09-30T02:10:07.217 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:07.217+0000 7f7c1d7fa700 1 -- 172.21.15.59:0/2400390968 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f7c24017100 con 0x7f7c280a5b70 2023-09-30T02:10:07.217 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:07.217+0000 7f7c1d7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:10:07.217 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:07.217+0000 7f7c1d7fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:10:07.217 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:10:07.217 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:10:07.220 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:07.219+0000 7f7c2f543700 10 librados: watch_flush enter 2023-09-30T02:10:07.220 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:07.219+0000 7f7c2f543700 10 librados: watch_flush exit 2023-09-30T02:10:07.220 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:07.219+0000 7f7c2f543700 20 client.4729.objecter shutdown clearing up homeless session... 2023-09-30T02:10:07.220 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:07.220+0000 7f7c2f543700 10 client.4729.objecter successfully canceled tick 2023-09-30T02:10:07.220 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:07.220+0000 7f7c2f543700 1 -- 172.21.15.59:0/2400390968 >> v2:172.21.15.110:6800/115870 conn(0x7f7c0c068960 msgr2=0x7f7c0c06ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:07.220 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:07.220+0000 7f7c2f543700 1 --2- 172.21.15.59:0/2400390968 >> v2:172.21.15.110:6800/115870 conn(0x7f7c0c068960 0x7f7c0c06ae10 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f7c180093f0 tx=0x7f7c18017000 comp rx=0 tx=0).stop 2023-09-30T02:10:07.220 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:07.220+0000 7f7c2f543700 10 monclient: shutdown 2023-09-30T02:10:07.221 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:07.220+0000 7f7c2f543700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:10:07.221 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:07.220+0000 7f7c2f543700 1 -- 172.21.15.59:0/2400390968 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7c280a5b70 msgr2=0x7f7c28144ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:07.221 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:07.220+0000 7f7c2f543700 1 --2- 172.21.15.59:0/2400390968 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7c280a5b70 0x7f7c28144ff0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f7c2400c590 tx=0x7f7c2400c8a0 comp rx=0 tx=0).stop 2023-09-30T02:10:07.221 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:07.220+0000 7f7c2f543700 1 -- 172.21.15.59:0/2400390968 shutdown_connections 2023-09-30T02:10:07.221 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:07.220+0000 7f7c2f543700 1 --2- 172.21.15.59:0/2400390968 >> v2:172.21.15.110:6800/115870 conn(0x7f7c0c068960 0x7f7c0c06ae10 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:07.221 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:07.220+0000 7f7c2f543700 1 --2- 172.21.15.59:0/2400390968 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f7c280a5b70 0x7f7c28144ff0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:07.221 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:07.220+0000 7f7c2f543700 1 --2- 172.21.15.59:0/2400390968 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7c28142290 0x7f7c2813ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:07.221 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:07.220+0000 7f7c2f543700 1 --2- 172.21.15.59:0/2400390968 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7c280a5f40 0x7f7c2813c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:07.221 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:07.220+0000 7f7c2f543700 1 -- 172.21.15.59:0/2400390968 >> 172.21.15.59:0/2400390968 conn(0x7f7c280631f0 msgr2=0x7f7c280af530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:10:07.222 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:07.220+0000 7f7c2f543700 1 -- 172.21.15.59:0/2400390968 shutdown_connections 2023-09-30T02:10:07.222 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:07.220+0000 7f7c2f543700 1 -- 172.21.15.59:0/2400390968 wait complete. 2023-09-30T02:10:07.222 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:07.220+0000 7f7c2f543700 1 librados: shutdown 2023-09-30T02:10:10.236 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:10:10.378 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.379+0000 7f927a7de700 10 monclient: get_monmap_and_config 2023-09-30T02:10:10.379 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.379+0000 7f927a7de700 10 monclient: build_initial_monmap 2023-09-30T02:10:10.379 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.379+0000 7f927a7de700 10 monclient: monmap: 2023-09-30T02:10:10.379 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:10:10.379 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:10.379 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:10:10.379805+0000 2023-09-30T02:10:10.379 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:10:10.379805+0000 2023-09-30T02:10:10.379 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:10:10.379 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:10.380 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:10:10.380 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:10:10.380 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:10:10.380 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:10:10.380 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:10:10.380 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:10.382 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.380+0000 7f927a7de700 1 Processor -- start 2023-09-30T02:10:10.382 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.380+0000 7f927a7de700 1 -- start start 2023-09-30T02:10:10.382 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.380+0000 7f927a7de700 10 monclient: init 2023-09-30T02:10:10.382 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.381+0000 7f927a7de700 10 monclient: _reopen_session rank -1 2023-09-30T02:10:10.382 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.381+0000 7f927a7de700 10 monclient: _add_conns ranks=[1,3,2,0,4] 2023-09-30T02:10:10.382 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.381+0000 7f927a7de700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f92740a5b70 addr v1:172.21.15.110:6789/0 2023-09-30T02:10:10.382 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.381+0000 7f927a7de700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f9274142290 0x7f92740b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:10.382 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.381+0000 7f927a7de700 10 monclient(hunting): picked mon.noname-a con 0x7f9274142290 addr v2:172.21.15.59:3300/0 2023-09-30T02:10:10.382 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.381+0000 7f927a7de700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f92740a5f40 0x7f927413af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:10.383 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.381+0000 7f927a7de700 10 monclient(hunting): picked mon.noname-b con 0x7f92740a5f40 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:10:10.383 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.381+0000 7f927a7de700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92740a6380 con 0x7f92740a5b70 2023-09-30T02:10:10.383 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.381+0000 7f927a7de700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:10.383 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.381+0000 7f927a7de700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92740a75a0 con 0x7f9274142290 2023-09-30T02:10:10.383 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.381+0000 7f927a7de700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:10.383 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.381+0000 7f927a7de700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92740af3f0 con 0x7f92740a5f40 2023-09-30T02:10:10.383 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.381+0000 7f927a7de700 10 monclient(hunting): _renew_subs 2023-09-30T02:10:10.383 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.381+0000 7f927a7de700 10 monclient(hunting): authenticate will time out at 2055.683218s 2023-09-30T02:10:10.383 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.381+0000 7f9278d7b700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f92740a5f40 0x7f927413af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:10.383 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.382+0000 7f9273fff700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f92740a5b70 0x7f9274141b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.59:56476/0 (socket says 172.21.15.59:56476) 2023-09-30T02:10:10.383 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.382+0000 7f9273fff700 1 -- 172.21.15.59:0/1648295955 learned_addr learned my addr 172.21.15.59:0/1648295955 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:10:10.384 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.382+0000 7f9278d7b700 10 monclient(hunting): get_auth_request con 0x7f92740a5f40 auth_method 0 2023-09-30T02:10:10.384 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.382+0000 7f9278d7b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:10:10.384 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.382+0000 7f9278d7b700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:10.384 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.382+0000 7f9278d7b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:10.384 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.382+0000 7f9278d7b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:10:10.384 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.382+0000 7f9278d7b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:10:10.384 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.382+0000 7f9278d7b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:10:10.385 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.385+0000 7f9278d7b700 10 monclient(hunting): handle_auth_done global_id 4625 payload 274 2023-09-30T02:10:10.385 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.385+0000 7f9278d7b700 1 -- 172.21.15.59:0/1648295955 >> v2:172.21.15.59:3300/0 conn(0x7f9274142290 msgr2=0x7f92740b1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:10:10.385 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.385+0000 7f9278d7b700 1 --2- 172.21.15.59:0/1648295955 >> v2:172.21.15.59:3300/0 conn(0x7f9274142290 0x7f92740b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:10.385 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.385+0000 7f9278d7b700 1 -- 172.21.15.59:0/1648295955 >> v1:172.21.15.110:6789/0 conn(0x7f92740a5b70 legacy=0x7f9274141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:10.385 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.385+0000 7f9278d7b700 10 monclient: _finish_hunting 0 2023-09-30T02:10:10.385 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.385+0000 7f9278d7b700 1 monclient: found mon.noname-b 2023-09-30T02:10:10.385 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.385+0000 7f9278d7b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:10:10.385 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.385+0000 7f9278d7b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.59:3301/0 2023-09-30T02:10:10.386 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.385+0000 7f9278d7b700 1 -- 172.21.15.59:0/1648295955 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f927413b470 con 0x7f92740a5f40 2023-09-30T02:10:10.386 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.387+0000 7f9278d7b700 1 --2- 172.21.15.59:0/1648295955 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f92740a5f40 0x7f927413af30 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f926c009e60 tx=0x7f926c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b6bcb4dd6337564 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:10:10.392 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.392+0000 7f9272ffd700 1 -- 172.21.15.59:0/1648295955 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f926c00e070 con 0x7f92740a5f40 2023-09-30T02:10:10.392 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.392+0000 7f9272ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:10:10.392 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.392+0000 7f9272ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:10:10.392 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.392+0000 7f9272ffd700 10 monclient: dump: 2023-09-30T02:10:10.392 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:10:10.392 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:10.392 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:10.393 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:10.393 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:10:10.393 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:10.393 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:10:10.393 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:10:10.393 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:10:10.393 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:10.393 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.392+0000 7f9272ffd700 10 monclient: _finish_auth 0 2023-09-30T02:10:10.393 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.392+0000 7f9272ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:10:10.393 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.392+0000 7f9272ffd700 1 -- 172.21.15.59:0/1648295955 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f926c00bc30 con 0x7f92740a5f40 2023-09-30T02:10:10.393 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.392+0000 7f9272ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:10:10.397 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.392+0000 7f9272ffd700 1 -- 172.21.15.59:0/1648295955 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f926c012430 con 0x7f92740a5f40 2023-09-30T02:10:10.398 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.392+0000 7f927a7de700 5 monclient: authenticate success, global_id 4625 2023-09-30T02:10:10.398 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.392+0000 7f927a7de700 10 monclient: get_monmap_and_config success 2023-09-30T02:10:10.398 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.392+0000 7f9272ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:10:10.398 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.392+0000 7f927a7de700 4 set_mon_vals no callback set 2023-09-30T02:10:10.398 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.392+0000 7f9272ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:10:10.398 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.392+0000 7f9272ffd700 10 monclient: dump: 2023-09-30T02:10:10.398 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:10:10.398 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:10.398 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:10.398 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:10.398 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:10:10.398 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:10.399 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:10:10.399 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:10:10.399 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:10:10.399 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:10.402 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.393+0000 7f927a7de700 10 monclient: shutdown 2023-09-30T02:10:10.402 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.393+0000 7f927a7de700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:10:10.402 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.393+0000 7f927a7de700 1 -- 172.21.15.59:0/1648295955 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f92740a5f40 msgr2=0x7f927413af30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:10.403 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.393+0000 7f927a7de700 1 --2- 172.21.15.59:0/1648295955 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f92740a5f40 0x7f927413af30 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f926c009e60 tx=0x7f926c009780 comp rx=0 tx=0).stop 2023-09-30T02:10:10.403 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.393+0000 7f927a7de700 1 -- 172.21.15.59:0/1648295955 shutdown_connections 2023-09-30T02:10:10.403 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.393+0000 7f927a7de700 1 --2- 172.21.15.59:0/1648295955 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f92740a5f40 0x7f927413af30 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:10.403 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.393+0000 7f927a7de700 1 --2- 172.21.15.59:0/1648295955 >> v2:172.21.15.59:3300/0 conn(0x7f9274142290 0x7f92740b1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:10.403 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.393+0000 7f927a7de700 1 -- 172.21.15.59:0/1648295955 >> 172.21.15.59:0/1648295955 conn(0x7f92740631f0 msgr2=0x7f92740635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:10:10.403 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.393+0000 7f927a7de700 1 -- 172.21.15.59:0/1648295955 shutdown_connections 2023-09-30T02:10:10.403 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.393+0000 7f927a7de700 1 -- 172.21.15.59:0/1648295955 wait complete. 2023-09-30T02:10:10.403 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.394+0000 7f927a7de700 10 monclient: build_initial_monmap 2023-09-30T02:10:10.403 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.394+0000 7f927a7de700 10 monclient: monmap: 2023-09-30T02:10:10.403 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:10:10.403 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:10:10.404 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:10:10.404 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:10:10.404 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:10:10.404 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:10:10.404 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:10:10.404 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:10:10.404 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:10:10.404 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:10.404 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.395+0000 7f927a7de700 1 librados: starting msgr at 2023-09-30T02:10:10.404 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.395+0000 7f927a7de700 1 librados: starting objecter 2023-09-30T02:10:10.404 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.395+0000 7f927a7de700 1 Processor -- start 2023-09-30T02:10:10.404 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.395+0000 7f927a7de700 1 -- start start 2023-09-30T02:10:10.404 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.395+0000 7f927a7de700 1 librados: setting wanted keys 2023-09-30T02:10:10.405 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.395+0000 7f927a7de700 1 librados: calling monclient init 2023-09-30T02:10:10.405 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.395+0000 7f927a7de700 10 monclient: init 2023-09-30T02:10:10.405 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.396+0000 7f927a7de700 10 monclient: _reopen_session rank -1 2023-09-30T02:10:10.405 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.396+0000 7f927a7de700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T02:10:10.405 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.396+0000 7f927a7de700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f92740a5b70 0x7f9274144fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:10.405 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.396+0000 7f927a7de700 10 monclient(hunting): picked mon.noname-b con 0x7f92740a5b70 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:10:10.405 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.396+0000 7f927a7de700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f92740a5f40 0x7f9274151580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:10.405 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.396+0000 7f927a7de700 10 monclient(hunting): picked mon.noname-c con 0x7f92740a5f40 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:10:10.405 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.396+0000 7f927a7de700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9274142290 0x7f9274151ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:10.405 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.396+0000 7f927a7de700 10 monclient(hunting): picked mon.noname-a con 0x7f9274142290 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:10:10.405 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.396+0000 7f927a7de700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:10.405 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.396+0000 7f927a7de700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f927413eb80 con 0x7f9274142290 2023-09-30T02:10:10.406 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.396+0000 7f927a7de700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:10.406 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.396+0000 7f927a7de700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92740ae730 con 0x7f92740a5b70 2023-09-30T02:10:10.406 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.396+0000 7f927a7de700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:10.406 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.396+0000 7f927a7de700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f927413f010 con 0x7f92740a5f40 2023-09-30T02:10:10.406 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.396+0000 7f927a7de700 10 monclient(hunting): _renew_subs 2023-09-30T02:10:10.406 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.396+0000 7f927a7de700 10 monclient(hunting): authenticate will time out at 2055.698340s 2023-09-30T02:10:10.406 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.397+0000 7f92737fe700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f92740a5f40 0x7f9274151580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:10.406 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.397+0000 7f92737fe700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f92740a5f40 0x7f9274151580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:57414/0 (socket says 172.21.15.59:57414) 2023-09-30T02:10:10.406 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.397+0000 7f92737fe700 1 -- 172.21.15.59:0/3900829234 learned_addr learned my addr 172.21.15.59:0/3900829234 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:10:10.406 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.397+0000 7f92737fe700 10 monclient(hunting): get_auth_request con 0x7f92740a5f40 auth_method 0 2023-09-30T02:10:10.406 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.397+0000 7f92737fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:10:10.406 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.397+0000 7f92737fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:10.407 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.397+0000 7f92737fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:10.407 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.398+0000 7f92737fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:10:10.407 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.398+0000 7f92737fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:10:10.407 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.398+0000 7f92737fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:10:10.407 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.398+0000 7f9278d7b700 1 --2- 172.21.15.59:0/3900829234 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9274142290 0x7f9274151ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:10.407 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.399+0000 7f9278d7b700 10 monclient(hunting): get_auth_request con 0x7f9274142290 auth_method 0 2023-09-30T02:10:10.407 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.399+0000 7f9278d7b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:10:10.407 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.399+0000 7f9278d7b700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:10.407 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.399+0000 7f9278d7b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:10.407 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.401+0000 7f9278d7b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:10:10.407 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.401+0000 7f9278d7b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:10:10.407 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.401+0000 7f9278d7b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:10:10.408 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.402+0000 7f92737fe700 10 monclient(hunting): handle_auth_done global_id 4628 payload 824 2023-09-30T02:10:10.408 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.402+0000 7f92737fe700 1 -- 172.21.15.59:0/3900829234 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f92740a5b70 msgr2=0x7f9274144fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:10.408 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.402+0000 7f92737fe700 1 --2- 172.21.15.59:0/3900829234 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f92740a5b70 0x7f9274144fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:10.408 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.402+0000 7f92737fe700 1 -- 172.21.15.59:0/3900829234 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9274142290 msgr2=0x7f9274151ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:10.408 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.402+0000 7f92737fe700 1 --2- 172.21.15.59:0/3900829234 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9274142290 0x7f9274151ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:10.408 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.402+0000 7f92737fe700 10 monclient: _finish_hunting 0 2023-09-30T02:10:10.408 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.402+0000 7f92737fe700 1 monclient: found mon.noname-c 2023-09-30T02:10:10.408 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.402+0000 7f92737fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:10:10.408 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.402+0000 7f92737fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.59:3301/0 2023-09-30T02:10:10.408 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.402+0000 7f92737fe700 1 -- 172.21.15.59:0/3900829234 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f926c009a20 con 0x7f92740a5f40 2023-09-30T02:10:10.409 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.404+0000 7f92737fe700 1 --2- 172.21.15.59:0/3900829234 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f92740a5f40 0x7f9274151580 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f926800b810 tx=0x7f92680091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:10:10.409 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.404+0000 7f925bfff700 10 client.?.objecter ms_handle_connect 0x7f92740a5f40 2023-09-30T02:10:10.409 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.404+0000 7f925bfff700 10 client.?.objecter resend_mon_ops 2023-09-30T02:10:10.410 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.410+0000 7f925bfff700 1 -- 172.21.15.59:0/3900829234 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f926800f510 con 0x7f92740a5f40 2023-09-30T02:10:10.410 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.410+0000 7f925bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:10:10.410 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.410+0000 7f925bfff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:10:10.410 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.410+0000 7f925bfff700 10 monclient: dump: 2023-09-30T02:10:10.410 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:10:10.410 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:10.410 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:10.410 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:10.410 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:10:10.411 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:10.411 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:10:10.411 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:10:10.411 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:10:10.411 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:10.411 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.410+0000 7f925bfff700 10 monclient: _finish_auth 0 2023-09-30T02:10:10.411 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.410+0000 7f925bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:10:10.411 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.410+0000 7f927a7de700 5 monclient: authenticate success, global_id 4628 2023-09-30T02:10:10.411 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.410+0000 7f927a7de700 10 monclient: _renew_subs 2023-09-30T02:10:10.411 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.410+0000 7f927a7de700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:10:10.411 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.410+0000 7f927a7de700 1 -- 172.21.15.59:0/3900829234 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f927413d1d0 con 0x7f92740a5f40 2023-09-30T02:10:10.411 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.410+0000 7f927a7de700 10 client.4628.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:10:10.411 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.410+0000 7f927a7de700 10 monclient: _renew_subs 2023-09-30T02:10:10.412 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.410+0000 7f927a7de700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:10:10.412 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.410+0000 7f927a7de700 1 -- 172.21.15.59:0/3900829234 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f927413e6a0 con 0x7f92740a5f40 2023-09-30T02:10:10.412 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.410+0000 7f927a7de700 1 librados: init done 2023-09-30T02:10:10.412 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.411+0000 7f927a7de700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:10:10.412 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.411+0000 7f927a7de700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:10:10.412 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.411+0000 7f927a7de700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:10:10.413 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.411+0000 7f927a7de700 1 -- 172.21.15.59:0/3900829234 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f927413e8f0 con 0x7f92740a5f40 2023-09-30T02:10:10.759 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.760+0000 7f925bfff700 1 -- 172.21.15.59:0/3900829234 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f926800f880 con 0x7f92740a5f40 2023-09-30T02:10:10.760 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.760+0000 7f925bfff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:10:10.760 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.760+0000 7f925bfff700 1 -- 172.21.15.59:0/3900829234 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f926800fa10 con 0x7f92740a5f40 2023-09-30T02:10:10.760 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.760+0000 7f925bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:10:10.760 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.760+0000 7f925bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:10:10.760 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.760+0000 7f925bfff700 10 monclient: dump: 2023-09-30T02:10:10.760 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:10:10.760 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:10.760 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:10.761 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:10.761 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:10:10.761 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:10.761 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:10:10.761 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:10:10.761 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:10:10.761 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:10.761 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.760+0000 7f92727fc700 4 set_mon_vals no callback set 2023-09-30T02:10:10.777 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.778+0000 7f925bfff700 1 -- 172.21.15.59:0/3900829234 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f926800fbb0 con 0x7f92740a5f40 2023-09-30T02:10:10.778 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.778+0000 7f925bfff700 1 --2- 172.21.15.59:0/3900829234 >> v2:172.21.15.110:6800/115870 conn(0x7f9260068890 0x7f926006ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:10.778 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.778+0000 7f925bfff700 1 -- 172.21.15.59:0/3900829234 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f9268084320 con 0x7f92740a5f40 2023-09-30T02:10:10.778 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.778+0000 7f925bfff700 10 client.4628.objecter ms_dispatch 0x7f9274001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:10:10.779 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.778+0000 7f925bfff700 3 client.4628.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:10:10.779 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.778+0000 7f925bfff700 3 client.4628.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:10:10.779 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.778+0000 7f925bfff700 20 client.4628.objecter dump_active .. 0 homeless 2023-09-30T02:10:10.779 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.779+0000 7f9273fff700 1 --2- 172.21.15.59:0/3900829234 >> v2:172.21.15.110:6800/115870 conn(0x7f9260068890 0x7f926006ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:10.779 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.778+0000 7f925bfff700 1 -- 172.21.15.59:0/3900829234 <== mon.2 v2:172.21.15.59:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9268084770 con 0x7f92740a5f40 2023-09-30T02:10:10.779 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.779+0000 7f925bfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:10:10.779 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.779+0000 7f925bfff700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:10:10.779 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.779+0000 7f9273fff700 10 monclient: get_auth_request con 0x7f9260068890 auth_method 0 2023-09-30T02:10:10.779 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.779+0000 7f9273fff700 1 --2- 172.21.15.59:0/3900829234 >> v2:172.21.15.110:6800/115870 conn(0x7f9260068890 0x7f926006ad40 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f92640016a0 tx=0x7f9264007b30 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:10:10.779 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:10.779+0000 7f925bfff700 10 client.4628.objecter ms_handle_connect 0x7f9260068890 2023-09-30T02:10:11.064 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:11.065+0000 7f927a7de700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:10:11.064 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:11.065+0000 7f927a7de700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:10:11.064 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:11.065+0000 7f927a7de700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:10:11.065 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:11.065+0000 7f927a7de700 1 -- 172.21.15.59:0/3900829234 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f927413ecf0 con 0x7f92740a5f40 2023-09-30T02:10:11.070 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:11.071+0000 7f925bfff700 1 -- 172.21.15.59:0/3900829234 <== mon.2 v2:172.21.15.59:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f9268051c20 con 0x7f92740a5f40 2023-09-30T02:10:11.070 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:11.071+0000 7f925bfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:10:11.070 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:11.071+0000 7f925bfff700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:10:11.071 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:10:11.071 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:10:11.073 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:11.073+0000 7f927a7de700 10 librados: watch_flush enter 2023-09-30T02:10:11.073 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:11.073+0000 7f927a7de700 10 librados: watch_flush exit 2023-09-30T02:10:11.073 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:11.073+0000 7f927a7de700 20 client.4628.objecter shutdown clearing up homeless session... 2023-09-30T02:10:11.073 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:11.073+0000 7f927a7de700 10 client.4628.objecter successfully canceled tick 2023-09-30T02:10:11.073 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:11.073+0000 7f927a7de700 1 -- 172.21.15.59:0/3900829234 >> v2:172.21.15.110:6800/115870 conn(0x7f9260068890 msgr2=0x7f926006ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:11.073 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:11.073+0000 7f927a7de700 1 --2- 172.21.15.59:0/3900829234 >> v2:172.21.15.110:6800/115870 conn(0x7f9260068890 0x7f926006ad40 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f92640016a0 tx=0x7f9264007b30 comp rx=0 tx=0).stop 2023-09-30T02:10:11.074 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:11.073+0000 7f927a7de700 10 monclient: shutdown 2023-09-30T02:10:11.074 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:11.073+0000 7f927a7de700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:10:11.074 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:11.073+0000 7f927a7de700 1 -- 172.21.15.59:0/3900829234 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f92740a5f40 msgr2=0x7f9274151580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:11.074 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:11.073+0000 7f927a7de700 1 --2- 172.21.15.59:0/3900829234 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f92740a5f40 0x7f9274151580 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f926800b810 tx=0x7f92680091d0 comp rx=0 tx=0).stop 2023-09-30T02:10:11.074 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:11.073+0000 7f927a7de700 1 -- 172.21.15.59:0/3900829234 shutdown_connections 2023-09-30T02:10:11.074 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:11.073+0000 7f927a7de700 1 --2- 172.21.15.59:0/3900829234 >> v2:172.21.15.110:6800/115870 conn(0x7f9260068890 0x7f926006ad40 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:11.074 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:11.073+0000 7f927a7de700 1 --2- 172.21.15.59:0/3900829234 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f92740a5b70 0x7f9274144fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:11.074 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:11.073+0000 7f927a7de700 1 --2- 172.21.15.59:0/3900829234 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9274142290 0x7f9274151ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:11.074 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:11.073+0000 7f927a7de700 1 --2- 172.21.15.59:0/3900829234 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f92740a5f40 0x7f9274151580 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:11.074 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:11.073+0000 7f927a7de700 1 -- 172.21.15.59:0/3900829234 >> 172.21.15.59:0/3900829234 conn(0x7f92740631f0 msgr2=0x7f92740635c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:10:11.074 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:11.073+0000 7f927a7de700 1 -- 172.21.15.59:0/3900829234 shutdown_connections 2023-09-30T02:10:11.074 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:11.073+0000 7f927a7de700 1 -- 172.21.15.59:0/3900829234 wait complete. 2023-09-30T02:10:11.075 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:11.073+0000 7f927a7de700 1 librados: shutdown 2023-09-30T02:10:14.089 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:10:14.215 INFO:tasks.ceph.mon.a.smithi059.stderr:==00:00:05:24.080 116256== Warning: unimplemented fcntl command: 1036 2023-09-30T02:10:14.220 INFO:tasks.ceph.mon.a.smithi059.stderr:==00:00:05:24.085 116256== Warning: unimplemented fcntl command: 1036 2023-09-30T02:10:14.228 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.228+0000 7f9e71a9e700 10 monclient: get_monmap_and_config 2023-09-30T02:10:14.228 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.228+0000 7f9e71a9e700 10 monclient: build_initial_monmap 2023-09-30T02:10:14.228 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.228+0000 7f9e71a9e700 10 monclient: monmap: 2023-09-30T02:10:14.228 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:10:14.228 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:14.229 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:10:14.229160+0000 2023-09-30T02:10:14.229 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:10:14.229160+0000 2023-09-30T02:10:14.229 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:10:14.229 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:14.229 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:10:14.229 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:10:14.229 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:10:14.229 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:10:14.229 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:10:14.229 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:14.230 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.229+0000 7f9e71a9e700 1 Processor -- start 2023-09-30T02:10:14.230 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.229+0000 7f9e71a9e700 1 -- start start 2023-09-30T02:10:14.231 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.229+0000 7f9e71a9e700 10 monclient: init 2023-09-30T02:10:14.231 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.230+0000 7f9e71a9e700 10 monclient: _reopen_session rank -1 2023-09-30T02:10:14.231 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.230+0000 7f9e71a9e700 10 monclient: _add_conns ranks=[4,0,1,2,3] 2023-09-30T02:10:14.231 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.230+0000 7f9e71a9e700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f9e6c0a5b70 0x7f9e6c0b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:14.231 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.230+0000 7f9e71a9e700 10 monclient(hunting): picked mon.noname-c con 0x7f9e6c0a5b70 addr v2:172.21.15.110:3300/0 2023-09-30T02:10:14.231 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.230+0000 7f9e71a9e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9e6c0a5f40 addr v1:172.21.15.59:6789/0 2023-09-30T02:10:14.231 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.230+0000 7f9e71a9e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9e6c142290 addr v1:172.21.15.110:6789/0 2023-09-30T02:10:14.231 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.230+0000 7f9e71a9e700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e6c0a75a0 con 0x7f9e6c0a5f40 2023-09-30T02:10:14.231 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.230+0000 7f9e71a9e700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e6c0af3f0 con 0x7f9e6c142290 2023-09-30T02:10:14.231 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.230+0000 7f9e71a9e700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:14.231 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.230+0000 7f9e71a9e700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e6c0a63e0 con 0x7f9e6c0a5b70 2023-09-30T02:10:14.231 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.230+0000 7f9e71a9e700 10 monclient(hunting): _renew_subs 2023-09-30T02:10:14.232 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.230+0000 7f9e71a9e700 10 monclient(hunting): authenticate will time out at 2059.532457s 2023-09-30T02:10:14.234 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.234+0000 7f9e6affd700 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f9e6c0a5f40 0x7f9e6c141b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:46858/0 (socket says 172.21.15.59:46858) 2023-09-30T02:10:14.234 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.234+0000 7f9e6affd700 1 -- 172.21.15.59:0/1482539453 learned_addr learned my addr 172.21.15.59:0/1482539453 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:10:14.248 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.249+0000 7f9e6a7fc700 1 -- 172.21.15.59:0/1482539453 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 661151672 0 0) 0x7f9e6c0a75a0 con 0x7f9e6c0a5f40 2023-09-30T02:10:14.249 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.249+0000 7f9e6a7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:14.249 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.249+0000 7f9e6a7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:14.249 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.249+0000 7f9e6a7fc700 10 monclient(hunting): my global_id is 4698 2023-09-30T02:10:14.249 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.249+0000 7f9e6a7fc700 1 -- 172.21.15.59:0/1482539453 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e54003570 con 0x7f9e6c0a5f40 2023-09-30T02:10:14.257 INFO:tasks.ceph.mon.a.smithi059.stderr:==00:00:05:24.123 116256== Warning: unimplemented fcntl command: 1036 2023-09-30T02:10:14.259 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.258+0000 7f9e6a7fc700 1 -- 172.21.15.59:0/1482539453 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2441405875 0 0) 0x7f9e54003570 con 0x7f9e6c0a5f40 2023-09-30T02:10:14.259 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.258+0000 7f9e6a7fc700 1 -- 172.21.15.59:0/1482539453 >> v2:172.21.15.110:3300/0 conn(0x7f9e6c0a5b70 msgr2=0x7f9e6c0b1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:10:14.259 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.258+0000 7f9e6a7fc700 1 --2- 172.21.15.59:0/1482539453 >> v2:172.21.15.110:3300/0 conn(0x7f9e6c0a5b70 0x7f9e6c0b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:14.259 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.258+0000 7f9e6a7fc700 1 -- 172.21.15.59:0/1482539453 >> v1:172.21.15.110:6789/0 conn(0x7f9e6c142290 legacy=0x7f9e6c13af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:10:14.260 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.258+0000 7f9e6a7fc700 10 monclient: _finish_hunting 0 2023-09-30T02:10:14.260 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.258+0000 7f9e6a7fc700 1 monclient: found mon.noname-a-legacy 2023-09-30T02:10:14.260 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.258+0000 7f9e6a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:10:14.260 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.258+0000 7f9e6a7fc700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.59:6789/0 2023-09-30T02:10:14.260 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.258+0000 7f9e6a7fc700 1 -- 172.21.15.59:0/1482539453 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e6c13b640 con 0x7f9e6c0a5f40 2023-09-30T02:10:14.260 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.258+0000 7f9e6a7fc700 10 monclient: _finish_auth 0 2023-09-30T02:10:14.260 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.258+0000 7f9e6a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:10:14.260 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.258+0000 7f9e6a7fc700 1 -- 172.21.15.59:0/1482539453 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2174283827 0 0) 0x7f9e60001470 con 0x7f9e6c0a5f40 2023-09-30T02:10:14.260 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.258+0000 7f9e71a9e700 5 monclient: authenticate success, global_id 4698 2023-09-30T02:10:14.260 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.258+0000 7f9e71a9e700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:10:14.260 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.258+0000 7f9e6a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:10:14.260 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.258+0000 7f9e6a7fc700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-09-30T02:10:14.261 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.258+0000 7f9e6a7fc700 10 monclient: dump: 2023-09-30T02:10:14.261 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:10:14.261 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:14.261 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:14.261 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:14.261 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:10:14.261 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:14.261 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:10:14.261 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:10:14.261 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:10:14.261 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:14.261 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.258+0000 7f9e6a7fc700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] but i'm connected to v1:172.21.15.59:6789/0, reconnecting 2023-09-30T02:10:14.261 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.258+0000 7f9e6a7fc700 10 monclient: _reopen_session rank -1 2023-09-30T02:10:14.262 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.258+0000 7f9e6a7fc700 1 -- 172.21.15.59:0/1482539453 >> v1:172.21.15.59:6789/0 conn(0x7f9e6c0a5f40 legacy=0x7f9e6c141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:14.262 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.258+0000 7f9e6a7fc700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T02:10:14.262 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.258+0000 7f9e6a7fc700 1 --2- 172.21.15.59:0/1482539453 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e54003b30 0x7f9e54003f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:14.262 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.258+0000 7f9e6a7fc700 10 monclient(hunting): picked mon.b con 0x7f9e54003b30 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:10:14.262 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.258+0000 7f9e6a7fc700 1 --2- 172.21.15.59:0/1482539453 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e54004810 0x7f9e54004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:14.262 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.258+0000 7f9e6a7fc700 10 monclient(hunting): picked mon.a con 0x7f9e54004810 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:10:14.262 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.258+0000 7f9e6a7fc700 1 --2- 172.21.15.59:0/1482539453 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9e54005410 0x7f9e5400bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:14.262 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.258+0000 7f9e6a7fc700 10 monclient(hunting): picked mon.c con 0x7f9e54005410 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:10:14.262 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.258+0000 7f9e6a7fc700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:14.262 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.258+0000 7f9e6a7fc700 1 -- 172.21.15.59:0/1482539453 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e6c0a63e0 con 0x7f9e54004810 2023-09-30T02:10:14.262 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.258+0000 7f9e6a7fc700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:14.263 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.258+0000 7f9e6a7fc700 1 -- 172.21.15.59:0/1482539453 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e5400c3a0 con 0x7f9e54003b30 2023-09-30T02:10:14.263 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.258+0000 7f9e6a7fc700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:14.263 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.258+0000 7f9e6a7fc700 1 -- 172.21.15.59:0/1482539453 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e5400c560 con 0x7f9e54005410 2023-09-30T02:10:14.263 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.258+0000 7f9e6a7fc700 10 monclient(hunting): _renew_subs 2023-09-30T02:10:14.263 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.258+0000 7f9e6a7fc700 10 monclient(hunting): _finish_auth 0 2023-09-30T02:10:14.263 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.258+0000 7f9e71a9e700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-09-30T02:10:14.273 INFO:tasks.ceph.mon.b.smithi110.stderr:==00:00:05:24.120 115869== Warning: unimplemented fcntl command: 1036 2023-09-30T02:10:14.275 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.276+0000 7f9e6b7fe700 1 --2- 172.21.15.59:0/1482539453 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e54003b30 0x7f9e54003f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:14.276 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.276+0000 7f9e6b7fe700 10 monclient(hunting): get_auth_request con 0x7f9e54003b30 auth_method 0 2023-09-30T02:10:14.276 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.276+0000 7f9e6b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:10:14.276 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.276+0000 7f9e6b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:14.276 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.276+0000 7f9e6b7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T02:10:14.277 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.278+0000 7f9e6b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:10:14.277 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.278+0000 7f9e6b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:10:14.277 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.278+0000 7f9e6b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T02:10:14.281 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.282+0000 7f9e6b7fe700 10 monclient(hunting): handle_auth_done global_id 4698 payload 293 2023-09-30T02:10:14.281 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.282+0000 7f9e6b7fe700 1 -- 172.21.15.59:0/1482539453 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9e54005410 msgr2=0x7f9e5400bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:10:14.282 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.282+0000 7f9e6b7fe700 1 --2- 172.21.15.59:0/1482539453 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9e54005410 0x7f9e5400bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:14.282 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.282+0000 7f9e6b7fe700 1 -- 172.21.15.59:0/1482539453 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e54004810 msgr2=0x7f9e54004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:10:14.282 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.282+0000 7f9e6b7fe700 1 --2- 172.21.15.59:0/1482539453 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e54004810 0x7f9e54004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:14.282 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.282+0000 7f9e6b7fe700 10 monclient: _finish_hunting 0 2023-09-30T02:10:14.282 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.282+0000 7f9e6b7fe700 1 monclient: found mon.b 2023-09-30T02:10:14.282 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.282+0000 7f9e6b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:10:14.282 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.282+0000 7f9e6b7fe700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:10:14.282 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.282+0000 7f9e6b7fe700 1 -- 172.21.15.59:0/1482539453 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e54003570 con 0x7f9e54003b30 2023-09-30T02:10:14.282 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.282+0000 7f9e6b7fe700 10 monclient: _finish_auth 0 2023-09-30T02:10:14.282 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.282+0000 7f9e6b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:10:14.282 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.282+0000 7f9e6bfff700 1 -- 172.21.15.59:0/1482539453 reap_dead start 2023-09-30T02:10:14.284 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.285+0000 7f9e6b7fe700 1 --2- 172.21.15.59:0/1482539453 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e54003b30 0x7f9e54003f20 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f9e5c0099c0 tx=0x7f9e5c009060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a78aca077382f5b7 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:10:14.285 INFO:tasks.ceph.mon.b.smithi110.stderr:==00:00:05:24.132 115869== Warning: unimplemented fcntl command: 1036 2023-09-30T02:10:14.296 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:05:24.161 116257== Warning: unimplemented fcntl command: 1036 2023-09-30T02:10:14.299 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:05:24.164 116257== Warning: unimplemented fcntl command: 1036 2023-09-30T02:10:14.301 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.301+0000 7f9e6a7fc700 1 -- 172.21.15.59:0/1482539453 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e5c011400 con 0x7f9e54003b30 2023-09-30T02:10:14.301 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.301+0000 7f9e6a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:10:14.301 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.301+0000 7f9e6a7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:10:14.301 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.301+0000 7f9e6a7fc700 10 monclient: dump: 2023-09-30T02:10:14.301 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:10:14.301 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:14.301 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:14.301 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:14.302 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:10:14.302 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:14.302 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:10:14.302 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:10:14.302 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:10:14.302 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:14.302 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.301+0000 7f9e6a7fc700 1 -- 172.21.15.59:0/1482539453 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e5c011560 con 0x7f9e54003b30 2023-09-30T02:10:14.302 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.301+0000 7f9e71a9e700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:10:14.302 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.301+0000 7f9e6a7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:10:14.302 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.301+0000 7f9e71a9e700 10 monclient: get_monmap_and_config success 2023-09-30T02:10:14.302 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.301+0000 7f9e6a7fc700 1 -- 172.21.15.59:0/1482539453 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e5c011920 con 0x7f9e54003b30 2023-09-30T02:10:14.302 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.301+0000 7f9e71a9e700 4 set_mon_vals no callback set 2023-09-30T02:10:14.303 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.301+0000 7f9e6a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:10:14.303 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.301+0000 7f9e6a7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:10:14.303 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.301+0000 7f9e6a7fc700 10 monclient: dump: 2023-09-30T02:10:14.303 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:10:14.303 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:14.303 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:14.303 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:14.303 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:10:14.303 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:14.303 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:10:14.303 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:10:14.303 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:10:14.304 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:14.305 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.302+0000 7f9e71a9e700 10 monclient: shutdown 2023-09-30T02:10:14.305 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.302+0000 7f9e71a9e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:10:14.305 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.302+0000 7f9e71a9e700 1 -- 172.21.15.59:0/1482539453 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e54003b30 msgr2=0x7f9e54003f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:14.306 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.302+0000 7f9e71a9e700 1 --2- 172.21.15.59:0/1482539453 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e54003b30 0x7f9e54003f20 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f9e5c0099c0 tx=0x7f9e5c009060 comp rx=0 tx=0).stop 2023-09-30T02:10:14.306 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.302+0000 7f9e71a9e700 1 -- 172.21.15.59:0/1482539453 shutdown_connections 2023-09-30T02:10:14.306 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.302+0000 7f9e71a9e700 1 --2- 172.21.15.59:0/1482539453 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e54003b30 0x7f9e54003f20 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:14.306 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.302+0000 7f9e71a9e700 1 -- 172.21.15.59:0/1482539453 >> 172.21.15.59:0/1482539453 conn(0x7f9e6c0631f0 msgr2=0x7f9e6c0635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:10:14.306 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.302+0000 7f9e71a9e700 1 -- 172.21.15.59:0/1482539453 shutdown_connections 2023-09-30T02:10:14.306 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.302+0000 7f9e71a9e700 1 -- 172.21.15.59:0/1482539453 wait complete. 2023-09-30T02:10:14.306 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.303+0000 7f9e71a9e700 10 monclient: build_initial_monmap 2023-09-30T02:10:14.306 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.303+0000 7f9e71a9e700 10 monclient: monmap: 2023-09-30T02:10:14.306 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:10:14.306 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:10:14.306 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:10:14.307 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:10:14.307 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:10:14.307 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:10:14.307 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:10:14.307 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:10:14.307 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:10:14.307 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:14.307 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.303+0000 7f9e71a9e700 1 librados: starting msgr at 2023-09-30T02:10:14.307 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.303+0000 7f9e71a9e700 1 librados: starting objecter 2023-09-30T02:10:14.307 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.303+0000 7f9e71a9e700 1 Processor -- start 2023-09-30T02:10:14.307 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.304+0000 7f9e71a9e700 1 -- start start 2023-09-30T02:10:14.308 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.304+0000 7f9e71a9e700 1 librados: setting wanted keys 2023-09-30T02:10:14.308 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.304+0000 7f9e71a9e700 1 librados: calling monclient init 2023-09-30T02:10:14.308 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.304+0000 7f9e71a9e700 10 monclient: init 2023-09-30T02:10:14.308 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.304+0000 7f9e71a9e700 10 monclient: _reopen_session rank -1 2023-09-30T02:10:14.308 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.304+0000 7f9e71a9e700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T02:10:14.308 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.304+0000 7f9e71a9e700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9e54003b30 0x7f9e6c13d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:14.308 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.304+0000 7f9e71a9e700 10 monclient(hunting): picked mon.noname-c con 0x7f9e54003b30 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:10:14.308 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.304+0000 7f9e71a9e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e6c13da60 0x7f9e6c14dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:14.308 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.304+0000 7f9e71a9e700 10 monclient(hunting): picked mon.noname-a con 0x7f9e6c13da60 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:10:14.308 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.304+0000 7f9e71a9e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e6c14e520 0x7f9e6c1510e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:14.308 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.304+0000 7f9e71a9e700 10 monclient(hunting): picked mon.noname-b con 0x7f9e6c14e520 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:10:14.308 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.304+0000 7f9e71a9e700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:14.309 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.304+0000 7f9e71a9e700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e6c13ded0 con 0x7f9e6c13da60 2023-09-30T02:10:14.309 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.304+0000 7f9e71a9e700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:14.309 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.304+0000 7f9e71a9e700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e6c13c0a0 con 0x7f9e6c14e520 2023-09-30T02:10:14.309 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.304+0000 7f9e71a9e700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:14.309 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.304+0000 7f9e71a9e700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e6c14db50 con 0x7f9e54003b30 2023-09-30T02:10:14.309 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.304+0000 7f9e71a9e700 10 monclient(hunting): _renew_subs 2023-09-30T02:10:14.309 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.304+0000 7f9e71a9e700 10 monclient(hunting): authenticate will time out at 2059.606603s 2023-09-30T02:10:14.312 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.313+0000 7f9e6b7fe700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9e54003b30 0x7f9e6c13d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:14.313 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.314+0000 7f9e6b7fe700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9e54003b30 0x7f9e6c13d520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:57422/0 (socket says 172.21.15.59:57422) 2023-09-30T02:10:14.313 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.314+0000 7f9e6b7fe700 1 -- 172.21.15.59:0/1587537549 learned_addr learned my addr 172.21.15.59:0/1587537549 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:10:14.313 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.314+0000 7f9e6b7fe700 10 monclient(hunting): get_auth_request con 0x7f9e54003b30 auth_method 0 2023-09-30T02:10:14.313 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.314+0000 7f9e6b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:10:14.313 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.314+0000 7f9e6b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:14.313 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.314+0000 7f9e6b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:14.314 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.315+0000 7f9e6bfff700 1 --2- 172.21.15.59:0/1587537549 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e6c14e520 0x7f9e6c1510e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:14.315 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.316+0000 7f9e6bfff700 10 monclient(hunting): get_auth_request con 0x7f9e6c14e520 auth_method 0 2023-09-30T02:10:14.315 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.316+0000 7f9e6bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:10:14.315 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.316+0000 7f9e6bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:14.315 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.316+0000 7f9e6bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:14.316 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.317+0000 7f9e6affd700 1 --2- 172.21.15.59:0/1587537549 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e6c13da60 0x7f9e6c14dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:14.317 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.317+0000 7f9e6bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:10:14.317 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.317+0000 7f9e6bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:10:14.317 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.317+0000 7f9e6bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:10:14.317 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.318+0000 7f9e6affd700 10 monclient(hunting): get_auth_request con 0x7f9e6c13da60 auth_method 0 2023-09-30T02:10:14.318 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.318+0000 7f9e6affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:10:14.318 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.318+0000 7f9e6affd700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:14.318 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.318+0000 7f9e6affd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:14.321 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.322+0000 7f9e6bfff700 10 monclient(hunting): handle_auth_done global_id 4732 payload 824 2023-09-30T02:10:14.321 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.322+0000 7f9e6bfff700 1 -- 172.21.15.59:0/1587537549 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9e54003b30 msgr2=0x7f9e6c13d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:14.322 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.322+0000 7f9e6bfff700 1 --2- 172.21.15.59:0/1587537549 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9e54003b30 0x7f9e6c13d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:14.322 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.322+0000 7f9e6bfff700 1 -- 172.21.15.59:0/1587537549 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e6c13da60 msgr2=0x7f9e6c14dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:14.322 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.322+0000 7f9e6bfff700 1 --2- 172.21.15.59:0/1587537549 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e6c13da60 0x7f9e6c14dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:14.322 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.322+0000 7f9e6bfff700 10 monclient: _finish_hunting 0 2023-09-30T02:10:14.322 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.322+0000 7f9e6bfff700 1 monclient: found mon.noname-b 2023-09-30T02:10:14.322 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.322+0000 7f9e6bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:10:14.322 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.322+0000 7f9e6bfff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:10:14.322 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.322+0000 7f9e6bfff700 1 -- 172.21.15.59:0/1587537549 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e6c13e030 con 0x7f9e6c14e520 2023-09-30T02:10:14.326 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.327+0000 7f9e6bfff700 1 --2- 172.21.15.59:0/1587537549 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e6c14e520 0x7f9e6c1510e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c0b1b60 tx=0x7f9e64001550 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:10:14.326 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.327+0000 7f9e537fe700 10 client.?.objecter ms_handle_connect 0x7f9e6c14e520 2023-09-30T02:10:14.326 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.327+0000 7f9e537fe700 10 client.?.objecter resend_mon_ops 2023-09-30T02:10:14.328 INFO:tasks.ceph.mon.b.smithi110.stderr:==00:00:05:24.175 115869== Warning: unimplemented fcntl command: 1036 2023-09-30T02:10:14.334 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:05:24.200 116257== Warning: unimplemented fcntl command: 1036 2023-09-30T02:10:14.409 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.408+0000 7f9e537fe700 1 -- 172.21.15.59:0/1587537549 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e64004d70 con 0x7f9e6c14e520 2023-09-30T02:10:14.409 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.408+0000 7f9e537fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:10:14.409 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.408+0000 7f9e537fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:10:14.409 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.408+0000 7f9e537fe700 10 monclient: dump: 2023-09-30T02:10:14.409 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:10:14.409 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:14.409 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:14.409 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:14.409 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:10:14.409 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:14.410 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:10:14.410 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:10:14.410 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:10:14.410 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:14.410 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.408+0000 7f9e537fe700 10 monclient: _finish_auth 0 2023-09-30T02:10:14.410 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.408+0000 7f9e537fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:10:14.410 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.408+0000 7f9e537fe700 1 -- 172.21.15.59:0/1587537549 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e64007190 con 0x7f9e6c14e520 2023-09-30T02:10:14.410 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.408+0000 7f9e71a9e700 5 monclient: authenticate success, global_id 4732 2023-09-30T02:10:14.410 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.408+0000 7f9e71a9e700 10 monclient: _renew_subs 2023-09-30T02:10:14.410 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.408+0000 7f9e71a9e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:10:14.411 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.408+0000 7f9e71a9e700 1 -- 172.21.15.59:0/1587537549 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e6c14e990 con 0x7f9e6c14e520 2023-09-30T02:10:14.411 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.408+0000 7f9e537fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:10:14.411 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.408+0000 7f9e537fe700 1 -- 172.21.15.59:0/1587537549 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e64007550 con 0x7f9e6c14e520 2023-09-30T02:10:14.411 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.408+0000 7f9e537fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:10:14.411 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.408+0000 7f9e537fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:10:14.411 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.408+0000 7f9e537fe700 10 monclient: dump: 2023-09-30T02:10:14.411 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:10:14.411 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:14.411 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:14.411 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:14.411 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:10:14.411 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:14.412 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:10:14.412 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:10:14.412 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:10:14.412 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:14.412 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.408+0000 7f9e69ffb700 4 set_mon_vals no callback set 2023-09-30T02:10:14.412 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.408+0000 7f9e71a9e700 10 client.4732.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:10:14.412 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.408+0000 7f9e71a9e700 10 monclient: _renew_subs 2023-09-30T02:10:14.412 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.408+0000 7f9e71a9e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:10:14.412 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.408+0000 7f9e71a9e700 1 -- 172.21.15.59:0/1587537549 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e6c14b680 con 0x7f9e6c14e520 2023-09-30T02:10:14.412 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.408+0000 7f9e71a9e700 1 librados: init done 2023-09-30T02:10:14.412 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.409+0000 7f9e71a9e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:10:14.413 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.409+0000 7f9e71a9e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:10:14.413 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.409+0000 7f9e71a9e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:10:14.413 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.409+0000 7f9e71a9e700 1 -- 172.21.15.59:0/1587537549 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e6c142030 con 0x7f9e6c14e520 2023-09-30T02:10:14.741 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.741+0000 7f9e537fe700 1 -- 172.21.15.59:0/1587537549 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f9e6401d3f0 con 0x7f9e6c14e520 2023-09-30T02:10:14.742 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.742+0000 7f9e537fe700 1 --2- 172.21.15.59:0/1587537549 >> v2:172.21.15.110:6800/115870 conn(0x7f9e58068960 0x7f9e5806ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:14.742 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.742+0000 7f9e537fe700 1 -- 172.21.15.59:0/1587537549 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f9e64083a60 con 0x7f9e6c14e520 2023-09-30T02:10:14.742 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.742+0000 7f9e537fe700 10 client.4732.objecter ms_dispatch 0x7f9e6c001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:10:14.742 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.742+0000 7f9e537fe700 3 client.4732.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:10:14.743 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.742+0000 7f9e537fe700 3 client.4732.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:10:14.743 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.742+0000 7f9e537fe700 20 client.4732.objecter dump_active .. 0 homeless 2023-09-30T02:10:14.743 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.742+0000 7f9e6b7fe700 1 --2- 172.21.15.59:0/1587537549 >> v2:172.21.15.110:6800/115870 conn(0x7f9e58068960 0x7f9e5806ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:14.743 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.742+0000 7f9e6b7fe700 10 monclient: get_auth_request con 0x7f9e58068960 auth_method 0 2023-09-30T02:10:14.743 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.743+0000 7f9e6b7fe700 1 --2- 172.21.15.59:0/1587537549 >> v2:172.21.15.110:6800/115870 conn(0x7f9e58068960 0x7f9e5806ae10 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f9e5c009780 tx=0x7f9e5c015000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:10:14.743 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.743+0000 7f9e537fe700 10 client.4732.objecter ms_handle_connect 0x7f9e58068960 2023-09-30T02:10:14.750 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.750+0000 7f9e537fe700 1 -- 172.21.15.59:0/1587537549 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9e64051380 con 0x7f9e6c14e520 2023-09-30T02:10:14.750 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.750+0000 7f9e537fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:10:14.750 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:14.750+0000 7f9e537fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:10:15.003 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:15.004+0000 7f9e71a9e700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:10:15.003 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:15.004+0000 7f9e71a9e700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:10:15.003 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:15.004+0000 7f9e71a9e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:10:15.003 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:15.004+0000 7f9e71a9e700 1 -- 172.21.15.59:0/1587537549 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f9e6c064a00 con 0x7f9e6c14e520 2023-09-30T02:10:15.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:15.009+0000 7f9e537fe700 1 -- 172.21.15.59:0/1587537549 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f9e640549a0 con 0x7f9e6c14e520 2023-09-30T02:10:15.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:15.009+0000 7f9e537fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:10:15.009 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:15.009+0000 7f9e537fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:10:15.009 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:10:15.009 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:10:15.011 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:15.011+0000 7f9e71a9e700 10 librados: watch_flush enter 2023-09-30T02:10:15.011 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:15.011+0000 7f9e71a9e700 10 librados: watch_flush exit 2023-09-30T02:10:15.011 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:15.011+0000 7f9e71a9e700 20 client.4732.objecter shutdown clearing up homeless session... 2023-09-30T02:10:15.011 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:15.011+0000 7f9e71a9e700 10 client.4732.objecter successfully canceled tick 2023-09-30T02:10:15.011 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:15.011+0000 7f9e71a9e700 1 -- 172.21.15.59:0/1587537549 >> v2:172.21.15.110:6800/115870 conn(0x7f9e58068960 msgr2=0x7f9e5806ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:15.011 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:15.011+0000 7f9e71a9e700 1 --2- 172.21.15.59:0/1587537549 >> v2:172.21.15.110:6800/115870 conn(0x7f9e58068960 0x7f9e5806ae10 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f9e5c009780 tx=0x7f9e5c015000 comp rx=0 tx=0).stop 2023-09-30T02:10:15.012 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:15.011+0000 7f9e71a9e700 10 monclient: shutdown 2023-09-30T02:10:15.012 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:15.011+0000 7f9e71a9e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:10:15.012 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:15.011+0000 7f9e71a9e700 1 -- 172.21.15.59:0/1587537549 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e6c14e520 msgr2=0x7f9e6c1510e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:15.012 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:15.011+0000 7f9e71a9e700 1 --2- 172.21.15.59:0/1587537549 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e6c14e520 0x7f9e6c1510e0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c0b1b60 tx=0x7f9e64001550 comp rx=0 tx=0).stop 2023-09-30T02:10:15.012 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:15.011+0000 7f9e71a9e700 1 -- 172.21.15.59:0/1587537549 shutdown_connections 2023-09-30T02:10:15.012 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:15.011+0000 7f9e71a9e700 1 --2- 172.21.15.59:0/1587537549 >> v2:172.21.15.110:6800/115870 conn(0x7f9e58068960 0x7f9e5806ae10 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:15.012 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:15.011+0000 7f9e71a9e700 1 --2- 172.21.15.59:0/1587537549 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f9e6c14e520 0x7f9e6c1510e0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:15.012 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:15.011+0000 7f9e71a9e700 1 --2- 172.21.15.59:0/1587537549 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f9e54003b30 0x7f9e6c13d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:15.012 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:15.011+0000 7f9e71a9e700 1 --2- 172.21.15.59:0/1587537549 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e6c13da60 0x7f9e6c14dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:15.012 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:15.011+0000 7f9e71a9e700 1 -- 172.21.15.59:0/1587537549 >> 172.21.15.59:0/1587537549 conn(0x7f9e6c0631f0 msgr2=0x7f9e6c13b830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:10:15.012 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:15.011+0000 7f9e71a9e700 1 -- 172.21.15.59:0/1587537549 shutdown_connections 2023-09-30T02:10:15.013 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:15.011+0000 7f9e71a9e700 1 -- 172.21.15.59:0/1587537549 wait complete. 2023-09-30T02:10:15.013 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:15.011+0000 7f9e71a9e700 1 librados: shutdown 2023-09-30T02:10:18.025 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:10:18.164 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.164+0000 7f1f1df0e700 10 monclient: get_monmap_and_config 2023-09-30T02:10:18.164 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.164+0000 7f1f1df0e700 10 monclient: build_initial_monmap 2023-09-30T02:10:18.164 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.164+0000 7f1f1df0e700 10 monclient: monmap: 2023-09-30T02:10:18.164 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:10:18.164 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:18.164 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:10:18.165089+0000 2023-09-30T02:10:18.164 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:10:18.165089+0000 2023-09-30T02:10:18.164 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:10:18.165 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:18.165 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:10:18.165 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:10:18.165 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:10:18.165 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:10:18.165 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:10:18.165 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:18.167 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.165+0000 7f1f1df0e700 1 Processor -- start 2023-09-30T02:10:18.167 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.165+0000 7f1f1df0e700 1 -- start start 2023-09-30T02:10:18.167 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.165+0000 7f1f1df0e700 10 monclient: init 2023-09-30T02:10:18.167 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.166+0000 7f1f1df0e700 10 monclient: _reopen_session rank -1 2023-09-30T02:10:18.167 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.166+0000 7f1f1df0e700 10 monclient: _add_conns ranks=[1,0,2,3,4] 2023-09-30T02:10:18.167 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.166+0000 7f1f1df0e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1f180a5b70 addr v1:172.21.15.110:6789/0 2023-09-30T02:10:18.167 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.166+0000 7f1f1df0e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1f18142290 addr v1:172.21.15.59:6789/0 2023-09-30T02:10:18.167 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.166+0000 7f1f1df0e700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1f180b1500 0x7f1f180a5f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:18.168 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.166+0000 7f1f1df0e700 10 monclient(hunting): picked mon.noname-b con 0x7f1f180b1500 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:10:18.168 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.166+0000 7f1f1df0e700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f180a75a0 con 0x7f1f18142290 2023-09-30T02:10:18.168 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.166+0000 7f1f1df0e700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f180af3f0 con 0x7f1f180a5b70 2023-09-30T02:10:18.168 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.166+0000 7f1f1df0e700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:18.168 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.166+0000 7f1f1df0e700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f180b1940 con 0x7f1f180b1500 2023-09-30T02:10:18.168 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.166+0000 7f1f1df0e700 10 monclient(hunting): _renew_subs 2023-09-30T02:10:18.168 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.166+0000 7f1f1df0e700 10 monclient(hunting): authenticate will time out at 2063.468449s 2023-09-30T02:10:18.168 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.167+0000 7f1f17fff700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1f180b1500 0x7f1f180a5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:18.168 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.167+0000 7f1f16ffd700 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f1f18142290 0x7f1f1813af30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:50072/0 (socket says 172.21.15.59:50072) 2023-09-30T02:10:18.168 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.167+0000 7f1f16ffd700 1 -- 172.21.15.59:0/495881401 learned_addr learned my addr 172.21.15.59:0/495881401 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:10:18.168 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.167+0000 7f1f17fff700 10 monclient(hunting): get_auth_request con 0x7f1f180b1500 auth_method 0 2023-09-30T02:10:18.169 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.167+0000 7f1f17fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:10:18.169 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.167+0000 7f1f17fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:18.169 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.167+0000 7f1f17fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:18.169 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.168+0000 7f1f17fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:10:18.170 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.168+0000 7f1f17fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:10:18.170 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.168+0000 7f1f17fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:10:18.170 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.170+0000 7f1f167fc700 1 -- 172.21.15.59:0/495881401 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 857342399 0 0) 0x7f1f180a75a0 con 0x7f1f18142290 2023-09-30T02:10:18.170 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.170+0000 7f1f167fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:18.170 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.170+0000 7f1f167fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:18.170 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.170+0000 7f1f167fc700 10 monclient(hunting): my global_id is 4704 2023-09-30T02:10:18.170 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.170+0000 7f1f167fc700 1 -- 172.21.15.59:0/495881401 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f04003570 con 0x7f1f18142290 2023-09-30T02:10:18.171 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.171+0000 7f1f167fc700 1 -- 172.21.15.59:0/495881401 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1081088410 0 0) 0x7f1f180af3f0 con 0x7f1f180a5b70 2023-09-30T02:10:18.171 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.171+0000 7f1f167fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:18.171 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.171+0000 7f1f167fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:18.171 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.171+0000 7f1f167fc700 10 monclient(hunting): my global_id is 4735 2023-09-30T02:10:18.171 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.171+0000 7f1f167fc700 1 -- 172.21.15.59:0/495881401 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f180a75a0 con 0x7f1f180a5b70 2023-09-30T02:10:18.172 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.171+0000 7f1f17fff700 10 monclient(hunting): handle_auth_done global_id 4634 payload 274 2023-09-30T02:10:18.172 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.171+0000 7f1f17fff700 1 -- 172.21.15.59:0/495881401 >> v1:172.21.15.110:6789/0 conn(0x7f1f180a5b70 legacy=0x7f1f18141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:18.172 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.171+0000 7f1f17fff700 1 -- 172.21.15.59:0/495881401 >> v1:172.21.15.59:6789/0 conn(0x7f1f18142290 legacy=0x7f1f1813af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:18.172 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.171+0000 7f1f17fff700 10 monclient: _finish_hunting 0 2023-09-30T02:10:18.172 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.171+0000 7f1f17fff700 1 monclient: found mon.noname-b 2023-09-30T02:10:18.172 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.171+0000 7f1f17fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:10:18.172 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.171+0000 7f1f17fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.59:3301/0 2023-09-30T02:10:18.172 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.171+0000 7f1f17fff700 1 -- 172.21.15.59:0/495881401 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f1813b640 con 0x7f1f180b1500 2023-09-30T02:10:18.173 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.174+0000 7f1f17fff700 1 --2- 172.21.15.59:0/495881401 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1f180b1500 0x7f1f180a5f40 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f1f10009e90 tx=0x7f1f10009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=822f1abc3c4e3f6c server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:10:18.179 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.179+0000 7f1f167fc700 1 -- 172.21.15.59:0/495881401 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f1000e070 con 0x7f1f180b1500 2023-09-30T02:10:18.179 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.179+0000 7f1f167fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:10:18.179 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.179+0000 7f1f167fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:10:18.179 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.179+0000 7f1f167fc700 10 monclient: dump: 2023-09-30T02:10:18.179 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:10:18.179 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:18.180 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:18.180 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:18.180 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:10:18.180 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:18.180 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:10:18.180 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:10:18.180 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:10:18.180 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:18.180 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.179+0000 7f1f167fc700 10 monclient: _finish_auth 0 2023-09-30T02:10:18.180 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.179+0000 7f1f167fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:10:18.181 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.179+0000 7f1f167fc700 1 -- 172.21.15.59:0/495881401 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f1000bb80 con 0x7f1f180b1500 2023-09-30T02:10:18.181 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.179+0000 7f1f1df0e700 5 monclient: authenticate success, global_id 4634 2023-09-30T02:10:18.181 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.179+0000 7f1f1df0e700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:10:18.181 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.179+0000 7f1f167fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:10:18.181 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.179+0000 7f1f167fc700 1 -- 172.21.15.59:0/495881401 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f10012430 con 0x7f1f180b1500 2023-09-30T02:10:18.181 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.179+0000 7f1f1df0e700 10 monclient: get_monmap_and_config success 2023-09-30T02:10:18.181 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.179+0000 7f1f167fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:10:18.181 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.179+0000 7f1f1df0e700 4 set_mon_vals no callback set 2023-09-30T02:10:18.181 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.179+0000 7f1f167fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:10:18.181 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.179+0000 7f1f167fc700 10 monclient: dump: 2023-09-30T02:10:18.181 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:10:18.182 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:18.182 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:18.182 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:18.182 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:10:18.182 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:18.182 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:10:18.182 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:10:18.182 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:10:18.182 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:18.182 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.179+0000 7f1f1df0e700 10 monclient: shutdown 2023-09-30T02:10:18.182 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.179+0000 7f1f1df0e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:10:18.182 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.179+0000 7f1f1df0e700 1 -- 172.21.15.59:0/495881401 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1f180b1500 msgr2=0x7f1f180a5f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:18.183 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.179+0000 7f1f1df0e700 1 --2- 172.21.15.59:0/495881401 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1f180b1500 0x7f1f180a5f40 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f1f10009e90 tx=0x7f1f10009780 comp rx=0 tx=0).stop 2023-09-30T02:10:18.183 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.179+0000 7f1f1df0e700 1 -- 172.21.15.59:0/495881401 shutdown_connections 2023-09-30T02:10:18.183 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.179+0000 7f1f1df0e700 1 --2- 172.21.15.59:0/495881401 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1f180b1500 0x7f1f180a5f40 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:18.183 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.180+0000 7f1f1df0e700 1 -- 172.21.15.59:0/495881401 >> 172.21.15.59:0/495881401 conn(0x7f1f180631f0 msgr2=0x7f1f180635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:10:18.183 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.180+0000 7f1f1df0e700 1 -- 172.21.15.59:0/495881401 shutdown_connections 2023-09-30T02:10:18.183 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.180+0000 7f1f1df0e700 1 -- 172.21.15.59:0/495881401 wait complete. 2023-09-30T02:10:18.186 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.180+0000 7f1f1df0e700 10 monclient: build_initial_monmap 2023-09-30T02:10:18.186 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.180+0000 7f1f1df0e700 10 monclient: monmap: 2023-09-30T02:10:18.186 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:10:18.186 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:10:18.186 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:10:18.186 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:10:18.186 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:10:18.186 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:10:18.186 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:10:18.187 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:10:18.187 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:10:18.187 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:18.187 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.180+0000 7f1f1df0e700 1 librados: starting msgr at 2023-09-30T02:10:18.187 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.180+0000 7f1f1df0e700 1 librados: starting objecter 2023-09-30T02:10:18.187 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.181+0000 7f1f1df0e700 1 Processor -- start 2023-09-30T02:10:18.187 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.181+0000 7f1f1df0e700 1 -- start start 2023-09-30T02:10:18.187 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.181+0000 7f1f1df0e700 1 librados: setting wanted keys 2023-09-30T02:10:18.187 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.181+0000 7f1f1df0e700 1 librados: calling monclient init 2023-09-30T02:10:18.187 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.181+0000 7f1f1df0e700 10 monclient: init 2023-09-30T02:10:18.187 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.182+0000 7f1f1df0e700 10 monclient: _reopen_session rank -1 2023-09-30T02:10:18.187 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.182+0000 7f1f1df0e700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T02:10:18.187 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.182+0000 7f1f1df0e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1f180a5b70 0x7f1f1806be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:18.188 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.182+0000 7f1f1df0e700 10 monclient(hunting): picked mon.noname-b con 0x7f1f180a5b70 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:10:18.188 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.182+0000 7f1f1df0e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f180b1500 0x7f1f18140a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:18.188 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.182+0000 7f1f1df0e700 10 monclient(hunting): picked mon.noname-a con 0x7f1f180b1500 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:10:18.188 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.182+0000 7f1f1df0e700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1f18142290 0x7f1f1813ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:18.188 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.182+0000 7f1f1df0e700 10 monclient(hunting): picked mon.noname-c con 0x7f1f18142290 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:10:18.188 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.182+0000 7f1f1df0e700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:18.188 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.182+0000 7f1f1df0e700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f1806b860 con 0x7f1f180b1500 2023-09-30T02:10:18.188 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.182+0000 7f1f1df0e700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:18.188 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.182+0000 7f1f1df0e700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f1806c3c0 con 0x7f1f180a5b70 2023-09-30T02:10:18.188 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.182+0000 7f1f1df0e700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:18.188 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.182+0000 7f1f1df0e700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f1813dee0 con 0x7f1f18142290 2023-09-30T02:10:18.188 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.182+0000 7f1f1df0e700 10 monclient(hunting): _renew_subs 2023-09-30T02:10:18.189 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.182+0000 7f1f1df0e700 10 monclient(hunting): authenticate will time out at 2063.484135s 2023-09-30T02:10:18.189 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.182+0000 7f1f16ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f180b1500 0x7f1f18140a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:18.189 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.183+0000 7f1f177fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1f180a5b70 0x7f1f1806be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:18.189 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.183+0000 7f1f17fff700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1f18142290 0x7f1f1813ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:18.189 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.183+0000 7f1f16ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f180b1500 0x7f1f18140a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40168/0 (socket says 172.21.15.59:40168) 2023-09-30T02:10:18.189 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.183+0000 7f1f16ffd700 1 -- 172.21.15.59:0/562466409 learned_addr learned my addr 172.21.15.59:0/562466409 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:10:18.189 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.183+0000 7f1f16ffd700 10 monclient(hunting): get_auth_request con 0x7f1f180b1500 auth_method 0 2023-09-30T02:10:18.189 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.183+0000 7f1f16ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:10:18.189 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.183+0000 7f1f16ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:18.189 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.183+0000 7f1f16ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:18.189 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.183+0000 7f1f177fe700 10 monclient(hunting): get_auth_request con 0x7f1f180a5b70 auth_method 0 2023-09-30T02:10:18.189 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.183+0000 7f1f177fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:10:18.189 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.183+0000 7f1f177fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:18.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.183+0000 7f1f177fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:18.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.183+0000 7f1f17fff700 10 monclient(hunting): get_auth_request con 0x7f1f18142290 auth_method 0 2023-09-30T02:10:18.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.183+0000 7f1f17fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:10:18.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.183+0000 7f1f17fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:18.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.183+0000 7f1f17fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:18.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.184+0000 7f1f16ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:10:18.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.184+0000 7f1f16ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:10:18.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.184+0000 7f1f16ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:10:18.190 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.184+0000 7f1f17fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:10:18.191 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.184+0000 7f1f17fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:10:18.191 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.184+0000 7f1f17fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:10:18.191 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.184+0000 7f1f177fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:10:18.191 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.184+0000 7f1f177fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:10:18.191 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.184+0000 7f1f177fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:10:18.191 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.188+0000 7f1f17fff700 10 monclient(hunting): handle_auth_done global_id 4637 payload 824 2023-09-30T02:10:18.192 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.188+0000 7f1f17fff700 1 -- 172.21.15.59:0/562466409 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1f180a5b70 msgr2=0x7f1f1806be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:18.192 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.188+0000 7f1f17fff700 1 --2- 172.21.15.59:0/562466409 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1f180a5b70 0x7f1f1806be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:18.192 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.188+0000 7f1f17fff700 1 -- 172.21.15.59:0/562466409 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f180b1500 msgr2=0x7f1f18140a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:18.192 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.188+0000 7f1f17fff700 1 --2- 172.21.15.59:0/562466409 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f180b1500 0x7f1f18140a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:18.192 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.188+0000 7f1f17fff700 10 monclient: _finish_hunting 0 2023-09-30T02:10:18.192 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.188+0000 7f1f17fff700 1 monclient: found mon.noname-c 2023-09-30T02:10:18.192 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.188+0000 7f1f17fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:10:18.192 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.188+0000 7f1f17fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.59:3301/0 2023-09-30T02:10:18.192 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.188+0000 7f1f17fff700 1 -- 172.21.15.59:0/562466409 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f10009a20 con 0x7f1f18142290 2023-09-30T02:10:18.193 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.191+0000 7f1f17fff700 1 --2- 172.21.15.59:0/562466409 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1f18142290 0x7f1f1813ea40 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f1f180b1fc0 tx=0x7f1f10019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:10:18.193 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.191+0000 7f1f0f7fe700 10 client.?.objecter ms_handle_connect 0x7f1f18142290 2023-09-30T02:10:18.193 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.191+0000 7f1f0f7fe700 10 client.?.objecter resend_mon_ops 2023-09-30T02:10:18.195 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.196+0000 7f1f0f7fe700 1 -- 172.21.15.59:0/562466409 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f10026480 con 0x7f1f18142290 2023-09-30T02:10:18.196 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.196+0000 7f1f0f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:10:18.196 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.196+0000 7f1f0f7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:10:18.196 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.196+0000 7f1f0f7fe700 10 monclient: dump: 2023-09-30T02:10:18.196 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:10:18.196 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:18.196 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:18.196 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:18.196 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:10:18.196 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:18.196 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:10:18.196 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:10:18.197 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:10:18.197 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:18.197 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.196+0000 7f1f0f7fe700 10 monclient: _finish_auth 0 2023-09-30T02:10:18.197 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.196+0000 7f1f0f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:10:18.197 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.196+0000 7f1f0f7fe700 1 -- 172.21.15.59:0/562466409 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f10026600 con 0x7f1f18142290 2023-09-30T02:10:18.197 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.196+0000 7f1f1df0e700 5 monclient: authenticate success, global_id 4637 2023-09-30T02:10:18.197 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.196+0000 7f1f1df0e700 10 monclient: _renew_subs 2023-09-30T02:10:18.197 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.196+0000 7f1f1df0e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:10:18.197 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.196+0000 7f1f1df0e700 1 -- 172.21.15.59:0/562466409 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f1806ba60 con 0x7f1f18142290 2023-09-30T02:10:18.197 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.196+0000 7f1f0f7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:10:18.197 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.196+0000 7f1f0f7fe700 1 -- 172.21.15.59:0/562466409 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f100269c0 con 0x7f1f18142290 2023-09-30T02:10:18.197 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.196+0000 7f1f0f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:10:18.198 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.196+0000 7f1f0f7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:10:18.198 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.196+0000 7f1f0f7fe700 10 monclient: dump: 2023-09-30T02:10:18.198 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:10:18.198 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:18.198 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:18.198 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:18.198 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:10:18.198 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:18.198 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:10:18.198 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:10:18.198 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:10:18.198 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:18.199 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.196+0000 7f1f15ffb700 4 set_mon_vals no callback set 2023-09-30T02:10:18.199 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.196+0000 7f1f1df0e700 10 client.4637.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:10:18.199 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.196+0000 7f1f1df0e700 10 monclient: _renew_subs 2023-09-30T02:10:18.199 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.196+0000 7f1f1df0e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:10:18.199 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.196+0000 7f1f1df0e700 1 -- 172.21.15.59:0/562466409 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f1813f000 con 0x7f1f18142290 2023-09-30T02:10:18.199 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.196+0000 7f1f1df0e700 1 librados: init done 2023-09-30T02:10:18.201 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.197+0000 7f1f1df0e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:10:18.201 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.197+0000 7f1f1df0e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:10:18.201 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.197+0000 7f1f1df0e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:10:18.202 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.197+0000 7f1f1df0e700 1 -- 172.21.15.59:0/562466409 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f18063800 con 0x7f1f18142290 2023-09-30T02:10:18.224 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.225+0000 7f1f0f7fe700 1 -- 172.21.15.59:0/562466409 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f1f1001c400 con 0x7f1f18142290 2023-09-30T02:10:18.225 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.225+0000 7f1f0f7fe700 1 --2- 172.21.15.59:0/562466409 >> v2:172.21.15.110:6800/115870 conn(0x7f1ef8068890 0x7f1ef806ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:18.225 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.225+0000 7f1f0f7fe700 1 -- 172.21.15.59:0/562466409 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f1f100863f0 con 0x7f1f18142290 2023-09-30T02:10:18.225 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.225+0000 7f1f0f7fe700 10 client.4637.objecter ms_dispatch 0x7f1f18001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:10:18.225 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.225+0000 7f1f0f7fe700 3 client.4637.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:10:18.225 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.225+0000 7f1f0f7fe700 3 client.4637.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:10:18.225 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.225+0000 7f1f0f7fe700 20 client.4637.objecter dump_active .. 0 homeless 2023-09-30T02:10:18.226 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.226+0000 7f1f177fe700 1 --2- 172.21.15.59:0/562466409 >> v2:172.21.15.110:6800/115870 conn(0x7f1ef8068890 0x7f1ef806ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:18.226 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.226+0000 7f1f177fe700 10 monclient: get_auth_request con 0x7f1ef8068890 auth_method 0 2023-09-30T02:10:18.226 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.226+0000 7f1f0f7fe700 10 client.4637.objecter ms_handle_connect 0x7f1ef8068890 2023-09-30T02:10:18.226 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.226+0000 7f1f177fe700 1 --2- 172.21.15.59:0/562466409 >> v2:172.21.15.110:6800/115870 conn(0x7f1ef8068890 0x7f1ef806ad40 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f1f08007fd0 tx=0x7f1f08007d50 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:10:18.499 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.500+0000 7f1f0f7fe700 1 -- 172.21.15.59:0/562466409 <== mon.2 v2:172.21.15.59:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1f10053c90 con 0x7f1f18142290 2023-09-30T02:10:18.499 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.500+0000 7f1f0f7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:10:18.499 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.500+0000 7f1f0f7fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:10:18.752 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.753+0000 7f1f1df0e700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:10:18.753 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.753+0000 7f1f1df0e700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:10:18.753 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.753+0000 7f1f1df0e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:10:18.753 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.753+0000 7f1f1df0e700 1 -- 172.21.15.59:0/562466409 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f1f18064a00 con 0x7f1f18142290 2023-09-30T02:10:18.798 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.799+0000 7f1f0f7fe700 1 -- 172.21.15.59:0/562466409 <== mon.2 v2:172.21.15.59:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f1f100572b0 con 0x7f1f18142290 2023-09-30T02:10:18.799 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.799+0000 7f1f0f7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:10:18.799 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.799+0000 7f1f0f7fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:10:18.799 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:10:18.799 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:10:18.801 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.801+0000 7f1f1df0e700 10 librados: watch_flush enter 2023-09-30T02:10:18.802 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.801+0000 7f1f1df0e700 10 librados: watch_flush exit 2023-09-30T02:10:18.802 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.801+0000 7f1f1df0e700 20 client.4637.objecter shutdown clearing up homeless session... 2023-09-30T02:10:18.802 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.801+0000 7f1f1df0e700 10 client.4637.objecter successfully canceled tick 2023-09-30T02:10:18.802 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.801+0000 7f1f1df0e700 1 -- 172.21.15.59:0/562466409 >> v2:172.21.15.110:6800/115870 conn(0x7f1ef8068890 msgr2=0x7f1ef806ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:18.802 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.801+0000 7f1f1df0e700 1 --2- 172.21.15.59:0/562466409 >> v2:172.21.15.110:6800/115870 conn(0x7f1ef8068890 0x7f1ef806ad40 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f1f08007fd0 tx=0x7f1f08007d50 comp rx=0 tx=0).stop 2023-09-30T02:10:18.802 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.801+0000 7f1f1df0e700 10 monclient: shutdown 2023-09-30T02:10:18.802 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.801+0000 7f1f1df0e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:10:18.802 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.801+0000 7f1f1df0e700 1 -- 172.21.15.59:0/562466409 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1f18142290 msgr2=0x7f1f1813ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:18.802 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.801+0000 7f1f1df0e700 1 --2- 172.21.15.59:0/562466409 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1f18142290 0x7f1f1813ea40 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f1f180b1fc0 tx=0x7f1f10019000 comp rx=0 tx=0).stop 2023-09-30T02:10:18.802 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.802+0000 7f1f1df0e700 1 -- 172.21.15.59:0/562466409 shutdown_connections 2023-09-30T02:10:18.803 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.802+0000 7f1f1df0e700 1 --2- 172.21.15.59:0/562466409 >> v2:172.21.15.110:6800/115870 conn(0x7f1ef8068890 0x7f1ef806ad40 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:18.803 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.802+0000 7f1f1df0e700 1 --2- 172.21.15.59:0/562466409 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1f180a5b70 0x7f1f1806be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:18.803 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.802+0000 7f1f1df0e700 1 --2- 172.21.15.59:0/562466409 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1f18142290 0x7f1f1813ea40 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:18.803 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.802+0000 7f1f1df0e700 1 --2- 172.21.15.59:0/562466409 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1f180b1500 0x7f1f18140a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:18.803 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.802+0000 7f1f1df0e700 1 -- 172.21.15.59:0/562466409 >> 172.21.15.59:0/562466409 conn(0x7f1f180631f0 msgr2=0x7f1f180b3320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:10:18.803 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.802+0000 7f1f1df0e700 1 -- 172.21.15.59:0/562466409 shutdown_connections 2023-09-30T02:10:18.803 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.802+0000 7f1f1df0e700 1 -- 172.21.15.59:0/562466409 wait complete. 2023-09-30T02:10:18.803 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:18.802+0000 7f1f1df0e700 1 librados: shutdown 2023-09-30T02:10:21.817 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:10:21.954 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.955+0000 7f54d76de700 10 monclient: get_monmap_and_config 2023-09-30T02:10:21.955 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.955+0000 7f54d76de700 10 monclient: build_initial_monmap 2023-09-30T02:10:21.955 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.955+0000 7f54d76de700 10 monclient: monmap: 2023-09-30T02:10:21.955 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:10:21.955 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:21.955 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:10:21.955879+0000 2023-09-30T02:10:21.955 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:10:21.955879+0000 2023-09-30T02:10:21.955 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:10:21.955 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:21.955 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:10:21.955 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:10:21.956 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:10:21.956 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:10:21.956 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:10:21.956 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:21.957 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.956+0000 7f54d76de700 1 Processor -- start 2023-09-30T02:10:21.957 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.956+0000 7f54d76de700 1 -- start start 2023-09-30T02:10:21.957 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.956+0000 7f54d76de700 10 monclient: init 2023-09-30T02:10:21.957 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.956+0000 7f54d76de700 10 monclient: _reopen_session rank -1 2023-09-30T02:10:21.957 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.956+0000 7f54d76de700 10 monclient: _add_conns ranks=[2,1,0,4,3] 2023-09-30T02:10:21.957 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.956+0000 7f54d76de700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f54d00a5b70 0x7f54d00b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:21.957 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.956+0000 7f54d76de700 10 monclient(hunting): picked mon.noname-b con 0x7f54d00a5b70 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:10:21.958 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.956+0000 7f54d76de700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f54d00a5f40 addr v1:172.21.15.110:6789/0 2023-09-30T02:10:21.958 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.956+0000 7f54d76de700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f54d0142290 addr v1:172.21.15.59:6789/0 2023-09-30T02:10:21.958 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.957+0000 7f54d76de700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54d00a63b0 con 0x7f54d0142290 2023-09-30T02:10:21.958 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.957+0000 7f54d76de700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54d00a75a0 con 0x7f54d00a5f40 2023-09-30T02:10:21.958 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.957+0000 7f54d76de700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:21.958 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.957+0000 7f54d76de700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54d01426d0 con 0x7f54d00a5b70 2023-09-30T02:10:21.958 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.957+0000 7f54d76de700 10 monclient(hunting): _renew_subs 2023-09-30T02:10:21.958 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.957+0000 7f54d76de700 10 monclient(hunting): authenticate will time out at 2067.259037s 2023-09-30T02:10:21.958 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.957+0000 7f54d547a700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f54d00a5b70 0x7f54d00b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:21.958 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.957+0000 7f54d5c7b700 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f54d0142290 0x7f54d013af30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:50086/0 (socket says 172.21.15.59:50086) 2023-09-30T02:10:21.958 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.957+0000 7f54d5c7b700 1 -- 172.21.15.59:0/2301656157 learned_addr learned my addr 172.21.15.59:0/2301656157 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:10:21.959 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.958+0000 7f54d547a700 10 monclient(hunting): get_auth_request con 0x7f54d00a5b70 auth_method 0 2023-09-30T02:10:21.959 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.958+0000 7f54d547a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:10:21.959 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.958+0000 7f54d547a700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:21.959 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.958+0000 7f54d547a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:21.959 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.958+0000 7f54d547a700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:10:21.959 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.958+0000 7f54d547a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:10:21.960 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.958+0000 7f54d547a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:10:21.960 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.960+0000 7f54c7fff700 1 -- 172.21.15.59:0/2301656157 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1512363157 0 0) 0x7f54d00a63b0 con 0x7f54d0142290 2023-09-30T02:10:21.960 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.960+0000 7f54c7fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:21.960 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.960+0000 7f54c7fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:21.960 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.960+0000 7f54c7fff700 10 monclient(hunting): my global_id is 4710 2023-09-30T02:10:21.960 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.960+0000 7f54c7fff700 1 -- 172.21.15.59:0/2301656157 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54bc003570 con 0x7f54d0142290 2023-09-30T02:10:21.961 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.960+0000 7f54c7fff700 1 -- 172.21.15.59:0/2301656157 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 60789961 0 0) 0x7f54d00a75a0 con 0x7f54d00a5f40 2023-09-30T02:10:21.961 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.961+0000 7f54c7fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:21.961 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.961+0000 7f54c7fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:21.961 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.961+0000 7f54c7fff700 10 monclient(hunting): my global_id is 4741 2023-09-30T02:10:21.961 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.961+0000 7f54c7fff700 1 -- 172.21.15.59:0/2301656157 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54d00a63b0 con 0x7f54d00a5f40 2023-09-30T02:10:21.961 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.961+0000 7f54d547a700 10 monclient(hunting): handle_auth_done global_id 4640 payload 274 2023-09-30T02:10:21.961 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.961+0000 7f54d547a700 1 -- 172.21.15.59:0/2301656157 >> v1:172.21.15.110:6789/0 conn(0x7f54d00a5f40 legacy=0x7f54d0141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:21.961 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.961+0000 7f54d547a700 1 -- 172.21.15.59:0/2301656157 >> v1:172.21.15.59:6789/0 conn(0x7f54d0142290 legacy=0x7f54d013af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:21.961 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.961+0000 7f54d547a700 10 monclient: _finish_hunting 0 2023-09-30T02:10:21.961 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.961+0000 7f54d547a700 1 monclient: found mon.noname-b 2023-09-30T02:10:21.962 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.961+0000 7f54d547a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:10:21.962 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.961+0000 7f54d547a700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.59:3301/0 2023-09-30T02:10:21.962 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.961+0000 7f54d547a700 1 -- 172.21.15.59:0/2301656157 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54d00af3f0 con 0x7f54d00a5b70 2023-09-30T02:10:21.962 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.963+0000 7f54d547a700 1 --2- 172.21.15.59:0/2301656157 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f54d00a5b70 0x7f54d00b1500 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f54cc009910 tx=0x7f54cc0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df60ced29bd5379d server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:10:21.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.968+0000 7f54c7fff700 1 -- 172.21.15.59:0/2301656157 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54cc011630 con 0x7f54d00a5b70 2023-09-30T02:10:21.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.968+0000 7f54c7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:10:21.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.968+0000 7f54c7fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:10:21.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.968+0000 7f54c7fff700 10 monclient: dump: 2023-09-30T02:10:21.967 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:10:21.968 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:21.968 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:21.968 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:21.968 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:10:21.968 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:21.968 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:10:21.968 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:10:21.968 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:10:21.968 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:21.968 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.968+0000 7f54c7fff700 10 monclient: _finish_auth 0 2023-09-30T02:10:21.968 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.968+0000 7f54c7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:10:21.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.968+0000 7f54c7fff700 1 -- 172.21.15.59:0/2301656157 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54cc011790 con 0x7f54d00a5b70 2023-09-30T02:10:21.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.968+0000 7f54c7fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:10:21.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.968+0000 7f54c7fff700 1 -- 172.21.15.59:0/2301656157 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54cc011b30 con 0x7f54d00a5b70 2023-09-30T02:10:21.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.968+0000 7f54d76de700 5 monclient: authenticate success, global_id 4640 2023-09-30T02:10:21.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.968+0000 7f54d76de700 10 monclient: get_monmap_and_config success 2023-09-30T02:10:21.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.968+0000 7f54d76de700 4 set_mon_vals no callback set 2023-09-30T02:10:21.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.968+0000 7f54c7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:10:21.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.968+0000 7f54c7fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:10:21.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.968+0000 7f54c7fff700 10 monclient: dump: 2023-09-30T02:10:21.969 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:10:21.969 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:21.970 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:21.970 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:21.970 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:10:21.970 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:21.970 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:10:21.970 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:10:21.970 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:10:21.970 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:21.973 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.968+0000 7f54d76de700 10 monclient: shutdown 2023-09-30T02:10:21.973 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.968+0000 7f54d76de700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:10:21.973 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.968+0000 7f54d76de700 1 -- 172.21.15.59:0/2301656157 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f54d00a5b70 msgr2=0x7f54d00b1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:21.973 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.968+0000 7f54d76de700 1 --2- 172.21.15.59:0/2301656157 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f54d00a5b70 0x7f54d00b1500 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f54cc009910 tx=0x7f54cc0091a0 comp rx=0 tx=0).stop 2023-09-30T02:10:21.974 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.968+0000 7f54d76de700 1 -- 172.21.15.59:0/2301656157 shutdown_connections 2023-09-30T02:10:21.974 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.968+0000 7f54d76de700 1 --2- 172.21.15.59:0/2301656157 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f54d00a5b70 0x7f54d00b1500 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:21.974 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.968+0000 7f54d76de700 1 -- 172.21.15.59:0/2301656157 >> 172.21.15.59:0/2301656157 conn(0x7f54d00631f0 msgr2=0x7f54d00635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:10:21.974 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.968+0000 7f54d76de700 1 -- 172.21.15.59:0/2301656157 shutdown_connections 2023-09-30T02:10:21.974 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.969+0000 7f54d76de700 1 -- 172.21.15.59:0/2301656157 wait complete. 2023-09-30T02:10:21.974 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.969+0000 7f54d76de700 10 monclient: build_initial_monmap 2023-09-30T02:10:21.974 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.969+0000 7f54d76de700 10 monclient: monmap: 2023-09-30T02:10:21.974 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:10:21.974 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:10:21.974 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:10:21.974 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:10:21.974 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:10:21.975 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:10:21.975 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:10:21.975 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:10:21.975 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:10:21.975 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:21.975 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.969+0000 7f54d76de700 1 librados: starting msgr at 2023-09-30T02:10:21.975 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.969+0000 7f54d76de700 1 librados: starting objecter 2023-09-30T02:10:21.975 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.970+0000 7f54d76de700 1 Processor -- start 2023-09-30T02:10:21.975 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.970+0000 7f54d76de700 1 -- start start 2023-09-30T02:10:21.975 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.970+0000 7f54d76de700 1 librados: setting wanted keys 2023-09-30T02:10:21.975 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.970+0000 7f54d76de700 1 librados: calling monclient init 2023-09-30T02:10:21.975 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.970+0000 7f54d76de700 10 monclient: init 2023-09-30T02:10:21.976 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.970+0000 7f54d76de700 10 monclient: _reopen_session rank -1 2023-09-30T02:10:21.976 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.970+0000 7f54d76de700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T02:10:21.976 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.970+0000 7f54d76de700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f54d00a5b70 0x7f54d013dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:21.976 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.970+0000 7f54d76de700 10 monclient(hunting): picked mon.noname-c con 0x7f54d00a5b70 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:10:21.976 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.970+0000 7f54d76de700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54d00a5f40 0x7f54d01403a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:21.976 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.970+0000 7f54d76de700 10 monclient(hunting): picked mon.noname-b con 0x7f54d00a5f40 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:10:21.976 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.970+0000 7f54d76de700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54d0142290 0x7f54d014d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:21.976 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.970+0000 7f54d76de700 10 monclient(hunting): picked mon.noname-a con 0x7f54d0142290 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:10:21.976 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.970+0000 7f54d76de700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:21.977 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.970+0000 7f54d76de700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54d013e670 con 0x7f54d0142290 2023-09-30T02:10:21.977 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.970+0000 7f54d76de700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:21.977 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.970+0000 7f54d76de700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54d0141020 con 0x7f54d00a5f40 2023-09-30T02:10:21.977 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.970+0000 7f54d76de700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:21.977 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.970+0000 7f54d76de700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54d013d460 con 0x7f54d00a5b70 2023-09-30T02:10:21.977 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.970+0000 7f54d76de700 10 monclient(hunting): _renew_subs 2023-09-30T02:10:21.977 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.970+0000 7f54d76de700 10 monclient(hunting): authenticate will time out at 2067.272947s 2023-09-30T02:10:21.977 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.971+0000 7f54d547a700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f54d00a5b70 0x7f54d013dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:21.977 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.971+0000 7f54d5c7b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54d0142290 0x7f54d014d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:21.977 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.971+0000 7f54d4c79700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54d00a5f40 0x7f54d01403a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:21.977 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.971+0000 7f54d547a700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f54d00a5b70 0x7f54d013dfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:44098/0 (socket says 172.21.15.59:44098) 2023-09-30T02:10:21.978 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.971+0000 7f54d547a700 1 -- 172.21.15.59:0/1091414275 learned_addr learned my addr 172.21.15.59:0/1091414275 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:10:21.978 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.971+0000 7f54d547a700 10 monclient(hunting): get_auth_request con 0x7f54d00a5b70 auth_method 0 2023-09-30T02:10:21.978 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.971+0000 7f54d547a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:10:21.978 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.971+0000 7f54d547a700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:21.978 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.971+0000 7f54d547a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:21.978 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.972+0000 7f54d4c79700 10 monclient(hunting): get_auth_request con 0x7f54d00a5f40 auth_method 0 2023-09-30T02:10:21.978 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.972+0000 7f54d4c79700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:10:21.978 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.972+0000 7f54d4c79700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:21.978 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.972+0000 7f54d4c79700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:21.978 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.972+0000 7f54d5c7b700 10 monclient(hunting): get_auth_request con 0x7f54d0142290 auth_method 0 2023-09-30T02:10:21.978 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.972+0000 7f54d5c7b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:10:21.978 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.972+0000 7f54d5c7b700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:21.979 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.972+0000 7f54d5c7b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:21.979 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.972+0000 7f54d547a700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:10:21.979 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.972+0000 7f54d547a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:10:21.979 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.972+0000 7f54d547a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:10:21.979 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.972+0000 7f54d5c7b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:10:21.979 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.972+0000 7f54d5c7b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:10:21.979 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.972+0000 7f54d5c7b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:10:21.979 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.972+0000 7f54d4c79700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:10:21.979 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.972+0000 7f54d4c79700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:10:21.979 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.972+0000 7f54d4c79700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:10:21.980 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.976+0000 7f54d547a700 10 monclient(hunting): handle_auth_done global_id 4643 payload 824 2023-09-30T02:10:21.980 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.976+0000 7f54d547a700 1 -- 172.21.15.59:0/1091414275 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54d00a5f40 msgr2=0x7f54d01403a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:21.980 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.976+0000 7f54d547a700 1 --2- 172.21.15.59:0/1091414275 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54d00a5f40 0x7f54d01403a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:21.980 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.976+0000 7f54d547a700 1 -- 172.21.15.59:0/1091414275 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54d0142290 msgr2=0x7f54d014d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:21.981 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.976+0000 7f54d547a700 1 --2- 172.21.15.59:0/1091414275 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54d0142290 0x7f54d014d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:21.981 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.976+0000 7f54d547a700 10 monclient: _finish_hunting 0 2023-09-30T02:10:21.981 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.976+0000 7f54d547a700 1 monclient: found mon.noname-c 2023-09-30T02:10:21.981 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.976+0000 7f54d547a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:10:21.981 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.976+0000 7f54d547a700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.59:3301/0 2023-09-30T02:10:21.981 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.976+0000 7f54d547a700 1 -- 172.21.15.59:0/1091414275 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54cc009420 con 0x7f54d00a5b70 2023-09-30T02:10:21.981 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.978+0000 7f54d547a700 1 --2- 172.21.15.59:0/1091414275 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f54d00a5b70 0x7f54d013dfc0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f54cc009fd0 tx=0x7f54cc017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:10:21.981 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.978+0000 7f54c57fa700 10 client.?.objecter ms_handle_connect 0x7f54d00a5b70 2023-09-30T02:10:21.981 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.978+0000 7f54c57fa700 10 client.?.objecter resend_mon_ops 2023-09-30T02:10:21.983 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.983+0000 7f54c57fa700 1 -- 172.21.15.59:0/1091414275 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54cc01f070 con 0x7f54d00a5b70 2023-09-30T02:10:21.983 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.983+0000 7f54c57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:10:21.983 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.983+0000 7f54c57fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:10:21.983 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.983+0000 7f54c57fa700 10 monclient: dump: 2023-09-30T02:10:21.983 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:10:21.983 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:21.983 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:21.983 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:21.983 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:10:21.983 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:21.984 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:10:21.984 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:10:21.984 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:10:21.984 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:21.984 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.983+0000 7f54c57fa700 10 monclient: _finish_auth 0 2023-09-30T02:10:21.984 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.983+0000 7f54c57fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:10:21.984 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.983+0000 7f54c57fa700 1 -- 172.21.15.59:0/1091414275 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54cc002f70 con 0x7f54d00a5b70 2023-09-30T02:10:21.984 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.983+0000 7f54c57fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:10:21.984 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.983+0000 7f54c57fa700 1 -- 172.21.15.59:0/1091414275 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54cc011900 con 0x7f54d00a5b70 2023-09-30T02:10:21.985 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.983+0000 7f54c57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:10:21.985 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.983+0000 7f54c57fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:10:21.985 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.983+0000 7f54c57fa700 10 monclient: dump: 2023-09-30T02:10:21.985 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:10:21.985 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:21.985 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:21.985 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:21.985 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:10:21.985 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:21.985 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:10:21.985 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:10:21.986 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:10:21.986 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:21.986 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.983+0000 7f54d76de700 5 monclient: authenticate success, global_id 4643 2023-09-30T02:10:21.986 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.983+0000 7f54d76de700 10 monclient: _renew_subs 2023-09-30T02:10:21.986 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.983+0000 7f54d76de700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:10:21.986 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.983+0000 7f54d76de700 1 -- 172.21.15.59:0/1091414275 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54d013d6e0 con 0x7f54d00a5b70 2023-09-30T02:10:21.986 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.983+0000 7f54c77fe700 4 set_mon_vals no callback set 2023-09-30T02:10:21.986 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.983+0000 7f54d76de700 10 client.4643.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:10:21.986 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.983+0000 7f54d76de700 10 monclient: _renew_subs 2023-09-30T02:10:21.986 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.983+0000 7f54d76de700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:10:21.987 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.983+0000 7f54d76de700 1 -- 172.21.15.59:0/1091414275 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54d014e000 con 0x7f54d00a5b70 2023-09-30T02:10:21.987 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.983+0000 7f54d76de700 1 librados: init done 2023-09-30T02:10:21.987 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.984+0000 7f54d76de700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:10:21.987 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.984+0000 7f54d76de700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:10:21.987 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.984+0000 7f54d76de700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:10:21.987 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:21.984+0000 7f54d76de700 1 -- 172.21.15.59:0/1091414275 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54d0064a00 con 0x7f54d00a5b70 2023-09-30T02:10:22.012 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:22.012+0000 7f54c57fa700 1 -- 172.21.15.59:0/1091414275 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f54cc023070 con 0x7f54d00a5b70 2023-09-30T02:10:22.012 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:22.012+0000 7f54c57fa700 1 --2- 172.21.15.59:0/1091414275 >> v2:172.21.15.110:6800/115870 conn(0x7f54b4068890 0x7f54b406ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:22.012 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:22.013+0000 7f54c57fa700 1 -- 172.21.15.59:0/1091414275 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f54cc0858b0 con 0x7f54d00a5b70 2023-09-30T02:10:22.012 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:22.013+0000 7f54c57fa700 10 client.4643.objecter ms_dispatch 0x7f54d0001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:10:22.012 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:22.013+0000 7f54c57fa700 3 client.4643.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:10:22.012 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:22.013+0000 7f54c57fa700 3 client.4643.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:10:22.013 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:22.013+0000 7f54c57fa700 20 client.4643.objecter dump_active .. 0 homeless 2023-09-30T02:10:22.013 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:22.013+0000 7f54d4c79700 1 --2- 172.21.15.59:0/1091414275 >> v2:172.21.15.110:6800/115870 conn(0x7f54b4068890 0x7f54b406ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:22.013 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:22.013+0000 7f54d4c79700 10 monclient: get_auth_request con 0x7f54b4068890 auth_method 0 2023-09-30T02:10:22.013 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:22.013+0000 7f54d4c79700 1 --2- 172.21.15.59:0/1091414275 >> v2:172.21.15.110:6800/115870 conn(0x7f54b4068890 0x7f54b406ad40 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f54d00a84a0 tx=0x7f54c0007d50 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:10:22.013 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:22.013+0000 7f54c57fa700 10 client.4643.objecter ms_handle_connect 0x7f54b4068890 2023-09-30T02:10:22.278 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:22.278+0000 7f54c57fa700 1 -- 172.21.15.59:0/1091414275 <== mon.2 v2:172.21.15.59:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f54cc053150 con 0x7f54d00a5b70 2023-09-30T02:10:22.278 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:22.278+0000 7f54c57fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:10:22.278 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:22.278+0000 7f54c57fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:10:22.567 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:22.567+0000 7f54d76de700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:10:22.567 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:22.567+0000 7f54d76de700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:10:22.567 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:22.567+0000 7f54d76de700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:10:22.567 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:22.567+0000 7f54d76de700 1 -- 172.21.15.59:0/1091414275 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f54d006a720 con 0x7f54d00a5b70 2023-09-30T02:10:22.572 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:22.573+0000 7f54c57fa700 1 -- 172.21.15.59:0/1091414275 <== mon.2 v2:172.21.15.59:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f54cc015100 con 0x7f54d00a5b70 2023-09-30T02:10:22.572 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:22.573+0000 7f54c57fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:10:22.572 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:22.573+0000 7f54c57fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:10:22.573 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:10:22.573 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:10:22.575 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:22.575+0000 7f54d76de700 10 librados: watch_flush enter 2023-09-30T02:10:22.575 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:22.575+0000 7f54d76de700 10 librados: watch_flush exit 2023-09-30T02:10:22.575 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:22.575+0000 7f54d76de700 20 client.4643.objecter shutdown clearing up homeless session... 2023-09-30T02:10:22.575 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:22.575+0000 7f54d76de700 10 client.4643.objecter successfully canceled tick 2023-09-30T02:10:22.575 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:22.575+0000 7f54d76de700 1 -- 172.21.15.59:0/1091414275 >> v2:172.21.15.110:6800/115870 conn(0x7f54b4068890 msgr2=0x7f54b406ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:22.575 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:22.575+0000 7f54d76de700 1 --2- 172.21.15.59:0/1091414275 >> v2:172.21.15.110:6800/115870 conn(0x7f54b4068890 0x7f54b406ad40 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f54d00a84a0 tx=0x7f54c0007d50 comp rx=0 tx=0).stop 2023-09-30T02:10:22.575 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:22.575+0000 7f54d76de700 10 monclient: shutdown 2023-09-30T02:10:22.575 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:22.575+0000 7f54d76de700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:10:22.576 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:22.575+0000 7f54d76de700 1 -- 172.21.15.59:0/1091414275 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f54d00a5b70 msgr2=0x7f54d013dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:22.576 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:22.575+0000 7f54d76de700 1 --2- 172.21.15.59:0/1091414275 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f54d00a5b70 0x7f54d013dfc0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f54cc009fd0 tx=0x7f54cc017000 comp rx=0 tx=0).stop 2023-09-30T02:10:22.576 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:22.575+0000 7f54d76de700 1 -- 172.21.15.59:0/1091414275 shutdown_connections 2023-09-30T02:10:22.576 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:22.575+0000 7f54d76de700 1 --2- 172.21.15.59:0/1091414275 >> v2:172.21.15.110:6800/115870 conn(0x7f54b4068890 0x7f54b406ad40 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:22.576 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:22.575+0000 7f54d76de700 1 --2- 172.21.15.59:0/1091414275 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54d0142290 0x7f54d014d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:22.576 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:22.575+0000 7f54d76de700 1 --2- 172.21.15.59:0/1091414275 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f54d00a5b70 0x7f54d013dfc0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:22.576 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:22.575+0000 7f54d76de700 1 --2- 172.21.15.59:0/1091414275 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f54d00a5f40 0x7f54d01403a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:22.576 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:22.575+0000 7f54d76de700 1 -- 172.21.15.59:0/1091414275 >> 172.21.15.59:0/1091414275 conn(0x7f54d00631f0 msgr2=0x7f54d00635c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:10:22.576 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:22.575+0000 7f54d76de700 1 -- 172.21.15.59:0/1091414275 shutdown_connections 2023-09-30T02:10:22.576 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:22.575+0000 7f54d76de700 1 -- 172.21.15.59:0/1091414275 wait complete. 2023-09-30T02:10:22.576 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:22.575+0000 7f54d76de700 1 librados: shutdown 2023-09-30T02:10:25.589 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:10:25.730 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.730+0000 7f762efa1700 10 monclient: get_monmap_and_config 2023-09-30T02:10:25.730 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.730+0000 7f762efa1700 10 monclient: build_initial_monmap 2023-09-30T02:10:25.730 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.731+0000 7f762efa1700 10 monclient: monmap: 2023-09-30T02:10:25.730 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:10:25.730 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:25.731 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:10:25.731365+0000 2023-09-30T02:10:25.731 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:10:25.731365+0000 2023-09-30T02:10:25.731 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:10:25.731 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:25.731 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:10:25.731 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:10:25.731 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:10:25.731 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:10:25.731 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:10:25.731 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:25.733 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.731+0000 7f762efa1700 1 Processor -- start 2023-09-30T02:10:25.733 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.731+0000 7f762efa1700 1 -- start start 2023-09-30T02:10:25.733 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.731+0000 7f762efa1700 10 monclient: init 2023-09-30T02:10:25.733 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.732+0000 7f762efa1700 10 monclient: _reopen_session rank -1 2023-09-30T02:10:25.733 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.732+0000 7f762efa1700 10 monclient: _add_conns ranks=[0,2,4,3,1] 2023-09-30T02:10:25.733 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.732+0000 7f762efa1700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f76280a5b70 addr v1:172.21.15.59:6789/0 2023-09-30T02:10:25.733 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.732+0000 7f762efa1700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7628142290 0x7f76280b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:25.734 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.732+0000 7f762efa1700 10 monclient(hunting): picked mon.noname-b con 0x7f7628142290 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:10:25.734 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.732+0000 7f762efa1700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f76280a5f40 0x7f762813af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:25.734 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.732+0000 7f762efa1700 10 monclient(hunting): picked mon.noname-c con 0x7f76280a5f40 addr v2:172.21.15.110:3300/0 2023-09-30T02:10:25.734 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.732+0000 7f762efa1700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76280a6380 con 0x7f76280a5b70 2023-09-30T02:10:25.734 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.732+0000 7f762efa1700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:25.734 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.732+0000 7f762efa1700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76280a75a0 con 0x7f76280a5f40 2023-09-30T02:10:25.734 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.732+0000 7f762efa1700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:25.734 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.732+0000 7f762efa1700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76280af3f0 con 0x7f7628142290 2023-09-30T02:10:25.734 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.732+0000 7f762efa1700 10 monclient(hunting): _renew_subs 2023-09-30T02:10:25.734 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.732+0000 7f762efa1700 10 monclient(hunting): authenticate will time out at 2071.034678s 2023-09-30T02:10:25.734 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.733+0000 7f7627fff700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7628142290 0x7f76280b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:25.734 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.733+0000 7f762cd3d700 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f76280a5b70 0x7f7628141b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:50090/0 (socket says 172.21.15.59:50090) 2023-09-30T02:10:25.735 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.733+0000 7f762cd3d700 1 -- 172.21.15.59:0/4146528287 learned_addr learned my addr 172.21.15.59:0/4146528287 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:10:25.735 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.733+0000 7f762d53e700 1 --2- 172.21.15.59:0/4146528287 >> v2:172.21.15.110:3300/0 conn(0x7f76280a5f40 0x7f762813af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:25.735 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.733+0000 7f7627fff700 10 monclient(hunting): get_auth_request con 0x7f7628142290 auth_method 0 2023-09-30T02:10:25.735 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.733+0000 7f7627fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:10:25.735 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.733+0000 7f7627fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:25.735 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.733+0000 7f7627fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:25.735 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.733+0000 7f762d53e700 10 monclient(hunting): get_auth_request con 0x7f76280a5f40 auth_method 0 2023-09-30T02:10:25.735 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.733+0000 7f762d53e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:10:25.735 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.733+0000 7f762d53e700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:25.735 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.733+0000 7f762d53e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:25.736 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.734+0000 7f7627fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:10:25.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.734+0000 7f7627fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:10:25.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.734+0000 7f7627fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:10:25.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.734+0000 7f762d53e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:10:25.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.734+0000 7f762d53e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:10:25.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.734+0000 7f762d53e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:10:25.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.737+0000 7f76277fe700 1 -- 172.21.15.59:0/4146528287 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 385404008 0 0) 0x7f76280a6380 con 0x7f76280a5b70 2023-09-30T02:10:25.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.737+0000 7f76277fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:25.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.737+0000 7f76277fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:25.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.737+0000 7f76277fe700 10 monclient(hunting): my global_id is 4716 2023-09-30T02:10:25.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.737+0000 7f76277fe700 1 -- 172.21.15.59:0/4146528287 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7614003570 con 0x7f76280a5b70 2023-09-30T02:10:25.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.737+0000 7f7627fff700 10 monclient(hunting): handle_auth_done global_id 4646 payload 274 2023-09-30T02:10:25.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.737+0000 7f7627fff700 1 -- 172.21.15.59:0/4146528287 >> v2:172.21.15.110:3300/0 conn(0x7f76280a5f40 msgr2=0x7f762813af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:10:25.738 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.737+0000 7f7627fff700 1 --2- 172.21.15.59:0/4146528287 >> v2:172.21.15.110:3300/0 conn(0x7f76280a5f40 0x7f762813af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:25.738 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.737+0000 7f7627fff700 1 -- 172.21.15.59:0/4146528287 >> v1:172.21.15.59:6789/0 conn(0x7f76280a5b70 legacy=0x7f7628141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:25.738 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.737+0000 7f7627fff700 10 monclient: _finish_hunting 0 2023-09-30T02:10:25.738 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.737+0000 7f7627fff700 1 monclient: found mon.noname-b 2023-09-30T02:10:25.738 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.737+0000 7f7627fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:10:25.738 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.737+0000 7f7627fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.59:3301/0 2023-09-30T02:10:25.738 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.737+0000 7f7627fff700 1 -- 172.21.15.59:0/4146528287 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f762813b470 con 0x7f7628142290 2023-09-30T02:10:25.738 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.739+0000 7f7627fff700 1 --2- 172.21.15.59:0/4146528287 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7628142290 0x7f76280b1500 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f76100098e0 tx=0x7f76100091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d216872241ad1f76 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:10:25.743 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.743+0000 7f76277fe700 1 -- 172.21.15.59:0/4146528287 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7610011630 con 0x7f7628142290 2023-09-30T02:10:25.743 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.743+0000 7f76277fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:10:25.743 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.743+0000 7f76277fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:10:25.743 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.743+0000 7f76277fe700 10 monclient: dump: 2023-09-30T02:10:25.743 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:10:25.743 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:25.743 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:25.743 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:25.743 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:10:25.743 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:25.744 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:10:25.744 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:10:25.744 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:10:25.744 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:25.744 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.743+0000 7f76277fe700 10 monclient: _finish_auth 0 2023-09-30T02:10:25.744 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.743+0000 7f76277fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:10:25.744 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.743+0000 7f76277fe700 1 -- 172.21.15.59:0/4146528287 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7610011790 con 0x7f7628142290 2023-09-30T02:10:25.744 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.743+0000 7f76277fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:10:25.744 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.743+0000 7f76277fe700 1 -- 172.21.15.59:0/4146528287 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7610011b30 con 0x7f7628142290 2023-09-30T02:10:25.744 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.743+0000 7f762efa1700 5 monclient: authenticate success, global_id 4646 2023-09-30T02:10:25.744 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.743+0000 7f762efa1700 10 monclient: get_monmap_and_config success 2023-09-30T02:10:25.744 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.743+0000 7f76277fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:10:25.745 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.743+0000 7f762efa1700 4 set_mon_vals no callback set 2023-09-30T02:10:25.745 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.743+0000 7f76277fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:10:25.745 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.743+0000 7f76277fe700 10 monclient: dump: 2023-09-30T02:10:25.745 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:10:25.745 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:25.745 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:25.745 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:25.745 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:10:25.745 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:25.745 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:10:25.745 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:10:25.745 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:10:25.745 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:25.748 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.744+0000 7f762efa1700 10 monclient: shutdown 2023-09-30T02:10:25.748 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.744+0000 7f762efa1700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:10:25.748 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.744+0000 7f762efa1700 1 -- 172.21.15.59:0/4146528287 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7628142290 msgr2=0x7f76280b1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:25.748 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.744+0000 7f762efa1700 1 --2- 172.21.15.59:0/4146528287 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7628142290 0x7f76280b1500 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f76100098e0 tx=0x7f76100091a0 comp rx=0 tx=0).stop 2023-09-30T02:10:25.749 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.744+0000 7f762efa1700 1 -- 172.21.15.59:0/4146528287 shutdown_connections 2023-09-30T02:10:25.749 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.744+0000 7f762efa1700 1 --2- 172.21.15.59:0/4146528287 >> v2:172.21.15.110:3300/0 conn(0x7f76280a5f40 0x7f762813af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:25.749 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.744+0000 7f762efa1700 1 --2- 172.21.15.59:0/4146528287 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f7628142290 0x7f76280b1500 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:25.749 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.744+0000 7f762efa1700 1 -- 172.21.15.59:0/4146528287 >> 172.21.15.59:0/4146528287 conn(0x7f76280631f0 msgr2=0x7f76280635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:10:25.749 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.744+0000 7f762efa1700 1 -- 172.21.15.59:0/4146528287 shutdown_connections 2023-09-30T02:10:25.749 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.744+0000 7f762efa1700 1 -- 172.21.15.59:0/4146528287 wait complete. 2023-09-30T02:10:25.749 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.745+0000 7f762efa1700 10 monclient: build_initial_monmap 2023-09-30T02:10:25.749 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.745+0000 7f762efa1700 10 monclient: monmap: 2023-09-30T02:10:25.749 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:10:25.749 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:10:25.749 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:10:25.749 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:10:25.750 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:10:25.750 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:10:25.750 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:10:25.750 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:10:25.750 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:10:25.750 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:25.750 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.745+0000 7f762efa1700 1 librados: starting msgr at 2023-09-30T02:10:25.750 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.745+0000 7f762efa1700 1 librados: starting objecter 2023-09-30T02:10:25.750 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.745+0000 7f762efa1700 1 Processor -- start 2023-09-30T02:10:25.750 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.746+0000 7f762efa1700 1 -- start start 2023-09-30T02:10:25.750 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.746+0000 7f762efa1700 1 librados: setting wanted keys 2023-09-30T02:10:25.750 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.746+0000 7f762efa1700 1 librados: calling monclient init 2023-09-30T02:10:25.751 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.746+0000 7f762efa1700 10 monclient: init 2023-09-30T02:10:25.751 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.746+0000 7f762efa1700 10 monclient: _reopen_session rank -1 2023-09-30T02:10:25.751 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.746+0000 7f762efa1700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T02:10:25.751 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.746+0000 7f762efa1700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76280a5b70 0x7f762813e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:25.751 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.746+0000 7f762efa1700 10 monclient(hunting): picked mon.noname-b con 0x7f76280a5b70 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:10:25.751 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.746+0000 7f762efa1700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f76280a5f40 0x7f76280a7a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:25.751 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.746+0000 7f762efa1700 10 monclient(hunting): picked mon.noname-c con 0x7f76280a5f40 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:10:25.751 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.746+0000 7f762efa1700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7628142290 0x7f762814dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:25.751 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.746+0000 7f762efa1700 10 monclient(hunting): picked mon.noname-a con 0x7f7628142290 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:10:25.751 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.746+0000 7f762efa1700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:25.751 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.746+0000 7f762efa1700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f762814da30 con 0x7f7628142290 2023-09-30T02:10:25.751 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.746+0000 7f762efa1700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:25.751 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.746+0000 7f762efa1700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f762813c030 con 0x7f76280a5b70 2023-09-30T02:10:25.752 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.746+0000 7f762efa1700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:25.752 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.746+0000 7f762efa1700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f762813d980 con 0x7f76280a5f40 2023-09-30T02:10:25.752 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.746+0000 7f762efa1700 10 monclient(hunting): _renew_subs 2023-09-30T02:10:25.752 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.746+0000 7f762efa1700 10 monclient(hunting): authenticate will time out at 2071.048662s 2023-09-30T02:10:25.752 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.747+0000 7f7627fff700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f76280a5f40 0x7f76280a7a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:25.752 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.747+0000 7f762d53e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7628142290 0x7f762814dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:25.752 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.747+0000 7f762cd3d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76280a5b70 0x7f762813e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:25.752 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.747+0000 7f7627fff700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f76280a5f40 0x7f76280a7a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:44122/0 (socket says 172.21.15.59:44122) 2023-09-30T02:10:25.752 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.747+0000 7f7627fff700 1 -- 172.21.15.59:0/573448832 learned_addr learned my addr 172.21.15.59:0/573448832 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:10:25.752 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.747+0000 7f7627fff700 10 monclient(hunting): get_auth_request con 0x7f76280a5f40 auth_method 0 2023-09-30T02:10:25.752 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.747+0000 7f7627fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:10:25.753 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.747+0000 7f7627fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:25.753 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.747+0000 7f7627fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:25.753 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.747+0000 7f762cd3d700 10 monclient(hunting): get_auth_request con 0x7f76280a5b70 auth_method 0 2023-09-30T02:10:25.753 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.747+0000 7f762cd3d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:10:25.753 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.747+0000 7f762cd3d700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:25.753 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.747+0000 7f762cd3d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:25.753 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.747+0000 7f762d53e700 10 monclient(hunting): get_auth_request con 0x7f7628142290 auth_method 0 2023-09-30T02:10:25.753 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.747+0000 7f762d53e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:10:25.753 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.747+0000 7f762d53e700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:25.753 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.747+0000 7f762d53e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:25.753 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.748+0000 7f762cd3d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:10:25.753 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.748+0000 7f762cd3d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:10:25.753 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.748+0000 7f762cd3d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:10:25.754 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.748+0000 7f7627fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:10:25.754 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.748+0000 7f7627fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:10:25.754 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.748+0000 7f7627fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:10:25.754 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.748+0000 7f762d53e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:10:25.754 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.748+0000 7f762d53e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:10:25.754 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.748+0000 7f762d53e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:10:25.755 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.751+0000 7f762cd3d700 10 monclient(hunting): handle_auth_done global_id 4750 payload 824 2023-09-30T02:10:25.755 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.751+0000 7f762cd3d700 1 -- 172.21.15.59:0/573448832 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f76280a5f40 msgr2=0x7f76280a7a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:25.755 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.751+0000 7f762cd3d700 1 --2- 172.21.15.59:0/573448832 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f76280a5f40 0x7f76280a7a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:25.755 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.751+0000 7f762cd3d700 1 -- 172.21.15.59:0/573448832 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7628142290 msgr2=0x7f762814dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:25.755 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.751+0000 7f762cd3d700 1 --2- 172.21.15.59:0/573448832 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7628142290 0x7f762814dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:25.755 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.751+0000 7f762cd3d700 10 monclient: _finish_hunting 0 2023-09-30T02:10:25.755 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.751+0000 7f762cd3d700 1 monclient: found mon.noname-b 2023-09-30T02:10:25.755 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.751+0000 7f762cd3d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:10:25.756 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.751+0000 7f762cd3d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:10:25.756 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.751+0000 7f762cd3d700 1 -- 172.21.15.59:0/573448832 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7610009420 con 0x7f76280a5b70 2023-09-30T02:10:25.756 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.754+0000 7f762cd3d700 1 --2- 172.21.15.59:0/573448832 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76280a5b70 0x7f762813e5c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f76180058d0 tx=0x7f7618007d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:10:25.756 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.754+0000 7f7624ff9700 10 client.?.objecter ms_handle_connect 0x7f76280a5b70 2023-09-30T02:10:25.756 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.754+0000 7f7624ff9700 10 client.?.objecter resend_mon_ops 2023-09-30T02:10:25.759 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.759+0000 7f7624ff9700 1 -- 172.21.15.59:0/573448832 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7618005b30 con 0x7f76280a5b70 2023-09-30T02:10:25.759 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.759+0000 7f7624ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:10:25.759 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.759+0000 7f7624ff9700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:10:25.759 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.759+0000 7f7624ff9700 10 monclient: dump: 2023-09-30T02:10:25.759 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:10:25.760 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:25.760 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:25.760 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:25.760 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:10:25.760 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:25.760 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:10:25.760 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:10:25.760 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:10:25.760 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:25.760 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.759+0000 7f7624ff9700 10 monclient: _finish_auth 0 2023-09-30T02:10:25.760 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.759+0000 7f7624ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:10:25.760 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.759+0000 7f7624ff9700 1 -- 172.21.15.59:0/573448832 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7618005cd0 con 0x7f76280a5b70 2023-09-30T02:10:25.761 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.759+0000 7f7624ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:10:25.761 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.759+0000 7f7624ff9700 1 -- 172.21.15.59:0/573448832 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7618007320 con 0x7f76280a5b70 2023-09-30T02:10:25.761 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.759+0000 7f7624ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:10:25.761 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.759+0000 7f7624ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:10:25.761 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.759+0000 7f7624ff9700 10 monclient: dump: 2023-09-30T02:10:25.761 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:10:25.761 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:25.761 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:25.761 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:25.761 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:10:25.761 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:25.761 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:10:25.762 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:10:25.762 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:10:25.762 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:25.762 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.759+0000 7f762efa1700 5 monclient: authenticate success, global_id 4750 2023-09-30T02:10:25.762 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.759+0000 7f762efa1700 10 monclient: _renew_subs 2023-09-30T02:10:25.762 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.759+0000 7f7626ffd700 4 set_mon_vals no callback set 2023-09-30T02:10:25.762 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.759+0000 7f762efa1700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:10:25.762 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.759+0000 7f762efa1700 1 -- 172.21.15.59:0/573448832 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76280a7fa0 con 0x7f76280a5b70 2023-09-30T02:10:25.762 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.759+0000 7f762efa1700 10 client.4750.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:10:25.762 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.759+0000 7f762efa1700 10 monclient: _renew_subs 2023-09-30T02:10:25.762 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.759+0000 7f762efa1700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:10:25.763 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.759+0000 7f762efa1700 1 -- 172.21.15.59:0/573448832 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f762814e5c0 con 0x7f76280a5b70 2023-09-30T02:10:25.763 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.759+0000 7f762efa1700 1 librados: init done 2023-09-30T02:10:25.763 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.760+0000 7f762efa1700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:10:25.763 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.760+0000 7f762efa1700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:10:25.763 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.760+0000 7f762efa1700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:10:25.763 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:25.760+0000 7f762efa1700 1 -- 172.21.15.59:0/573448832 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f762814dd10 con 0x7f76280a5b70 2023-09-30T02:10:26.019 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:26.020+0000 7f7624ff9700 1 -- 172.21.15.59:0/573448832 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f7618007690 con 0x7f76280a5b70 2023-09-30T02:10:26.020 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:26.020+0000 7f7624ff9700 1 --2- 172.21.15.59:0/573448832 >> v2:172.21.15.110:6800/115870 conn(0x7f7608068910 0x7f760806adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:26.021 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:26.021+0000 7f7624ff9700 1 -- 172.21.15.59:0/573448832 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f7618083950 con 0x7f76280a5b70 2023-09-30T02:10:26.021 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:26.021+0000 7f7624ff9700 10 client.4750.objecter ms_dispatch 0x7f7628001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:10:26.021 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:26.021+0000 7f7624ff9700 3 client.4750.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:10:26.021 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:26.021+0000 7f7624ff9700 3 client.4750.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:10:26.021 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:26.021+0000 7f7624ff9700 20 client.4750.objecter dump_active .. 0 homeless 2023-09-30T02:10:26.021 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:26.021+0000 7f7627fff700 1 --2- 172.21.15.59:0/573448832 >> v2:172.21.15.110:6800/115870 conn(0x7f7608068910 0x7f760806adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:26.021 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:26.021+0000 7f7627fff700 10 monclient: get_auth_request con 0x7f7608068910 auth_method 0 2023-09-30T02:10:26.021 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:26.021+0000 7f7627fff700 1 --2- 172.21.15.59:0/573448832 >> v2:172.21.15.110:6800/115870 conn(0x7f7608068910 0x7f760806adc0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f762813ac70 tx=0x7f7610017000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:10:26.021 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:26.021+0000 7f7624ff9700 10 client.4750.objecter ms_handle_connect 0x7f7608068910 2023-09-30T02:10:26.028 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:26.029+0000 7f7624ff9700 1 -- 172.21.15.59:0/573448832 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f76180512a0 con 0x7f76280a5b70 2023-09-30T02:10:26.028 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:26.029+0000 7f7624ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:10:26.028 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:26.029+0000 7f7624ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:10:26.282 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:26.283+0000 7f762efa1700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:10:26.283 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:26.283+0000 7f762efa1700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:10:26.283 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:26.283+0000 7f762efa1700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:10:26.283 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:26.283+0000 7f762efa1700 1 -- 172.21.15.59:0/573448832 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f7628064a00 con 0x7f76280a5b70 2023-09-30T02:10:26.338 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:26.338+0000 7f7624ff9700 1 -- 172.21.15.59:0/573448832 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f76180548c0 con 0x7f76280a5b70 2023-09-30T02:10:26.338 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:26.338+0000 7f7624ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:10:26.338 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:26.338+0000 7f7624ff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:10:26.339 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:10:26.339 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:10:26.341 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:26.340+0000 7f762efa1700 10 librados: watch_flush enter 2023-09-30T02:10:26.341 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:26.341+0000 7f762efa1700 10 librados: watch_flush exit 2023-09-30T02:10:26.341 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:26.341+0000 7f762efa1700 20 client.4750.objecter shutdown clearing up homeless session... 2023-09-30T02:10:26.341 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:26.341+0000 7f762efa1700 10 client.4750.objecter successfully canceled tick 2023-09-30T02:10:26.341 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:26.341+0000 7f762efa1700 1 -- 172.21.15.59:0/573448832 >> v2:172.21.15.110:6800/115870 conn(0x7f7608068910 msgr2=0x7f760806adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:26.341 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:26.341+0000 7f762efa1700 1 --2- 172.21.15.59:0/573448832 >> v2:172.21.15.110:6800/115870 conn(0x7f7608068910 0x7f760806adc0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f762813ac70 tx=0x7f7610017000 comp rx=0 tx=0).stop 2023-09-30T02:10:26.341 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:26.341+0000 7f762efa1700 10 monclient: shutdown 2023-09-30T02:10:26.341 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:26.341+0000 7f762efa1700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:10:26.341 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:26.341+0000 7f762efa1700 1 -- 172.21.15.59:0/573448832 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76280a5b70 msgr2=0x7f762813e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:26.341 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:26.341+0000 7f762efa1700 1 --2- 172.21.15.59:0/573448832 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76280a5b70 0x7f762813e5c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f76180058d0 tx=0x7f7618007d10 comp rx=0 tx=0).stop 2023-09-30T02:10:26.342 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:26.341+0000 7f762efa1700 1 -- 172.21.15.59:0/573448832 shutdown_connections 2023-09-30T02:10:26.342 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:26.341+0000 7f762efa1700 1 --2- 172.21.15.59:0/573448832 >> v2:172.21.15.110:6800/115870 conn(0x7f7608068910 0x7f760806adc0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:26.342 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:26.341+0000 7f762efa1700 1 --2- 172.21.15.59:0/573448832 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f76280a5b70 0x7f762813e5c0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:26.342 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:26.341+0000 7f762efa1700 1 --2- 172.21.15.59:0/573448832 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7628142290 0x7f762814dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:26.342 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:26.341+0000 7f762efa1700 1 --2- 172.21.15.59:0/573448832 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f76280a5f40 0x7f76280a7a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:26.342 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:26.341+0000 7f762efa1700 1 -- 172.21.15.59:0/573448832 >> 172.21.15.59:0/573448832 conn(0x7f76280631f0 msgr2=0x7f7628141b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:10:26.342 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:26.341+0000 7f762efa1700 1 -- 172.21.15.59:0/573448832 shutdown_connections 2023-09-30T02:10:26.342 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:26.341+0000 7f762efa1700 1 -- 172.21.15.59:0/573448832 wait complete. 2023-09-30T02:10:26.342 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:26.341+0000 7f762efa1700 1 librados: shutdown 2023-09-30T02:10:29.356 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:10:29.493 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.494+0000 7f6de4811700 10 monclient: get_monmap_and_config 2023-09-30T02:10:29.493 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.494+0000 7f6de4811700 10 monclient: build_initial_monmap 2023-09-30T02:10:29.493 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.494+0000 7f6de4811700 10 monclient: monmap: 2023-09-30T02:10:29.494 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:10:29.494 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:29.494 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:10:29.494512+0000 2023-09-30T02:10:29.494 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:10:29.494512+0000 2023-09-30T02:10:29.494 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:10:29.494 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:29.495 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:10:29.495 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:10:29.495 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:10:29.495 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:10:29.495 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:10:29.495 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:29.497 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.494+0000 7f6de4811700 1 Processor -- start 2023-09-30T02:10:29.497 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.494+0000 7f6de4811700 1 -- start start 2023-09-30T02:10:29.497 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.494+0000 7f6de4811700 10 monclient: init 2023-09-30T02:10:29.497 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.495+0000 7f6de4811700 10 monclient: _reopen_session rank -1 2023-09-30T02:10:29.498 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.495+0000 7f6de4811700 10 monclient: _add_conns ranks=[3,1,4,2,0] 2023-09-30T02:10:29.498 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.495+0000 7f6de4811700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f6de00a5b70 0x7f6de00b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:29.498 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.495+0000 7f6de4811700 10 monclient(hunting): picked mon.noname-a con 0x7f6de00a5b70 addr v2:172.21.15.59:3300/0 2023-09-30T02:10:29.498 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.495+0000 7f6de4811700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6de00a5f40 addr v1:172.21.15.110:6789/0 2023-09-30T02:10:29.498 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.495+0000 7f6de4811700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f6de0142290 0x7f6de013af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:29.501 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.495+0000 7f6de4811700 10 monclient(hunting): picked mon.noname-c con 0x7f6de0142290 addr v2:172.21.15.110:3300/0 2023-09-30T02:10:29.501 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.495+0000 7f6de4811700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6de00a75a0 con 0x7f6de00a5f40 2023-09-30T02:10:29.501 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.495+0000 7f6de4811700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:29.501 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.495+0000 7f6de4811700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6de00a63e0 con 0x7f6de00a5b70 2023-09-30T02:10:29.502 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.495+0000 7f6de4811700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:29.502 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.495+0000 7f6de4811700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6de01426b0 con 0x7f6de0142290 2023-09-30T02:10:29.502 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.495+0000 7f6de4811700 10 monclient(hunting): _renew_subs 2023-09-30T02:10:29.502 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.495+0000 7f6de4811700 10 monclient(hunting): authenticate will time out at 2074.797860s 2023-09-30T02:10:29.502 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.496+0000 7f6dded9d700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f6de0142290 0x7f6de013af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:29.502 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.496+0000 7f6dde59c700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f6de00a5b70 0x7f6de00b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:29.502 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.497+0000 7f6dded9d700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f6de0142290 0x7f6de013af30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:34894/0 (socket says 172.21.15.59:34894) 2023-09-30T02:10:29.503 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.497+0000 7f6dded9d700 1 -- 172.21.15.59:0/1837734774 learned_addr learned my addr 172.21.15.59:0/1837734774 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:10:29.503 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.497+0000 7f6dded9d700 10 monclient(hunting): get_auth_request con 0x7f6de0142290 auth_method 0 2023-09-30T02:10:29.503 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.497+0000 7f6dded9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:10:29.503 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.497+0000 7f6dded9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:29.503 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.497+0000 7f6dded9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:29.503 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.497+0000 7f6dde59c700 10 monclient(hunting): get_auth_request con 0x7f6de00a5b70 auth_method 0 2023-09-30T02:10:29.504 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.497+0000 7f6dde59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:10:29.504 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.497+0000 7f6dde59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:29.504 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.497+0000 7f6dde59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:29.504 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.497+0000 7f6dded9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:10:29.504 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.497+0000 7f6dded9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:10:29.504 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.497+0000 7f6dded9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:10:29.505 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.498+0000 7f6dde59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:10:29.505 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.498+0000 7f6dde59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:10:29.505 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.498+0000 7f6dde59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:10:29.506 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.500+0000 7f6dded9d700 10 monclient(hunting): handle_auth_done global_id 4753 payload 274 2023-09-30T02:10:29.507 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.500+0000 7f6dded9d700 1 -- 172.21.15.59:0/1837734774 >> v2:172.21.15.59:3300/0 conn(0x7f6de00a5b70 msgr2=0x7f6de00b1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:10:29.507 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.500+0000 7f6dded9d700 1 --2- 172.21.15.59:0/1837734774 >> v2:172.21.15.59:3300/0 conn(0x7f6de00a5b70 0x7f6de00b1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:29.507 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.500+0000 7f6dded9d700 1 -- 172.21.15.59:0/1837734774 >> v1:172.21.15.110:6789/0 conn(0x7f6de00a5f40 legacy=0x7f6de0141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:10:29.507 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.500+0000 7f6dded9d700 10 monclient: _finish_hunting 0 2023-09-30T02:10:29.507 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.500+0000 7f6dded9d700 1 monclient: found mon.noname-c 2023-09-30T02:10:29.507 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.500+0000 7f6dded9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:10:29.508 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.500+0000 7f6dded9d700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3300/0 2023-09-30T02:10:29.508 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.500+0000 7f6dded9d700 1 -- 172.21.15.59:0/1837734774 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6de00af3f0 con 0x7f6de0142290 2023-09-30T02:10:29.508 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.504+0000 7f6dded9d700 1 --2- 172.21.15.59:0/1837734774 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6de0142290 0x7f6de013af30 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f6de00b0640 tx=0x7f6dd8009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e9bc262ed8030d9 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:10:29.514 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.508+0000 7f6ddd59a700 1 -- 172.21.15.59:0/1837734774 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dd800e070 con 0x7f6de0142290 2023-09-30T02:10:29.514 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.508+0000 7f6ddd59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:10:29.514 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.508+0000 7f6ddd59a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:10:29.514 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.508+0000 7f6ddd59a700 10 monclient: dump: 2023-09-30T02:10:29.514 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:10:29.515 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:29.515 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:29.515 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:29.515 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:10:29.515 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:29.515 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:10:29.516 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:10:29.516 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:10:29.516 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:29.516 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.508+0000 7f6ddd59a700 10 monclient: _finish_auth 0 2023-09-30T02:10:29.516 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.508+0000 7f6ddd59a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:10:29.516 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.508+0000 7f6de4811700 5 monclient: authenticate success, global_id 4753 2023-09-30T02:10:29.517 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.508+0000 7f6ddd59a700 1 -- 172.21.15.59:0/1837734774 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dd800b9e0 con 0x7f6de0142290 2023-09-30T02:10:29.517 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.508+0000 7f6ddd59a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:10:29.517 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.508+0000 7f6de4811700 10 monclient: get_monmap_and_config success 2023-09-30T02:10:29.517 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.508+0000 7f6ddd59a700 1 -- 172.21.15.59:0/1837734774 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dd80092e0 con 0x7f6de0142290 2023-09-30T02:10:29.517 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.508+0000 7f6ddd59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:10:29.517 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.508+0000 7f6de4811700 4 set_mon_vals no callback set 2023-09-30T02:10:29.518 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.508+0000 7f6ddd59a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:10:29.518 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.508+0000 7f6ddd59a700 10 monclient: dump: 2023-09-30T02:10:29.518 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:10:29.518 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:29.518 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:29.518 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:29.519 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:10:29.519 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:29.519 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:10:29.519 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:10:29.519 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:10:29.519 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:29.519 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.509+0000 7f6de4811700 10 monclient: shutdown 2023-09-30T02:10:29.519 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.509+0000 7f6de4811700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:10:29.520 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.509+0000 7f6de4811700 1 -- 172.21.15.59:0/1837734774 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6de0142290 msgr2=0x7f6de013af30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:29.520 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.509+0000 7f6de4811700 1 --2- 172.21.15.59:0/1837734774 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6de0142290 0x7f6de013af30 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f6de00b0640 tx=0x7f6dd8009840 comp rx=0 tx=0).stop 2023-09-30T02:10:29.520 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.509+0000 7f6de4811700 1 -- 172.21.15.59:0/1837734774 shutdown_connections 2023-09-30T02:10:29.520 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.509+0000 7f6de4811700 1 --2- 172.21.15.59:0/1837734774 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6de0142290 0x7f6de013af30 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:29.520 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.509+0000 7f6de4811700 1 --2- 172.21.15.59:0/1837734774 >> v2:172.21.15.59:3300/0 conn(0x7f6de00a5b70 0x7f6de00b1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:29.521 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.509+0000 7f6de4811700 1 -- 172.21.15.59:0/1837734774 >> 172.21.15.59:0/1837734774 conn(0x7f6de00631f0 msgr2=0x7f6de00635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:10:29.521 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.509+0000 7f6de4811700 1 -- 172.21.15.59:0/1837734774 shutdown_connections 2023-09-30T02:10:29.521 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.509+0000 7f6de4811700 1 -- 172.21.15.59:0/1837734774 wait complete. 2023-09-30T02:10:29.521 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.509+0000 7f6de4811700 10 monclient: build_initial_monmap 2023-09-30T02:10:29.521 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.509+0000 7f6de4811700 10 monclient: monmap: 2023-09-30T02:10:29.521 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:10:29.522 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:10:29.522 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:10:29.522 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:10:29.522 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:10:29.522 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:10:29.522 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:10:29.523 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:10:29.523 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:10:29.523 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:29.523 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.510+0000 7f6de4811700 1 librados: starting msgr at 2023-09-30T02:10:29.523 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.510+0000 7f6de4811700 1 librados: starting objecter 2023-09-30T02:10:29.523 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.510+0000 7f6de4811700 1 Processor -- start 2023-09-30T02:10:29.523 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.510+0000 7f6de4811700 1 -- start start 2023-09-30T02:10:29.523 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.510+0000 7f6de4811700 1 librados: setting wanted keys 2023-09-30T02:10:29.523 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.510+0000 7f6de4811700 1 librados: calling monclient init 2023-09-30T02:10:29.523 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.510+0000 7f6de4811700 10 monclient: init 2023-09-30T02:10:29.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.511+0000 7f6de4811700 10 monclient: _reopen_session rank -1 2023-09-30T02:10:29.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.511+0000 7f6de4811700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T02:10:29.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.511+0000 7f6de4811700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6de00a5b70 0x7f6de013d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:29.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.511+0000 7f6de4811700 10 monclient(hunting): picked mon.noname-c con 0x7f6de00a5b70 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:10:29.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.511+0000 7f6de4811700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6de00a5f40 0x7f6de014db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:29.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.511+0000 7f6de4811700 10 monclient(hunting): picked mon.noname-b con 0x7f6de00a5f40 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:10:29.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.511+0000 7f6de4811700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6de0142290 0x7f6de014e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:29.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.511+0000 7f6de4811700 10 monclient(hunting): picked mon.noname-a con 0x7f6de0142290 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:10:29.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.511+0000 7f6de4811700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:29.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.511+0000 7f6de4811700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6de014e5d0 con 0x7f6de0142290 2023-09-30T02:10:29.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.511+0000 7f6de4811700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:29.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.511+0000 7f6de4811700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6de014d650 con 0x7f6de00a5f40 2023-09-30T02:10:29.524 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.511+0000 7f6de4811700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:29.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.511+0000 7f6de4811700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6de01455a0 con 0x7f6de00a5b70 2023-09-30T02:10:29.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.511+0000 7f6de4811700 10 monclient(hunting): _renew_subs 2023-09-30T02:10:29.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.511+0000 7f6de4811700 10 monclient(hunting): authenticate will time out at 2074.813140s 2023-09-30T02:10:29.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.511+0000 7f6dde59c700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6de00a5b70 0x7f6de013d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:29.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.512+0000 7f6dded9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6de0142290 0x7f6de014e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:29.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.512+0000 7f6dddd9b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6de00a5f40 0x7f6de014db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:29.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.512+0000 7f6dde59c700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6de00a5b70 0x7f6de013d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:52734/0 (socket says 172.21.15.59:52734) 2023-09-30T02:10:29.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.512+0000 7f6dde59c700 1 -- 172.21.15.59:0/1563474004 learned_addr learned my addr 172.21.15.59:0/1563474004 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:10:29.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.512+0000 7f6dde59c700 10 monclient(hunting): get_auth_request con 0x7f6de00a5b70 auth_method 0 2023-09-30T02:10:29.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.512+0000 7f6dde59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:10:29.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.512+0000 7f6dde59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:29.525 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.512+0000 7f6dde59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:29.526 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.512+0000 7f6dddd9b700 10 monclient(hunting): get_auth_request con 0x7f6de00a5f40 auth_method 0 2023-09-30T02:10:29.526 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.512+0000 7f6dddd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:10:29.526 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.512+0000 7f6dddd9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:29.526 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.512+0000 7f6dddd9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:29.526 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.512+0000 7f6dded9d700 10 monclient(hunting): get_auth_request con 0x7f6de0142290 auth_method 0 2023-09-30T02:10:29.526 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.512+0000 7f6dded9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:10:29.526 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.512+0000 7f6dded9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:29.526 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.512+0000 7f6dded9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:29.526 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.512+0000 7f6dde59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:10:29.526 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.512+0000 7f6dde59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:10:29.526 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.512+0000 7f6dde59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:10:29.527 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.513+0000 7f6dddd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:10:29.527 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.513+0000 7f6dddd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:10:29.527 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.513+0000 7f6dddd9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:10:29.527 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.513+0000 7f6dded9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:10:29.527 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.513+0000 7f6dded9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:10:29.527 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.513+0000 7f6dded9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:10:29.529 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.516+0000 7f6dde59c700 10 monclient(hunting): handle_auth_done global_id 4652 payload 824 2023-09-30T02:10:29.529 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.516+0000 7f6dde59c700 1 -- 172.21.15.59:0/1563474004 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6de00a5f40 msgr2=0x7f6de014db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:29.529 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.516+0000 7f6dde59c700 1 --2- 172.21.15.59:0/1563474004 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6de00a5f40 0x7f6de014db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:29.529 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.516+0000 7f6dde59c700 1 -- 172.21.15.59:0/1563474004 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6de0142290 msgr2=0x7f6de014e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:29.529 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.516+0000 7f6dde59c700 1 --2- 172.21.15.59:0/1563474004 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6de0142290 0x7f6de014e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:29.529 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.516+0000 7f6dde59c700 10 monclient: _finish_hunting 0 2023-09-30T02:10:29.529 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.516+0000 7f6dde59c700 1 monclient: found mon.noname-c 2023-09-30T02:10:29.529 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.516+0000 7f6dde59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:10:29.529 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.516+0000 7f6dde59c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.59:3301/0 2023-09-30T02:10:29.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.516+0000 7f6dde59c700 1 -- 172.21.15.59:0/1563474004 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dd8009ae0 con 0x7f6de00a5b70 2023-09-30T02:10:29.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.517+0000 7f6dddd9b700 1 --2- 172.21.15.59:0/1563474004 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6de00a5f40 0x7f6de014db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T02:10:29.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.520+0000 7f6dde59c700 1 --2- 172.21.15.59:0/1563474004 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6de00a5b70 0x7f6de013d5f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f6dd000ec80 tx=0x7f6dd00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:10:29.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.520+0000 7f6dce7fc700 10 client.?.objecter ms_handle_connect 0x7f6de00a5b70 2023-09-30T02:10:29.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.520+0000 7f6dce7fc700 10 client.?.objecter resend_mon_ops 2023-09-30T02:10:29.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.524+0000 7f6dce7fc700 1 -- 172.21.15.59:0/1563474004 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dd000c850 con 0x7f6de00a5b70 2023-09-30T02:10:29.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.524+0000 7f6dce7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:10:29.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.524+0000 7f6dce7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:10:29.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.524+0000 7f6dce7fc700 10 monclient: dump: 2023-09-30T02:10:29.530 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:10:29.530 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:29.531 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:29.531 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:29.531 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:10:29.531 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:29.531 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:10:29.531 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:10:29.531 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:10:29.531 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:29.531 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.524+0000 7f6dce7fc700 10 monclient: _finish_auth 0 2023-09-30T02:10:29.531 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.524+0000 7f6dce7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:10:29.531 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.524+0000 7f6dce7fc700 1 -- 172.21.15.59:0/1563474004 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dd000c9b0 con 0x7f6de00a5b70 2023-09-30T02:10:29.531 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.524+0000 7f6dce7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:10:29.532 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.524+0000 7f6dce7fc700 1 -- 172.21.15.59:0/1563474004 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dd000cd50 con 0x7f6de00a5b70 2023-09-30T02:10:29.532 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.524+0000 7f6dce7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:10:29.532 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.524+0000 7f6dce7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:10:29.532 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.524+0000 7f6dce7fc700 10 monclient: dump: 2023-09-30T02:10:29.532 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:10:29.532 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:29.532 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:29.532 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:29.532 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:10:29.532 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:29.532 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:10:29.532 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:10:29.533 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:10:29.533 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:29.533 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.524+0000 7f6de4811700 5 monclient: authenticate success, global_id 4652 2023-09-30T02:10:29.533 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.524+0000 7f6de4811700 10 monclient: _renew_subs 2023-09-30T02:10:29.533 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.524+0000 7f6ddcd99700 4 set_mon_vals no callback set 2023-09-30T02:10:29.533 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.524+0000 7f6de4811700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:10:29.533 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.524+0000 7f6de4811700 1 -- 172.21.15.59:0/1563474004 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6de0143840 con 0x7f6de00a5b70 2023-09-30T02:10:29.533 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.524+0000 7f6de4811700 10 client.4652.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:10:29.533 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.524+0000 7f6de4811700 10 monclient: _renew_subs 2023-09-30T02:10:29.533 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.524+0000 7f6de4811700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:10:29.533 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.524+0000 7f6de4811700 1 -- 172.21.15.59:0/1563474004 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6de013dd70 con 0x7f6de00a5b70 2023-09-30T02:10:29.533 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.524+0000 7f6de4811700 1 librados: init done 2023-09-30T02:10:29.534 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.525+0000 7f6de4811700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:10:29.534 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.525+0000 7f6de4811700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:10:29.534 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.525+0000 7f6de4811700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:10:29.534 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.525+0000 7f6de4811700 1 -- 172.21.15.59:0/1563474004 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6de0064a00 con 0x7f6de00a5b70 2023-09-30T02:10:29.675 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.674+0000 7f6dce7fc700 1 -- 172.21.15.59:0/1563474004 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f6dd0013680 con 0x7f6de00a5b70 2023-09-30T02:10:29.675 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.675+0000 7f6dce7fc700 1 --2- 172.21.15.59:0/1563474004 >> v2:172.21.15.110:6800/115870 conn(0x7f6dc00689e0 0x7f6dc006ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:29.675 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.675+0000 7f6dce7fc700 1 -- 172.21.15.59:0/1563474004 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f6dd00848a0 con 0x7f6de00a5b70 2023-09-30T02:10:29.675 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.675+0000 7f6dce7fc700 10 client.4652.objecter ms_dispatch 0x7f6de0001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:10:29.675 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.675+0000 7f6dce7fc700 3 client.4652.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:10:29.675 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.675+0000 7f6dce7fc700 3 client.4652.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:10:29.675 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.675+0000 7f6dce7fc700 20 client.4652.objecter dump_active .. 0 homeless 2023-09-30T02:10:29.675 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.675+0000 7f6dddd9b700 1 --2- 172.21.15.59:0/1563474004 >> v2:172.21.15.110:6800/115870 conn(0x7f6dc00689e0 0x7f6dc006ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:29.675 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.675+0000 7f6dddd9b700 10 monclient: get_auth_request con 0x7f6dc00689e0 auth_method 0 2023-09-30T02:10:29.676 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.676+0000 7f6dce7fc700 10 client.4652.objecter ms_handle_connect 0x7f6dc00689e0 2023-09-30T02:10:29.676 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.676+0000 7f6dddd9b700 1 --2- 172.21.15.59:0/1563474004 >> v2:172.21.15.110:6800/115870 conn(0x7f6dc00689e0 0x7f6dc006ae90 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f6de013e810 tx=0x7f6dd4007ea0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:10:29.822 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.823+0000 7f6dce7fc700 1 -- 172.21.15.59:0/1563474004 <== mon.2 v2:172.21.15.59:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6dd00521c0 con 0x7f6de00a5b70 2023-09-30T02:10:29.822 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.823+0000 7f6dce7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:10:29.822 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:29.823+0000 7f6dce7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:10:30.075 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:30.075+0000 7f6de4811700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:10:30.075 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:30.075+0000 7f6de4811700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:10:30.075 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:30.075+0000 7f6de4811700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:10:30.075 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:30.075+0000 7f6de4811700 1 -- 172.21.15.59:0/1563474004 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f6de006a720 con 0x7f6de00a5b70 2023-09-30T02:10:30.079 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:30.080+0000 7f6dce7fc700 1 -- 172.21.15.59:0/1563474004 <== mon.2 v2:172.21.15.59:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f6dd00557e0 con 0x7f6de00a5b70 2023-09-30T02:10:30.079 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:30.080+0000 7f6dce7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:10:30.080 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:30.080+0000 7f6dce7fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:10:30.080 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:10:30.080 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:10:30.082 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:30.082+0000 7f6de4811700 10 librados: watch_flush enter 2023-09-30T02:10:30.082 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:30.082+0000 7f6de4811700 10 librados: watch_flush exit 2023-09-30T02:10:30.082 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:30.082+0000 7f6de4811700 20 client.4652.objecter shutdown clearing up homeless session... 2023-09-30T02:10:30.083 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:30.082+0000 7f6de4811700 10 client.4652.objecter successfully canceled tick 2023-09-30T02:10:30.083 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:30.082+0000 7f6de4811700 1 -- 172.21.15.59:0/1563474004 >> v2:172.21.15.110:6800/115870 conn(0x7f6dc00689e0 msgr2=0x7f6dc006ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:30.083 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:30.082+0000 7f6de4811700 1 --2- 172.21.15.59:0/1563474004 >> v2:172.21.15.110:6800/115870 conn(0x7f6dc00689e0 0x7f6dc006ae90 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f6de013e810 tx=0x7f6dd4007ea0 comp rx=0 tx=0).stop 2023-09-30T02:10:30.083 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:30.082+0000 7f6de4811700 10 monclient: shutdown 2023-09-30T02:10:30.083 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:30.082+0000 7f6de4811700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:10:30.083 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:30.082+0000 7f6de4811700 1 -- 172.21.15.59:0/1563474004 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6de00a5b70 msgr2=0x7f6de013d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:30.083 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:30.082+0000 7f6de4811700 1 --2- 172.21.15.59:0/1563474004 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6de00a5b70 0x7f6de013d5f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f6dd000ec80 tx=0x7f6dd00091a0 comp rx=0 tx=0).stop 2023-09-30T02:10:30.083 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:30.083+0000 7f6de4811700 1 -- 172.21.15.59:0/1563474004 shutdown_connections 2023-09-30T02:10:30.083 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:30.083+0000 7f6de4811700 1 --2- 172.21.15.59:0/1563474004 >> v2:172.21.15.110:6800/115870 conn(0x7f6dc00689e0 0x7f6dc006ae90 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:30.083 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:30.083+0000 7f6de4811700 1 --2- 172.21.15.59:0/1563474004 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6de0142290 0x7f6de014e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:30.083 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:30.083+0000 7f6de4811700 1 --2- 172.21.15.59:0/1563474004 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6de00a5b70 0x7f6de013d5f0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:30.084 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:30.083+0000 7f6de4811700 1 --2- 172.21.15.59:0/1563474004 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6de00a5f40 0x7f6de014db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:30.084 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:30.083+0000 7f6de4811700 1 -- 172.21.15.59:0/1563474004 >> 172.21.15.59:0/1563474004 conn(0x7f6de00631f0 msgr2=0x7f6de00635c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:10:30.084 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:30.083+0000 7f6de4811700 1 -- 172.21.15.59:0/1563474004 shutdown_connections 2023-09-30T02:10:30.084 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:30.083+0000 7f6de4811700 1 -- 172.21.15.59:0/1563474004 wait complete. 2023-09-30T02:10:30.084 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:30.083+0000 7f6de4811700 1 librados: shutdown 2023-09-30T02:10:33.096 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:10:33.235 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.236+0000 7fdd0f652700 10 monclient: get_monmap_and_config 2023-09-30T02:10:33.236 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.236+0000 7fdd0f652700 10 monclient: build_initial_monmap 2023-09-30T02:10:33.236 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.236+0000 7fdd0f652700 10 monclient: monmap: 2023-09-30T02:10:33.236 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:10:33.236 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:33.236 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:10:33.236518+0000 2023-09-30T02:10:33.236 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:10:33.236518+0000 2023-09-30T02:10:33.236 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:10:33.236 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:33.236 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:10:33.236 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:10:33.237 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:10:33.237 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:10:33.237 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:10:33.237 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:33.237 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.236+0000 7fdd0f652700 1 Processor -- start 2023-09-30T02:10:33.239 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.237+0000 7fdd0f652700 1 -- start start 2023-09-30T02:10:33.239 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.237+0000 7fdd0f652700 10 monclient: init 2023-09-30T02:10:33.239 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.237+0000 7fdd0f652700 10 monclient: _reopen_session rank -1 2023-09-30T02:10:33.239 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.237+0000 7fdd0f652700 10 monclient: _add_conns ranks=[2,1,0,3,4] 2023-09-30T02:10:33.239 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.237+0000 7fdd0f652700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdd080a5b70 0x7fdd080b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:33.239 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.237+0000 7fdd0f652700 10 monclient(hunting): picked mon.noname-b con 0x7fdd080a5b70 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:10:33.239 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.237+0000 7fdd0f652700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdd080a5f40 addr v1:172.21.15.110:6789/0 2023-09-30T02:10:33.239 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.237+0000 7fdd0f652700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdd08142290 addr v1:172.21.15.59:6789/0 2023-09-30T02:10:33.239 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.237+0000 7fdd0f652700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd080a63b0 con 0x7fdd08142290 2023-09-30T02:10:33.239 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.237+0000 7fdd0f652700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd080a75a0 con 0x7fdd080a5f40 2023-09-30T02:10:33.239 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.237+0000 7fdd0f652700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:33.240 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.237+0000 7fdd0f652700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd081426d0 con 0x7fdd080a5b70 2023-09-30T02:10:33.240 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.237+0000 7fdd0f652700 10 monclient(hunting): _renew_subs 2023-09-30T02:10:33.240 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.237+0000 7fdd0f652700 10 monclient(hunting): authenticate will time out at 2078.539907s 2023-09-30T02:10:33.240 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.238+0000 7fdd0d3ee700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdd080a5b70 0x7fdd080b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:33.240 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.238+0000 7fdd0dbef700 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fdd08142290 0x7fdd0813af30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:36616/0 (socket says 172.21.15.59:36616) 2023-09-30T02:10:33.240 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.238+0000 7fdd0dbef700 1 -- 172.21.15.59:0/1737606067 learned_addr learned my addr 172.21.15.59:0/1737606067 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:10:33.240 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.238+0000 7fdd0cbed700 1 --1- 172.21.15.59:0/1737606067 >> v1:172.21.15.110:6789/0 conn(0x7fdd080a5f40 0x7fdd08141b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.59:49614/0 (socket says 172.21.15.59:49614) 2023-09-30T02:10:33.240 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.238+0000 7fdd0d3ee700 10 monclient(hunting): get_auth_request con 0x7fdd080a5b70 auth_method 0 2023-09-30T02:10:33.240 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.238+0000 7fdd0d3ee700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:10:33.240 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.238+0000 7fdd0d3ee700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:33.240 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.238+0000 7fdd0d3ee700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:33.241 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.239+0000 7fdd0d3ee700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:10:33.241 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.239+0000 7fdd0d3ee700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:10:33.241 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.239+0000 7fdd0d3ee700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:10:33.242 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.242+0000 7fdcfffff700 1 -- 172.21.15.59:0/1737606067 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2100394516 0 0) 0x7fdd080a75a0 con 0x7fdd080a5f40 2023-09-30T02:10:33.242 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.242+0000 7fdcfffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:33.242 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.242+0000 7fdcfffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:33.242 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.242+0000 7fdcfffff700 10 monclient(hunting): my global_id is 4759 2023-09-30T02:10:33.242 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.242+0000 7fdcfffff700 1 -- 172.21.15.59:0/1737606067 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcf4003570 con 0x7fdd080a5f40 2023-09-30T02:10:33.242 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.242+0000 7fdcfffff700 1 -- 172.21.15.59:0/1737606067 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3693991134 0 0) 0x7fdd080a63b0 con 0x7fdd08142290 2023-09-30T02:10:33.242 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.242+0000 7fdcfffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:33.242 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.242+0000 7fdcfffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:33.242 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.242+0000 7fdcfffff700 10 monclient(hunting): my global_id is 4728 2023-09-30T02:10:33.242 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.242+0000 7fdcfffff700 1 -- 172.21.15.59:0/1737606067 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd080a75a0 con 0x7fdd08142290 2023-09-30T02:10:33.243 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.242+0000 7fdd0d3ee700 10 monclient(hunting): handle_auth_done global_id 4655 payload 274 2023-09-30T02:10:33.243 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.242+0000 7fdd0d3ee700 1 -- 172.21.15.59:0/1737606067 >> v1:172.21.15.110:6789/0 conn(0x7fdd080a5f40 legacy=0x7fdd08141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:33.243 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.242+0000 7fdd0d3ee700 1 -- 172.21.15.59:0/1737606067 >> v1:172.21.15.59:6789/0 conn(0x7fdd08142290 legacy=0x7fdd0813af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:33.243 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.242+0000 7fdd0d3ee700 10 monclient: _finish_hunting 0 2023-09-30T02:10:33.243 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.242+0000 7fdd0d3ee700 1 monclient: found mon.noname-b 2023-09-30T02:10:33.243 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.242+0000 7fdd0d3ee700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:10:33.243 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.242+0000 7fdd0d3ee700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.59:3301/0 2023-09-30T02:10:33.243 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.242+0000 7fdd0d3ee700 1 -- 172.21.15.59:0/1737606067 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd080af3f0 con 0x7fdd080a5b70 2023-09-30T02:10:33.244 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.245+0000 7fdd0d3ee700 1 --2- 172.21.15.59:0/1737606067 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdd080a5b70 0x7fdd080b1500 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fdd04009e90 tx=0x7fdd04009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac4e8629fd5d0413 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:10:33.249 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.249+0000 7fdcfffff700 1 -- 172.21.15.59:0/1737606067 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd0400e070 con 0x7fdd080a5b70 2023-09-30T02:10:33.249 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.249+0000 7fdcfffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:10:33.249 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.249+0000 7fdcfffff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:10:33.250 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.249+0000 7fdcfffff700 10 monclient: dump: 2023-09-30T02:10:33.250 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:10:33.250 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:33.250 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:33.250 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:33.250 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:10:33.251 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:33.251 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:10:33.251 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:10:33.251 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:10:33.251 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:33.251 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.249+0000 7fdcfffff700 10 monclient: _finish_auth 0 2023-09-30T02:10:33.252 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.249+0000 7fdcfffff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:10:33.252 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.249+0000 7fdcfffff700 1 -- 172.21.15.59:0/1737606067 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd0400bb80 con 0x7fdd080a5b70 2023-09-30T02:10:33.252 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.249+0000 7fdd0f652700 5 monclient: authenticate success, global_id 4655 2023-09-30T02:10:33.252 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.249+0000 7fdd0f652700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:10:33.252 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.249+0000 7fdcfffff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:10:33.252 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.249+0000 7fdcfffff700 1 -- 172.21.15.59:0/1737606067 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd04012430 con 0x7fdd080a5b70 2023-09-30T02:10:33.252 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.249+0000 7fdcfffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:10:33.253 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.249+0000 7fdcfffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:10:33.253 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.249+0000 7fdcfffff700 10 monclient: dump: 2023-09-30T02:10:33.253 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:10:33.253 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:33.253 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:33.253 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:33.254 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:10:33.254 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:33.254 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:10:33.254 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:10:33.254 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:10:33.254 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:33.255 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.249+0000 7fdd0f652700 10 monclient: get_monmap_and_config success 2023-09-30T02:10:33.255 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.249+0000 7fdd0f652700 4 set_mon_vals no callback set 2023-09-30T02:10:33.258 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.250+0000 7fdd0f652700 10 monclient: shutdown 2023-09-30T02:10:33.258 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.250+0000 7fdd0f652700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:10:33.258 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.250+0000 7fdd0f652700 1 -- 172.21.15.59:0/1737606067 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdd080a5b70 msgr2=0x7fdd080b1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:33.259 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.250+0000 7fdd0f652700 1 --2- 172.21.15.59:0/1737606067 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdd080a5b70 0x7fdd080b1500 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fdd04009e90 tx=0x7fdd04009780 comp rx=0 tx=0).stop 2023-09-30T02:10:33.259 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.250+0000 7fdd0f652700 1 -- 172.21.15.59:0/1737606067 shutdown_connections 2023-09-30T02:10:33.259 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.250+0000 7fdd0f652700 1 --2- 172.21.15.59:0/1737606067 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdd080a5b70 0x7fdd080b1500 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:33.259 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.250+0000 7fdd0f652700 1 -- 172.21.15.59:0/1737606067 >> 172.21.15.59:0/1737606067 conn(0x7fdd080631f0 msgr2=0x7fdd080635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:10:33.259 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.250+0000 7fdd0f652700 1 -- 172.21.15.59:0/1737606067 shutdown_connections 2023-09-30T02:10:33.259 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.250+0000 7fdd0f652700 1 -- 172.21.15.59:0/1737606067 wait complete. 2023-09-30T02:10:33.259 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.251+0000 7fdd0f652700 10 monclient: build_initial_monmap 2023-09-30T02:10:33.259 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.251+0000 7fdd0f652700 10 monclient: monmap: 2023-09-30T02:10:33.259 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:10:33.259 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:10:33.259 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:10:33.259 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:10:33.260 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:10:33.260 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:10:33.260 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:10:33.260 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:10:33.260 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:10:33.260 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:33.260 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.252+0000 7fdd0f652700 1 librados: starting msgr at 2023-09-30T02:10:33.260 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.252+0000 7fdd0f652700 1 librados: starting objecter 2023-09-30T02:10:33.260 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.252+0000 7fdd0f652700 1 Processor -- start 2023-09-30T02:10:33.260 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.252+0000 7fdd0f652700 1 -- start start 2023-09-30T02:10:33.260 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.252+0000 7fdd0f652700 1 librados: setting wanted keys 2023-09-30T02:10:33.260 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.252+0000 7fdd0f652700 1 librados: calling monclient init 2023-09-30T02:10:33.261 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.252+0000 7fdd0f652700 10 monclient: init 2023-09-30T02:10:33.261 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.253+0000 7fdd0f652700 10 monclient: _reopen_session rank -1 2023-09-30T02:10:33.261 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.253+0000 7fdd0f652700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T02:10:33.261 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.253+0000 7fdd0f652700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdd080a5b70 0x7fdd08145120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:33.261 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.253+0000 7fdd0f652700 10 monclient(hunting): picked mon.noname-c con 0x7fdd080a5b70 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:10:33.261 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.253+0000 7fdd0f652700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd080a5f40 0x7fdd0813e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:33.261 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.253+0000 7fdd0f652700 10 monclient(hunting): picked mon.noname-a con 0x7fdd080a5f40 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:10:33.261 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.253+0000 7fdd0f652700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdd08142290 0x7fdd0814db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:33.261 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.253+0000 7fdd0f652700 10 monclient(hunting): picked mon.noname-b con 0x7fdd08142290 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:10:33.261 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.253+0000 7fdd0f652700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:33.261 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.253+0000 7fdd0f652700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd0813ef50 con 0x7fdd080a5f40 2023-09-30T02:10:33.261 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.253+0000 7fdd0f652700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:33.261 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.253+0000 7fdd0f652700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd0814b6a0 con 0x7fdd08142290 2023-09-30T02:10:33.262 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.253+0000 7fdd0f652700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:33.262 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.253+0000 7fdd0f652700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd0813e570 con 0x7fdd080a5b70 2023-09-30T02:10:33.262 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.253+0000 7fdd0f652700 10 monclient(hunting): _renew_subs 2023-09-30T02:10:33.262 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.253+0000 7fdd0f652700 10 monclient(hunting): authenticate will time out at 2078.555406s 2023-09-30T02:10:33.262 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.253+0000 7fdd0d3ee700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdd080a5b70 0x7fdd08145120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:33.262 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.254+0000 7fdd0cbed700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd080a5f40 0x7fdd0813e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:33.262 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.254+0000 7fdd0dbef700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdd08142290 0x7fdd0814db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:33.262 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.254+0000 7fdd0d3ee700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdd080a5b70 0x7fdd08145120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:52756/0 (socket says 172.21.15.59:52756) 2023-09-30T02:10:33.262 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.254+0000 7fdd0d3ee700 1 -- 172.21.15.59:0/3419586065 learned_addr learned my addr 172.21.15.59:0/3419586065 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:10:33.262 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.254+0000 7fdd0d3ee700 10 monclient(hunting): get_auth_request con 0x7fdd080a5b70 auth_method 0 2023-09-30T02:10:33.262 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.254+0000 7fdd0d3ee700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:10:33.263 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.254+0000 7fdd0d3ee700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:33.263 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.254+0000 7fdd0d3ee700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:33.263 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.254+0000 7fdd0dbef700 10 monclient(hunting): get_auth_request con 0x7fdd08142290 auth_method 0 2023-09-30T02:10:33.263 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.254+0000 7fdd0dbef700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:10:33.263 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.254+0000 7fdd0dbef700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:33.263 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.254+0000 7fdd0dbef700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:33.263 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.254+0000 7fdd0cbed700 10 monclient(hunting): get_auth_request con 0x7fdd080a5f40 auth_method 0 2023-09-30T02:10:33.263 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.254+0000 7fdd0cbed700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:10:33.263 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.254+0000 7fdd0cbed700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:33.263 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.254+0000 7fdd0cbed700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:33.263 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.254+0000 7fdd0d3ee700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:10:33.263 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.254+0000 7fdd0d3ee700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:10:33.263 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.254+0000 7fdd0d3ee700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:10:33.264 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.255+0000 7fdd0dbef700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:10:33.264 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.255+0000 7fdd0dbef700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:10:33.264 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.255+0000 7fdd0dbef700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:10:33.264 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.255+0000 7fdd0cbed700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:10:33.264 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.255+0000 7fdd0cbed700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:10:33.264 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.255+0000 7fdd0cbed700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:10:33.264 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.258+0000 7fdd0d3ee700 10 monclient(hunting): handle_auth_done global_id 4658 payload 824 2023-09-30T02:10:33.264 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.258+0000 7fdd0d3ee700 1 -- 172.21.15.59:0/3419586065 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdd08142290 msgr2=0x7fdd0814db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:33.264 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.258+0000 7fdd0d3ee700 1 --2- 172.21.15.59:0/3419586065 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdd08142290 0x7fdd0814db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:33.264 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.258+0000 7fdd0d3ee700 1 -- 172.21.15.59:0/3419586065 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd080a5f40 msgr2=0x7fdd0813e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:33.264 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.258+0000 7fdd0d3ee700 1 --2- 172.21.15.59:0/3419586065 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd080a5f40 0x7fdd0813e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:33.264 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.258+0000 7fdd0d3ee700 10 monclient: _finish_hunting 0 2023-09-30T02:10:33.265 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.258+0000 7fdd0d3ee700 1 monclient: found mon.noname-c 2023-09-30T02:10:33.265 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.258+0000 7fdd0d3ee700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:10:33.265 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.258+0000 7fdd0d3ee700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.59:3301/0 2023-09-30T02:10:33.265 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.258+0000 7fdd0d3ee700 1 -- 172.21.15.59:0/3419586065 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd04009a20 con 0x7fdd080a5b70 2023-09-30T02:10:33.266 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.261+0000 7fdd0d3ee700 1 --2- 172.21.15.59:0/3419586065 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdd080a5b70 0x7fdd08145120 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fdd040097c0 tx=0x7fdd04019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:10:33.266 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.261+0000 7fdcfd7fa700 10 client.?.objecter ms_handle_connect 0x7fdd080a5b70 2023-09-30T02:10:33.267 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.261+0000 7fdcfd7fa700 10 client.?.objecter resend_mon_ops 2023-09-30T02:10:33.267 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.265+0000 7fdcfd7fa700 1 -- 172.21.15.59:0/3419586065 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd04012cd0 con 0x7fdd080a5b70 2023-09-30T02:10:33.267 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.265+0000 7fdcfd7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:10:33.267 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.265+0000 7fdcfd7fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:10:33.267 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.265+0000 7fdcfd7fa700 10 monclient: dump: 2023-09-30T02:10:33.267 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:10:33.267 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:33.267 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:33.267 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:33.267 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:10:33.267 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:33.267 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:10:33.268 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:10:33.268 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:10:33.268 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:33.268 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.265+0000 7fdcfd7fa700 10 monclient: _finish_auth 0 2023-09-30T02:10:33.268 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.265+0000 7fdcfd7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:10:33.268 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.265+0000 7fdcfd7fa700 1 -- 172.21.15.59:0/3419586065 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd04012e30 con 0x7fdd080a5b70 2023-09-30T02:10:33.268 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.265+0000 7fdcfd7fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:10:33.268 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.265+0000 7fdcfd7fa700 1 -- 172.21.15.59:0/3419586065 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd0401b670 con 0x7fdd080a5b70 2023-09-30T02:10:33.268 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.265+0000 7fdd0f652700 5 monclient: authenticate success, global_id 4658 2023-09-30T02:10:33.268 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.265+0000 7fdd0f652700 10 monclient: _renew_subs 2023-09-30T02:10:33.268 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.265+0000 7fdd0f652700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:10:33.268 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.265+0000 7fdd0f652700 1 -- 172.21.15.59:0/3419586065 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd0813e790 con 0x7fdd080a5b70 2023-09-30T02:10:33.268 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.265+0000 7fdcff7fe700 4 set_mon_vals no callback set 2023-09-30T02:10:33.269 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.265+0000 7fdcfd7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:10:33.269 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.265+0000 7fdcfd7fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:10:33.269 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.265+0000 7fdcfd7fa700 10 monclient: dump: 2023-09-30T02:10:33.269 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:10:33.269 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:33.269 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:33.269 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:33.269 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:10:33.269 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:33.269 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:10:33.269 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:10:33.269 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:10:33.270 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:33.270 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.265+0000 7fdd0f652700 10 client.4658.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:10:33.270 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.265+0000 7fdd0f652700 10 monclient: _renew_subs 2023-09-30T02:10:33.270 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.265+0000 7fdd0f652700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:10:33.270 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.265+0000 7fdd0f652700 1 -- 172.21.15.59:0/3419586065 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd0814e2f0 con 0x7fdd080a5b70 2023-09-30T02:10:33.270 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.265+0000 7fdd0f652700 1 librados: init done 2023-09-30T02:10:33.270 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.266+0000 7fdd0f652700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:10:33.270 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.266+0000 7fdd0f652700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:10:33.270 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.266+0000 7fdd0f652700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:10:33.270 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.266+0000 7fdd0f652700 1 -- 172.21.15.59:0/3419586065 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd08064a00 con 0x7fdd080a5b70 2023-09-30T02:10:33.489 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.489+0000 7fdcfd7fa700 1 -- 172.21.15.59:0/3419586065 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fdd0401ba60 con 0x7fdd080a5b70 2023-09-30T02:10:33.489 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.490+0000 7fdcfd7fa700 1 --2- 172.21.15.59:0/3419586065 >> v2:172.21.15.110:6800/115870 conn(0x7fdcec068960 0x7fdcec06ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:33.490 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.490+0000 7fdd0cbed700 1 --2- 172.21.15.59:0/3419586065 >> v2:172.21.15.110:6800/115870 conn(0x7fdcec068960 0x7fdcec06ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:33.490 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.490+0000 7fdd0cbed700 10 monclient: get_auth_request con 0x7fdcec068960 auth_method 0 2023-09-30T02:10:33.490 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.491+0000 7fdcfd7fa700 10 client.4658.objecter ms_handle_connect 0x7fdcec068960 2023-09-30T02:10:33.490 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.491+0000 7fdd0cbed700 1 --2- 172.21.15.59:0/3419586065 >> v2:172.21.15.110:6800/115870 conn(0x7fdcec068960 0x7fdcec06ae10 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fdd080a7fe0 tx=0x7fdcf8007d40 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:10:33.538 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.538+0000 7fdcfd7fa700 1 -- 172.21.15.59:0/3419586065 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fdd04057bd0 con 0x7fdd080a5b70 2023-09-30T02:10:33.539 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.538+0000 7fdcfd7fa700 10 client.4658.objecter ms_dispatch 0x7fdd08001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:10:33.539 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.538+0000 7fdcfd7fa700 3 client.4658.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:10:33.539 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.538+0000 7fdcfd7fa700 3 client.4658.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:10:33.539 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.538+0000 7fdcfd7fa700 20 client.4658.objecter dump_active .. 0 homeless 2023-09-30T02:10:33.567 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.567+0000 7fdcfd7fa700 1 -- 172.21.15.59:0/3419586065 <== mon.2 v2:172.21.15.59:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fdd04053640 con 0x7fdd080a5b70 2023-09-30T02:10:33.567 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.567+0000 7fdcfd7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:10:33.567 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.567+0000 7fdcfd7fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:10:33.865 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.866+0000 7fdd0f652700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:10:33.865 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.866+0000 7fdd0f652700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:10:33.865 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.866+0000 7fdd0f652700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:10:33.866 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.866+0000 7fdd0f652700 1 -- 172.21.15.59:0/3419586065 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fdd0806a720 con 0x7fdd080a5b70 2023-09-30T02:10:33.931 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.932+0000 7fdcfd7fa700 1 -- 172.21.15.59:0/3419586065 <== mon.2 v2:172.21.15.59:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fdd0402a080 con 0x7fdd080a5b70 2023-09-30T02:10:33.932 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.932+0000 7fdcfd7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:10:33.932 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.932+0000 7fdcfd7fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:10:33.932 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:10:33.932 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:10:33.936 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.934+0000 7fdd0f652700 10 librados: watch_flush enter 2023-09-30T02:10:33.936 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.934+0000 7fdd0f652700 10 librados: watch_flush exit 2023-09-30T02:10:33.936 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.934+0000 7fdd0f652700 20 client.4658.objecter shutdown clearing up homeless session... 2023-09-30T02:10:33.936 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.934+0000 7fdd0f652700 10 client.4658.objecter successfully canceled tick 2023-09-30T02:10:33.936 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.935+0000 7fdd0f652700 1 -- 172.21.15.59:0/3419586065 >> v2:172.21.15.110:6800/115870 conn(0x7fdcec068960 msgr2=0x7fdcec06ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:33.937 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.935+0000 7fdd0f652700 1 --2- 172.21.15.59:0/3419586065 >> v2:172.21.15.110:6800/115870 conn(0x7fdcec068960 0x7fdcec06ae10 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fdd080a7fe0 tx=0x7fdcf8007d40 comp rx=0 tx=0).stop 2023-09-30T02:10:33.937 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.935+0000 7fdd0f652700 10 monclient: shutdown 2023-09-30T02:10:33.937 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.935+0000 7fdd0f652700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:10:33.937 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.935+0000 7fdd0f652700 1 -- 172.21.15.59:0/3419586065 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdd080a5b70 msgr2=0x7fdd08145120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:33.937 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.935+0000 7fdd0f652700 1 --2- 172.21.15.59:0/3419586065 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdd080a5b70 0x7fdd08145120 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fdd040097c0 tx=0x7fdd04019000 comp rx=0 tx=0).stop 2023-09-30T02:10:33.938 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.935+0000 7fdd0f652700 1 -- 172.21.15.59:0/3419586065 shutdown_connections 2023-09-30T02:10:33.938 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.935+0000 7fdd0f652700 1 --2- 172.21.15.59:0/3419586065 >> v2:172.21.15.110:6800/115870 conn(0x7fdcec068960 0x7fdcec06ae10 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:33.938 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.935+0000 7fdd0f652700 1 --2- 172.21.15.59:0/3419586065 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdd08142290 0x7fdd0814db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:33.938 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.935+0000 7fdd0f652700 1 --2- 172.21.15.59:0/3419586065 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdd080a5b70 0x7fdd08145120 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:33.938 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.935+0000 7fdd0f652700 1 --2- 172.21.15.59:0/3419586065 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdd080a5f40 0x7fdd0813e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:33.939 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.935+0000 7fdd0f652700 1 -- 172.21.15.59:0/3419586065 >> 172.21.15.59:0/3419586065 conn(0x7fdd080631f0 msgr2=0x7fdd08138220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:10:33.939 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.935+0000 7fdd0f652700 1 -- 172.21.15.59:0/3419586065 shutdown_connections 2023-09-30T02:10:33.939 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.935+0000 7fdd0f652700 1 -- 172.21.15.59:0/3419586065 wait complete. 2023-09-30T02:10:33.939 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:33.935+0000 7fdd0f652700 1 librados: shutdown 2023-09-30T02:10:34.290 INFO:tasks.ceph.mon.a.smithi059.stderr:==00:00:05:44.156 116256== Warning: unimplemented fcntl command: 1036 2023-09-30T02:10:34.297 INFO:tasks.ceph.mon.a.smithi059.stderr:==00:00:05:44.162 116256== Warning: unimplemented fcntl command: 1036 2023-09-30T02:10:34.326 INFO:tasks.ceph.mon.a.smithi059.stderr:==00:00:05:44.191 116256== Warning: unimplemented fcntl command: 1036 2023-09-30T02:10:34.350 INFO:tasks.ceph.mon.b.smithi110.stderr:==00:00:05:44.197 115869== Warning: unimplemented fcntl command: 1036 2023-09-30T02:10:34.353 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:05:44.219 116257== Warning: unimplemented fcntl command: 1036 2023-09-30T02:10:34.354 INFO:tasks.ceph.mon.b.smithi110.stderr:==00:00:05:44.201 115869== Warning: unimplemented fcntl command: 1036 2023-09-30T02:10:34.358 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:05:44.223 116257== Warning: unimplemented fcntl command: 1036 2023-09-30T02:10:34.379 INFO:tasks.ceph.mon.b.smithi110.stderr:==00:00:05:44.226 115869== Warning: unimplemented fcntl command: 1036 2023-09-30T02:10:34.385 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:05:44.251 116257== Warning: unimplemented fcntl command: 1036 2023-09-30T02:10:36.953 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:10:37.086 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.085+0000 7fb6a4b24700 10 monclient: get_monmap_and_config 2023-09-30T02:10:37.086 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.085+0000 7fb6a4b24700 10 monclient: build_initial_monmap 2023-09-30T02:10:37.086 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.085+0000 7fb6a4b24700 10 monclient: monmap: 2023-09-30T02:10:37.086 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:10:37.086 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:37.087 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:10:37.086976+0000 2023-09-30T02:10:37.087 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:10:37.086976+0000 2023-09-30T02:10:37.087 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:10:37.087 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:37.087 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:10:37.087 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:10:37.087 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:10:37.087 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:10:37.087 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:10:37.087 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:37.087 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.086+0000 7fb6a4b24700 1 Processor -- start 2023-09-30T02:10:37.088 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.086+0000 7fb6a4b24700 1 -- start start 2023-09-30T02:10:37.088 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.086+0000 7fb6a4b24700 10 monclient: init 2023-09-30T02:10:37.090 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.087+0000 7fb6a4b24700 10 monclient: _reopen_session rank -1 2023-09-30T02:10:37.090 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.087+0000 7fb6a4b24700 10 monclient: _add_conns ranks=[2,3,1,0,4] 2023-09-30T02:10:37.090 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.087+0000 7fb6a4b24700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb6a00a5b70 0x7fb6a00b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:37.090 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.087+0000 7fb6a4b24700 10 monclient(hunting): picked mon.noname-b con 0x7fb6a00a5b70 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:10:37.090 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.087+0000 7fb6a4b24700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fb6a00a5f40 0x7fb6a0141b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:37.090 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.087+0000 7fb6a4b24700 10 monclient(hunting): picked mon.noname-a con 0x7fb6a00a5f40 addr v2:172.21.15.59:3300/0 2023-09-30T02:10:37.090 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.087+0000 7fb6a4b24700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb6a01420c0 addr v1:172.21.15.110:6789/0 2023-09-30T02:10:37.090 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.087+0000 7fb6a4b24700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6a00a63b0 con 0x7fb6a01420c0 2023-09-30T02:10:37.090 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.087+0000 7fb6a4b24700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:37.091 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.087+0000 7fb6a4b24700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6a00a75a0 con 0x7fb6a00a5f40 2023-09-30T02:10:37.091 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.087+0000 7fb6a4b24700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:37.091 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.087+0000 7fb6a4b24700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a00af3f0 con 0x7fb6a00a5b70 2023-09-30T02:10:37.091 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.087+0000 7fb6a4b24700 10 monclient(hunting): _renew_subs 2023-09-30T02:10:37.091 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.087+0000 7fb6a4b24700 10 monclient(hunting): authenticate will time out at 2082.390168s 2023-09-30T02:10:37.091 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.087+0000 7fb69e59c700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb6a00a5b70 0x7fb6a00b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:37.091 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.087+0000 7fb69dd9b700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fb6a00a5f40 0x7fb6a0141b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:37.091 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.088+0000 7fb69ed9d700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fb6a01420c0 0x7fb6a013e230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.59:49628/0 (socket says 172.21.15.59:49628) 2023-09-30T02:10:37.091 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.088+0000 7fb69ed9d700 1 -- 172.21.15.59:0/1124776327 learned_addr learned my addr 172.21.15.59:0/1124776327 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:10:37.091 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.088+0000 7fb69e59c700 10 monclient(hunting): get_auth_request con 0x7fb6a00a5b70 auth_method 0 2023-09-30T02:10:37.091 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.088+0000 7fb69e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:10:37.091 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.088+0000 7fb69e59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:37.092 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.088+0000 7fb69e59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:37.092 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.088+0000 7fb69dd9b700 10 monclient(hunting): get_auth_request con 0x7fb6a00a5f40 auth_method 0 2023-09-30T02:10:37.092 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.088+0000 7fb69dd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:10:37.092 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.088+0000 7fb69dd9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:37.092 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.088+0000 7fb69dd9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:37.092 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.089+0000 7fb69e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:10:37.092 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.089+0000 7fb69e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:10:37.092 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.089+0000 7fb69e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:10:37.092 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.089+0000 7fb69dd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:10:37.092 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.089+0000 7fb69dd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:10:37.092 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.089+0000 7fb69dd9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:10:37.093 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.091+0000 7fb69d59a700 1 -- 172.21.15.59:0/1124776327 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1409893237 0 0) 0x7fb6a00a63b0 con 0x7fb6a01420c0 2023-09-30T02:10:37.094 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.091+0000 7fb69d59a700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:37.094 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.091+0000 7fb69d59a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:37.094 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.091+0000 7fb69d59a700 10 monclient(hunting): my global_id is 4765 2023-09-30T02:10:37.094 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.091+0000 7fb69d59a700 1 -- 172.21.15.59:0/1124776327 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb688003570 con 0x7fb6a01420c0 2023-09-30T02:10:37.094 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.092+0000 7fb69dd9b700 10 monclient(hunting): handle_auth_done global_id 4734 payload 274 2023-09-30T02:10:37.094 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.092+0000 7fb69dd9b700 1 -- 172.21.15.59:0/1124776327 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb6a00a5b70 msgr2=0x7fb6a00b1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:10:37.094 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.092+0000 7fb69dd9b700 1 --2- 172.21.15.59:0/1124776327 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb6a00a5b70 0x7fb6a00b1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:37.094 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.092+0000 7fb69dd9b700 1 -- 172.21.15.59:0/1124776327 >> v1:172.21.15.110:6789/0 conn(0x7fb6a01420c0 legacy=0x7fb6a013e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:37.094 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.092+0000 7fb69dd9b700 10 monclient: _finish_hunting 0 2023-09-30T02:10:37.094 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.092+0000 7fb69dd9b700 1 monclient: found mon.noname-a 2023-09-30T02:10:37.094 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.092+0000 7fb69dd9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:10:37.095 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.092+0000 7fb69dd9b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.59:3300/0 2023-09-30T02:10:37.095 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.092+0000 7fb69dd9b700 1 -- 172.21.15.59:0/1124776327 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6a0142500 con 0x7fb6a00a5f40 2023-09-30T02:10:37.095 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.095+0000 7fb69dd9b700 1 --2- 172.21.15.59:0/1124776327 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb6a00a5f40 0x7fb6a0141b80 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb6940098e0 tx=0x7fb6940091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4035492853d7e605 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:10:37.098 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.098+0000 7fb69d59a700 1 -- 172.21.15.59:0/1124776327 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb694011630 con 0x7fb6a00a5f40 2023-09-30T02:10:37.098 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.098+0000 7fb69d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:10:37.098 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.098+0000 7fb69d59a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T02:10:37.098 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.098+0000 7fb69d59a700 10 monclient: dump: 2023-09-30T02:10:37.098 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:10:37.098 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:37.099 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:37.099 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:37.099 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:10:37.099 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:37.099 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:10:37.099 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:10:37.099 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:10:37.099 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:37.099 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.098+0000 7fb69d59a700 10 monclient: _finish_auth 0 2023-09-30T02:10:37.099 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.098+0000 7fb69d59a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:10:37.099 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.098+0000 7fb6a4b24700 5 monclient: authenticate success, global_id 4734 2023-09-30T02:10:37.099 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.098+0000 7fb6a4b24700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:10:37.100 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.100+0000 7fb69d59a700 1 -- 172.21.15.59:0/1124776327 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb694011790 con 0x7fb6a00a5f40 2023-09-30T02:10:37.101 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.100+0000 7fb69d59a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:10:37.101 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.100+0000 7fb69d59a700 1 -- 172.21.15.59:0/1124776327 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb694011b10 con 0x7fb6a00a5f40 2023-09-30T02:10:37.101 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.100+0000 7fb6a4b24700 10 monclient: get_monmap_and_config success 2023-09-30T02:10:37.101 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.100+0000 7fb6a4b24700 4 set_mon_vals no callback set 2023-09-30T02:10:37.101 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.100+0000 7fb69d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:10:37.101 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.100+0000 7fb69d59a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T02:10:37.101 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.100+0000 7fb69d59a700 10 monclient: dump: 2023-09-30T02:10:37.101 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:10:37.101 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:37.101 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:37.101 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:37.102 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:10:37.102 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:37.102 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:10:37.102 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:10:37.102 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:10:37.102 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:37.105 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.101+0000 7fb6a4b24700 10 monclient: shutdown 2023-09-30T02:10:37.105 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.101+0000 7fb6a4b24700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:10:37.105 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.101+0000 7fb6a4b24700 1 -- 172.21.15.59:0/1124776327 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb6a00a5f40 msgr2=0x7fb6a0141b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:37.106 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.101+0000 7fb6a4b24700 1 --2- 172.21.15.59:0/1124776327 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb6a00a5f40 0x7fb6a0141b80 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fb6940098e0 tx=0x7fb6940091a0 comp rx=0 tx=0).stop 2023-09-30T02:10:37.106 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.101+0000 7fb6a4b24700 1 -- 172.21.15.59:0/1124776327 shutdown_connections 2023-09-30T02:10:37.106 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.101+0000 7fb6a4b24700 1 --2- 172.21.15.59:0/1124776327 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb6a00a5b70 0x7fb6a00b1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:37.106 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.101+0000 7fb6a4b24700 1 --2- 172.21.15.59:0/1124776327 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb6a00a5f40 0x7fb6a0141b80 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:37.106 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.101+0000 7fb6a4b24700 1 -- 172.21.15.59:0/1124776327 >> 172.21.15.59:0/1124776327 conn(0x7fb6a00631f0 msgr2=0x7fb6a00635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:10:37.106 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.101+0000 7fb6a4b24700 1 -- 172.21.15.59:0/1124776327 shutdown_connections 2023-09-30T02:10:37.106 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.101+0000 7fb6a4b24700 1 -- 172.21.15.59:0/1124776327 wait complete. 2023-09-30T02:10:37.106 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.102+0000 7fb6a4b24700 10 monclient: build_initial_monmap 2023-09-30T02:10:37.106 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.102+0000 7fb6a4b24700 10 monclient: monmap: 2023-09-30T02:10:37.106 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:10:37.106 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:10:37.106 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:10:37.107 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:10:37.107 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:10:37.107 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:10:37.107 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:10:37.107 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:10:37.107 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:10:37.108 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:37.108 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.102+0000 7fb6a4b24700 1 librados: starting msgr at 2023-09-30T02:10:37.108 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.102+0000 7fb6a4b24700 1 librados: starting objecter 2023-09-30T02:10:37.108 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.102+0000 7fb6a4b24700 1 Processor -- start 2023-09-30T02:10:37.108 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.102+0000 7fb6a4b24700 1 -- start start 2023-09-30T02:10:37.108 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.102+0000 7fb6a4b24700 1 librados: setting wanted keys 2023-09-30T02:10:37.108 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.102+0000 7fb6a4b24700 1 librados: calling monclient init 2023-09-30T02:10:37.108 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.102+0000 7fb6a4b24700 10 monclient: init 2023-09-30T02:10:37.108 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.103+0000 7fb6a4b24700 10 monclient: _reopen_session rank -1 2023-09-30T02:10:37.108 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.103+0000 7fb6a4b24700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T02:10:37.108 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.103+0000 7fb6a4b24700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb6a00a5b70 0x7fb6a00a7db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:37.108 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.103+0000 7fb6a4b24700 10 monclient(hunting): picked mon.noname-b con 0x7fb6a00a5b70 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:10:37.109 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.103+0000 7fb6a4b24700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb6a00a5f40 0x7fb6a00b5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:37.109 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.103+0000 7fb6a4b24700 10 monclient(hunting): picked mon.noname-a con 0x7fb6a00a5f40 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:10:37.109 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.103+0000 7fb6a4b24700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb6a01420c0 0x7fb6a014de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:37.109 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.103+0000 7fb6a4b24700 10 monclient(hunting): picked mon.noname-c con 0x7fb6a01420c0 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:10:37.109 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.103+0000 7fb6a4b24700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:37.109 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.103+0000 7fb6a4b24700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a00a9110 con 0x7fb6a00a5f40 2023-09-30T02:10:37.109 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.103+0000 7fb6a4b24700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:37.109 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.103+0000 7fb6a4b24700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a014d8f0 con 0x7fb6a00a5b70 2023-09-30T02:10:37.109 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.103+0000 7fb6a4b24700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:37.109 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.103+0000 7fb6a4b24700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a00a82f0 con 0x7fb6a01420c0 2023-09-30T02:10:37.109 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.103+0000 7fb6a4b24700 10 monclient(hunting): _renew_subs 2023-09-30T02:10:37.109 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.103+0000 7fb6a4b24700 10 monclient(hunting): authenticate will time out at 2082.406363s 2023-09-30T02:10:37.110 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.103+0000 7fb69ed9d700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb6a01420c0 0x7fb6a014de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:37.110 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.103+0000 7fb69e59c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb6a00a5b70 0x7fb6a00a7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:37.110 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.104+0000 7fb69dd9b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb6a00a5f40 0x7fb6a00b5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:37.110 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.104+0000 7fb69e59c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb6a00a5b70 0x7fb6a00a7db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:34932/0 (socket says 172.21.15.59:34932) 2023-09-30T02:10:37.110 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.104+0000 7fb69e59c700 1 -- 172.21.15.59:0/2490507988 learned_addr learned my addr 172.21.15.59:0/2490507988 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:10:37.110 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.104+0000 7fb69e59c700 10 monclient(hunting): get_auth_request con 0x7fb6a00a5b70 auth_method 0 2023-09-30T02:10:37.110 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.104+0000 7fb69e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:10:37.110 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.104+0000 7fb69e59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:37.110 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.104+0000 7fb69e59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:37.110 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.104+0000 7fb69ed9d700 10 monclient(hunting): get_auth_request con 0x7fb6a01420c0 auth_method 0 2023-09-30T02:10:37.111 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.104+0000 7fb69ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:10:37.111 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.104+0000 7fb69ed9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:37.111 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.104+0000 7fb69ed9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:37.111 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.104+0000 7fb69dd9b700 10 monclient(hunting): get_auth_request con 0x7fb6a00a5f40 auth_method 0 2023-09-30T02:10:37.111 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.104+0000 7fb69dd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:10:37.111 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.104+0000 7fb69dd9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:37.111 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.104+0000 7fb69dd9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:37.111 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.104+0000 7fb69ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:10:37.111 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.104+0000 7fb69ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:10:37.111 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.104+0000 7fb69ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:10:37.111 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.105+0000 7fb69e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:10:37.111 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.105+0000 7fb69e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:10:37.111 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.105+0000 7fb69e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:10:37.112 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.105+0000 7fb69dd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:10:37.112 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.105+0000 7fb69dd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:10:37.112 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.105+0000 7fb69dd9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:10:37.112 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.109+0000 7fb69e59c700 10 monclient(hunting): handle_auth_done global_id 4768 payload 824 2023-09-30T02:10:37.113 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.109+0000 7fb69e59c700 1 -- 172.21.15.59:0/2490507988 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb6a01420c0 msgr2=0x7fb6a014de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:37.113 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.109+0000 7fb69e59c700 1 --2- 172.21.15.59:0/2490507988 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb6a01420c0 0x7fb6a014de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:37.113 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.109+0000 7fb69e59c700 1 -- 172.21.15.59:0/2490507988 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb6a00a5f40 msgr2=0x7fb6a00b5240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:37.113 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.109+0000 7fb69e59c700 1 --2- 172.21.15.59:0/2490507988 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb6a00a5f40 0x7fb6a00b5240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:37.113 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.109+0000 7fb69e59c700 10 monclient: _finish_hunting 0 2023-09-30T02:10:37.113 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.109+0000 7fb69e59c700 1 monclient: found mon.noname-b 2023-09-30T02:10:37.113 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.109+0000 7fb69e59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:10:37.113 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.109+0000 7fb69e59c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:10:37.113 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.109+0000 7fb69e59c700 1 -- 172.21.15.59:0/2490507988 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb694009420 con 0x7fb6a00a5b70 2023-09-30T02:10:37.113 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.112+0000 7fb69e59c700 1 --2- 172.21.15.59:0/2490507988 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb6a00a5b70 0x7fb6a00a7db0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fb69000ec80 tx=0x7fb6900091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:10:37.113 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.112+0000 7fb68e7fc700 10 client.?.objecter ms_handle_connect 0x7fb6a00a5b70 2023-09-30T02:10:37.113 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.112+0000 7fb68e7fc700 10 client.?.objecter resend_mon_ops 2023-09-30T02:10:37.118 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.117+0000 7fb68e7fc700 1 -- 172.21.15.59:0/2490507988 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb690009690 con 0x7fb6a00a5b70 2023-09-30T02:10:37.118 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.117+0000 7fb68e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:10:37.118 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.117+0000 7fb68e7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:10:37.118 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.117+0000 7fb68e7fc700 10 monclient: dump: 2023-09-30T02:10:37.118 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:10:37.118 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:37.118 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:37.118 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:37.118 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:10:37.118 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:37.119 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:10:37.119 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:10:37.119 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:10:37.119 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:37.119 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.117+0000 7fb68e7fc700 10 monclient: _finish_auth 0 2023-09-30T02:10:37.119 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.117+0000 7fb68e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:10:37.119 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.117+0000 7fb68e7fc700 1 -- 172.21.15.59:0/2490507988 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb69000c850 con 0x7fb6a00a5b70 2023-09-30T02:10:37.119 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.117+0000 7fb6a4b24700 5 monclient: authenticate success, global_id 4768 2023-09-30T02:10:37.119 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.117+0000 7fb6a4b24700 10 monclient: _renew_subs 2023-09-30T02:10:37.119 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.117+0000 7fb6a4b24700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:10:37.119 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.117+0000 7fb6a4b24700 1 -- 172.21.15.59:0/2490507988 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6a00b5780 con 0x7fb6a00a5b70 2023-09-30T02:10:37.119 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.117+0000 7fb68e7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:10:37.120 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.117+0000 7fb68e7fc700 1 -- 172.21.15.59:0/2490507988 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb69000cc10 con 0x7fb6a00a5b70 2023-09-30T02:10:37.120 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.117+0000 7fb68e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:10:37.120 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.117+0000 7fb68e7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:10:37.120 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.117+0000 7fb68e7fc700 10 monclient: dump: 2023-09-30T02:10:37.120 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:10:37.120 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:37.120 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:37.120 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:37.120 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:10:37.120 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:37.120 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:10:37.120 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:10:37.120 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:10:37.121 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:37.121 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.117+0000 7fb69cd99700 4 set_mon_vals no callback set 2023-09-30T02:10:37.121 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.117+0000 7fb6a4b24700 10 client.4768.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:10:37.121 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.117+0000 7fb6a4b24700 10 monclient: _renew_subs 2023-09-30T02:10:37.121 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.117+0000 7fb6a4b24700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:10:37.121 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.117+0000 7fb6a4b24700 1 -- 172.21.15.59:0/2490507988 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6a00b2ac0 con 0x7fb6a00a5b70 2023-09-30T02:10:37.121 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.117+0000 7fb6a4b24700 1 librados: init done 2023-09-30T02:10:37.121 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.118+0000 7fb6a4b24700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:10:37.122 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.118+0000 7fb6a4b24700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:10:37.122 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.118+0000 7fb6a4b24700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:10:37.122 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.118+0000 7fb6a4b24700 1 -- 172.21.15.59:0/2490507988 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6a014dc60 con 0x7fb6a00a5b70 2023-09-30T02:10:37.225 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.224+0000 7fb68e7fc700 1 -- 172.21.15.59:0/2490507988 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fb690013470 con 0x7fb6a00a5b70 2023-09-30T02:10:37.225 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.225+0000 7fb68e7fc700 1 --2- 172.21.15.59:0/2490507988 >> v2:172.21.15.110:6800/115870 conn(0x7fb680068910 0x7fb68006adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:37.226 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.225+0000 7fb68e7fc700 1 -- 172.21.15.59:0/2490507988 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fb6900848b0 con 0x7fb6a00a5b70 2023-09-30T02:10:37.226 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.225+0000 7fb68e7fc700 10 client.4768.objecter ms_dispatch 0x7fb6a0001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:10:37.226 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.225+0000 7fb68e7fc700 3 client.4768.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:10:37.226 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.225+0000 7fb68e7fc700 3 client.4768.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:10:37.226 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.225+0000 7fb68e7fc700 20 client.4768.objecter dump_active .. 0 homeless 2023-09-30T02:10:37.226 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.225+0000 7fb69dd9b700 1 --2- 172.21.15.59:0/2490507988 >> v2:172.21.15.110:6800/115870 conn(0x7fb680068910 0x7fb68006adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:37.226 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.225+0000 7fb69dd9b700 10 monclient: get_auth_request con 0x7fb680068910 auth_method 0 2023-09-30T02:10:37.227 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.225+0000 7fb69dd9b700 1 --2- 172.21.15.59:0/2490507988 >> v2:172.21.15.110:6800/115870 conn(0x7fb680068910 0x7fb68006adc0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb6940091e0 tx=0x7fb694017000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:10:37.227 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.225+0000 7fb68e7fc700 10 client.4768.objecter ms_handle_connect 0x7fb680068910 2023-09-30T02:10:37.384 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.383+0000 7fb68e7fc700 1 -- 172.21.15.59:0/2490507988 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb690052150 con 0x7fb6a00a5b70 2023-09-30T02:10:37.384 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.383+0000 7fb68e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:10:37.384 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.383+0000 7fb68e7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:10:37.635 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.635+0000 7fb6a4b24700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:10:37.636 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.635+0000 7fb6a4b24700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:10:37.636 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.635+0000 7fb6a4b24700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:10:37.636 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.635+0000 7fb6a4b24700 1 -- 172.21.15.59:0/2490507988 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fb6a0064a00 con 0x7fb6a00a5b70 2023-09-30T02:10:37.642 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.642+0000 7fb68e7fc700 1 -- 172.21.15.59:0/2490507988 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fb690055770 con 0x7fb6a00a5b70 2023-09-30T02:10:37.642 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.642+0000 7fb68e7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:10:37.643 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.642+0000 7fb68e7fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:10:37.643 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:10:37.643 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:10:37.645 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.644+0000 7fb6a4b24700 10 librados: watch_flush enter 2023-09-30T02:10:37.645 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.644+0000 7fb6a4b24700 10 librados: watch_flush exit 2023-09-30T02:10:37.646 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.644+0000 7fb6a4b24700 20 client.4768.objecter shutdown clearing up homeless session... 2023-09-30T02:10:37.646 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.644+0000 7fb6a4b24700 10 client.4768.objecter successfully canceled tick 2023-09-30T02:10:37.646 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.644+0000 7fb6a4b24700 1 -- 172.21.15.59:0/2490507988 >> v2:172.21.15.110:6800/115870 conn(0x7fb680068910 msgr2=0x7fb68006adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:37.646 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.644+0000 7fb6a4b24700 1 --2- 172.21.15.59:0/2490507988 >> v2:172.21.15.110:6800/115870 conn(0x7fb680068910 0x7fb68006adc0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb6940091e0 tx=0x7fb694017000 comp rx=0 tx=0).stop 2023-09-30T02:10:37.646 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.644+0000 7fb6a4b24700 10 monclient: shutdown 2023-09-30T02:10:37.646 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.644+0000 7fb6a4b24700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:10:37.646 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.644+0000 7fb6a4b24700 1 -- 172.21.15.59:0/2490507988 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb6a00a5b70 msgr2=0x7fb6a00a7db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:37.646 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.644+0000 7fb6a4b24700 1 --2- 172.21.15.59:0/2490507988 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb6a00a5b70 0x7fb6a00a7db0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fb69000ec80 tx=0x7fb6900091a0 comp rx=0 tx=0).stop 2023-09-30T02:10:37.646 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.644+0000 7fb6a4b24700 1 -- 172.21.15.59:0/2490507988 shutdown_connections 2023-09-30T02:10:37.647 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.644+0000 7fb6a4b24700 1 --2- 172.21.15.59:0/2490507988 >> v2:172.21.15.110:6800/115870 conn(0x7fb680068910 0x7fb68006adc0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:37.647 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.644+0000 7fb6a4b24700 1 --2- 172.21.15.59:0/2490507988 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fb6a00a5b70 0x7fb6a00a7db0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:37.647 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.644+0000 7fb6a4b24700 1 --2- 172.21.15.59:0/2490507988 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fb6a01420c0 0x7fb6a014de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:37.647 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.645+0000 7fb6a4b24700 1 --2- 172.21.15.59:0/2490507988 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb6a00a5f40 0x7fb6a00b5240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:37.648 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.645+0000 7fb6a4b24700 1 -- 172.21.15.59:0/2490507988 >> 172.21.15.59:0/2490507988 conn(0x7fb6a00631f0 msgr2=0x7fb6a00b1330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:10:37.648 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.645+0000 7fb6a4b24700 1 -- 172.21.15.59:0/2490507988 shutdown_connections 2023-09-30T02:10:37.648 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.645+0000 7fb6a4b24700 1 -- 172.21.15.59:0/2490507988 wait complete. 2023-09-30T02:10:37.648 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:37.645+0000 7fb6a4b24700 1 librados: shutdown 2023-09-30T02:10:40.660 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:10:40.797 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.797+0000 7fe875582700 10 monclient: get_monmap_and_config 2023-09-30T02:10:40.798 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.797+0000 7fe875582700 10 monclient: build_initial_monmap 2023-09-30T02:10:40.798 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.797+0000 7fe875582700 10 monclient: monmap: 2023-09-30T02:10:40.798 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:10:40.798 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:40.798 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:10:40.798813+0000 2023-09-30T02:10:40.798 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:10:40.798813+0000 2023-09-30T02:10:40.798 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:10:40.798 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:40.798 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:10:40.798 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:10:40.799 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:10:40.799 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:10:40.799 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:10:40.799 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:40.801 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.798+0000 7fe875582700 1 Processor -- start 2023-09-30T02:10:40.801 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.798+0000 7fe875582700 1 -- start start 2023-09-30T02:10:40.801 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.798+0000 7fe875582700 10 monclient: init 2023-09-30T02:10:40.801 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.798+0000 7fe875582700 10 monclient: _reopen_session rank -1 2023-09-30T02:10:40.801 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.798+0000 7fe875582700 10 monclient: _add_conns ranks=[3,2,1,0,4] 2023-09-30T02:10:40.801 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.798+0000 7fe875582700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fe8700b1890 0x7fe8700a5df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:40.801 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.798+0000 7fe875582700 10 monclient(hunting): picked mon.noname-a con 0x7fe8700b1890 addr v2:172.21.15.59:3300/0 2023-09-30T02:10:40.801 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.798+0000 7fe875582700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe8700b10a0 0x7fe87013e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:40.802 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.798+0000 7fe875582700 10 monclient(hunting): picked mon.noname-b con 0x7fe8700b10a0 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:10:40.802 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.798+0000 7fe875582700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe8700b1c60 addr v1:172.21.15.110:6789/0 2023-09-30T02:10:40.802 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.798+0000 7fe875582700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe87013e200 con 0x7fe8700b1c60 2023-09-30T02:10:40.802 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.798+0000 7fe875582700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:40.802 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.798+0000 7fe875582700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8700a7350 con 0x7fe8700b1890 2023-09-30T02:10:40.802 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.798+0000 7fe875582700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:40.802 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.798+0000 7fe875582700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe870142280 con 0x7fe8700b10a0 2023-09-30T02:10:40.802 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.798+0000 7fe875582700 10 monclient(hunting): _renew_subs 2023-09-30T02:10:40.802 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.798+0000 7fe875582700 10 monclient(hunting): authenticate will time out at 2086.101866s 2023-09-30T02:10:40.802 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.799+0000 7fe86e7fc700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe8700b10a0 0x7fe87013e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:40.802 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.799+0000 7fe86f7fe700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fe8700b1c60 0x7fe870141b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.59:34820/0 (socket says 172.21.15.59:34820) 2023-09-30T02:10:40.803 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.799+0000 7fe86f7fe700 1 -- 172.21.15.59:0/643530282 learned_addr learned my addr 172.21.15.59:0/643530282 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:10:40.803 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.799+0000 7fe86effd700 1 --2- 172.21.15.59:0/643530282 >> v2:172.21.15.59:3300/0 conn(0x7fe8700b1890 0x7fe8700a5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:40.803 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.800+0000 7fe86e7fc700 10 monclient(hunting): get_auth_request con 0x7fe8700b10a0 auth_method 0 2023-09-30T02:10:40.803 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.800+0000 7fe86e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:10:40.803 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.800+0000 7fe86e7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:40.803 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.800+0000 7fe86e7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:40.803 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.800+0000 7fe86effd700 10 monclient(hunting): get_auth_request con 0x7fe8700b1890 auth_method 0 2023-09-30T02:10:40.803 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.800+0000 7fe86effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:10:40.803 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.800+0000 7fe86effd700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:40.803 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.800+0000 7fe86effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:40.803 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.800+0000 7fe86e7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:10:40.804 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.800+0000 7fe86e7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:10:40.804 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.800+0000 7fe86e7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:10:40.804 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.800+0000 7fe86effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:10:40.804 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.800+0000 7fe86effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:10:40.804 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.800+0000 7fe86effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:10:40.804 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.803+0000 7fe86e7fc700 10 monclient(hunting): handle_auth_done global_id 4667 payload 274 2023-09-30T02:10:40.805 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.803+0000 7fe86e7fc700 1 -- 172.21.15.59:0/643530282 >> v2:172.21.15.59:3300/0 conn(0x7fe8700b1890 msgr2=0x7fe8700a5df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:10:40.805 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.803+0000 7fe86e7fc700 1 --2- 172.21.15.59:0/643530282 >> v2:172.21.15.59:3300/0 conn(0x7fe8700b1890 0x7fe8700a5df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:40.805 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.803+0000 7fe86e7fc700 1 -- 172.21.15.59:0/643530282 >> v1:172.21.15.110:6789/0 conn(0x7fe8700b1c60 legacy=0x7fe870141b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:40.805 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.803+0000 7fe86e7fc700 10 monclient: _finish_hunting 0 2023-09-30T02:10:40.805 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.803+0000 7fe86e7fc700 1 monclient: found mon.noname-b 2023-09-30T02:10:40.805 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.803+0000 7fe86e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:10:40.805 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.803+0000 7fe86e7fc700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.59:3301/0 2023-09-30T02:10:40.805 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.803+0000 7fe86e7fc700 1 -- 172.21.15.59:0/643530282 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8701423f0 con 0x7fe8700b10a0 2023-09-30T02:10:40.806 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.805+0000 7fe86e7fc700 1 --2- 172.21.15.59:0/643530282 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe8700b10a0 0x7fe87013e7e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fe8640098e0 tx=0x7fe8640091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d20a30bbedeb569 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:10:40.810 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.809+0000 7fe86dffb700 1 -- 172.21.15.59:0/643530282 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe86400b640 con 0x7fe8700b10a0 2023-09-30T02:10:40.810 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.809+0000 7fe86dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:10:40.810 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.810+0000 7fe86dffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:10:40.810 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.810+0000 7fe86dffb700 10 monclient: dump: 2023-09-30T02:10:40.810 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:10:40.811 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:40.811 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:40.811 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:40.811 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:10:40.811 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:40.811 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:10:40.811 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:10:40.811 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:10:40.811 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:40.811 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.810+0000 7fe86dffb700 10 monclient: _finish_auth 0 2023-09-30T02:10:40.811 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.810+0000 7fe86dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:10:40.811 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.810+0000 7fe875582700 5 monclient: authenticate success, global_id 4667 2023-09-30T02:10:40.812 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.810+0000 7fe86dffb700 1 -- 172.21.15.59:0/643530282 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe86400b7a0 con 0x7fe8700b10a0 2023-09-30T02:10:40.812 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.810+0000 7fe875582700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:10:40.812 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.810+0000 7fe86dffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:10:40.812 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.810+0000 7fe875582700 10 monclient: get_monmap_and_config success 2023-09-30T02:10:40.812 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.810+0000 7fe875582700 4 set_mon_vals no callback set 2023-09-30T02:10:40.812 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.810+0000 7fe86dffb700 1 -- 172.21.15.59:0/643530282 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe86400bb40 con 0x7fe8700b10a0 2023-09-30T02:10:40.812 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.810+0000 7fe86dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:10:40.812 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.810+0000 7fe86dffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:10:40.812 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.810+0000 7fe86dffb700 10 monclient: dump: 2023-09-30T02:10:40.812 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:10:40.812 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:40.813 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:40.813 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:40.813 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:10:40.813 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:40.813 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:10:40.813 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:10:40.813 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:10:40.813 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:40.818 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.810+0000 7fe875582700 10 monclient: shutdown 2023-09-30T02:10:40.818 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.810+0000 7fe875582700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:10:40.818 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.810+0000 7fe875582700 1 -- 172.21.15.59:0/643530282 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe8700b10a0 msgr2=0x7fe87013e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:40.818 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.810+0000 7fe875582700 1 --2- 172.21.15.59:0/643530282 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe8700b10a0 0x7fe87013e7e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fe8640098e0 tx=0x7fe8640091a0 comp rx=0 tx=0).stop 2023-09-30T02:10:40.818 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.810+0000 7fe875582700 1 -- 172.21.15.59:0/643530282 shutdown_connections 2023-09-30T02:10:40.819 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.810+0000 7fe875582700 1 --2- 172.21.15.59:0/643530282 >> v2:172.21.15.59:3300/0 conn(0x7fe8700b1890 0x7fe8700a5df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:40.819 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.810+0000 7fe875582700 1 --2- 172.21.15.59:0/643530282 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe8700b10a0 0x7fe87013e7e0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:40.819 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.810+0000 7fe875582700 1 -- 172.21.15.59:0/643530282 >> 172.21.15.59:0/643530282 conn(0x7fe8700631f0 msgr2=0x7fe8700635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:10:40.819 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.810+0000 7fe875582700 1 -- 172.21.15.59:0/643530282 shutdown_connections 2023-09-30T02:10:40.819 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.810+0000 7fe875582700 1 -- 172.21.15.59:0/643530282 wait complete. 2023-09-30T02:10:40.819 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.812+0000 7fe875582700 10 monclient: build_initial_monmap 2023-09-30T02:10:40.819 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.812+0000 7fe875582700 10 monclient: monmap: 2023-09-30T02:10:40.819 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:10:40.819 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:10:40.819 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:10:40.819 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:10:40.819 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:10:40.820 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:10:40.820 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:10:40.820 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:10:40.820 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:10:40.820 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:40.820 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.812+0000 7fe875582700 1 librados: starting msgr at 2023-09-30T02:10:40.820 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.812+0000 7fe875582700 1 librados: starting objecter 2023-09-30T02:10:40.820 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.812+0000 7fe875582700 1 Processor -- start 2023-09-30T02:10:40.820 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.812+0000 7fe875582700 1 -- start start 2023-09-30T02:10:40.820 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.812+0000 7fe875582700 1 librados: setting wanted keys 2023-09-30T02:10:40.820 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.812+0000 7fe875582700 1 librados: calling monclient init 2023-09-30T02:10:40.820 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.812+0000 7fe875582700 10 monclient: init 2023-09-30T02:10:40.821 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.813+0000 7fe875582700 10 monclient: _reopen_session rank -1 2023-09-30T02:10:40.821 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.813+0000 7fe875582700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T02:10:40.821 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.813+0000 7fe875582700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe8700b10a0 0x7fe8700ab4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:40.821 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.813+0000 7fe875582700 10 monclient(hunting): picked mon.noname-b con 0x7fe8700b10a0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:10:40.821 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.813+0000 7fe875582700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe8700b1890 0x7fe870150fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:40.821 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.813+0000 7fe875582700 10 monclient(hunting): picked mon.noname-c con 0x7fe8700b1890 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:10:40.821 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.813+0000 7fe875582700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8700b1c60 0x7fe87014f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:40.821 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.813+0000 7fe875582700 10 monclient(hunting): picked mon.noname-a con 0x7fe8700b1c60 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:10:40.821 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.813+0000 7fe875582700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:40.821 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.813+0000 7fe875582700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8700a95f0 con 0x7fe8700b1c60 2023-09-30T02:10:40.821 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.813+0000 7fe875582700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:40.822 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.813+0000 7fe875582700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe870150b10 con 0x7fe8700b10a0 2023-09-30T02:10:40.822 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.813+0000 7fe875582700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:40.822 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.813+0000 7fe875582700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8700acfe0 con 0x7fe8700b1890 2023-09-30T02:10:40.822 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.813+0000 7fe875582700 10 monclient(hunting): _renew_subs 2023-09-30T02:10:40.822 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.813+0000 7fe875582700 10 monclient(hunting): authenticate will time out at 2086.116548s 2023-09-30T02:10:40.822 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.814+0000 7fe86e7fc700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe8700b1890 0x7fe870150fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:40.822 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.814+0000 7fe86f7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8700b1c60 0x7fe87014f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:40.822 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.814+0000 7fe86effd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe8700b10a0 0x7fe8700ab4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:40.822 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.814+0000 7fe86e7fc700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe8700b1890 0x7fe870150fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:50808/0 (socket says 172.21.15.59:50808) 2023-09-30T02:10:40.822 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.814+0000 7fe86e7fc700 1 -- 172.21.15.59:0/2732506929 learned_addr learned my addr 172.21.15.59:0/2732506929 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:10:40.822 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.814+0000 7fe86e7fc700 10 monclient(hunting): get_auth_request con 0x7fe8700b1890 auth_method 0 2023-09-30T02:10:40.822 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.814+0000 7fe86e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:10:40.823 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.814+0000 7fe86e7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:40.823 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.814+0000 7fe86e7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:40.823 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.814+0000 7fe86f7fe700 10 monclient(hunting): get_auth_request con 0x7fe8700b1c60 auth_method 0 2023-09-30T02:10:40.823 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.814+0000 7fe86f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:10:40.823 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.814+0000 7fe86f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:40.823 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.814+0000 7fe86f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:40.823 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.814+0000 7fe86effd700 10 monclient(hunting): get_auth_request con 0x7fe8700b10a0 auth_method 0 2023-09-30T02:10:40.823 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.814+0000 7fe86effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:10:40.823 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.814+0000 7fe86effd700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:40.823 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.814+0000 7fe86effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:40.823 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.815+0000 7fe86e7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:10:40.823 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.815+0000 7fe86e7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:10:40.824 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.815+0000 7fe86e7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:10:40.824 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.815+0000 7fe86f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:10:40.824 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.815+0000 7fe86f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:10:40.824 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.815+0000 7fe86f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:10:40.824 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.815+0000 7fe86effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:10:40.824 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.815+0000 7fe86effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:10:40.824 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.815+0000 7fe86effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:10:40.826 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.818+0000 7fe86e7fc700 10 monclient(hunting): handle_auth_done global_id 4670 payload 824 2023-09-30T02:10:40.826 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.818+0000 7fe86e7fc700 1 -- 172.21.15.59:0/2732506929 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe8700b10a0 msgr2=0x7fe8700ab4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:40.827 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.818+0000 7fe86e7fc700 1 --2- 172.21.15.59:0/2732506929 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe8700b10a0 0x7fe8700ab4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:40.827 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.818+0000 7fe86e7fc700 1 -- 172.21.15.59:0/2732506929 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8700b1c60 msgr2=0x7fe87014f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:40.827 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.818+0000 7fe86e7fc700 1 --2- 172.21.15.59:0/2732506929 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8700b1c60 0x7fe87014f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:40.827 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.818+0000 7fe86e7fc700 10 monclient: _finish_hunting 0 2023-09-30T02:10:40.827 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.818+0000 7fe86e7fc700 1 monclient: found mon.noname-c 2023-09-30T02:10:40.827 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.818+0000 7fe86e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:10:40.827 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.818+0000 7fe86e7fc700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.59:3301/0 2023-09-30T02:10:40.827 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.818+0000 7fe86e7fc700 1 -- 172.21.15.59:0/2732506929 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe864009420 con 0x7fe8700b1890 2023-09-30T02:10:40.827 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.821+0000 7fe86e7fc700 1 --2- 172.21.15.59:0/2732506929 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe8700b1890 0x7fe870150fa0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe8640093f0 tx=0x7fe864017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:10:40.827 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.821+0000 7fe856ffd700 10 client.?.objecter ms_handle_connect 0x7fe8700b1890 2023-09-30T02:10:40.827 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.821+0000 7fe856ffd700 10 client.?.objecter resend_mon_ops 2023-09-30T02:10:40.827 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.825+0000 7fe856ffd700 1 -- 172.21.15.59:0/2732506929 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe86401ba70 con 0x7fe8700b1890 2023-09-30T02:10:40.828 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.825+0000 7fe856ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:10:40.828 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.825+0000 7fe856ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:10:40.828 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.825+0000 7fe856ffd700 10 monclient: dump: 2023-09-30T02:10:40.828 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:10:40.828 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:40.828 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:40.828 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:40.828 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:10:40.828 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:40.828 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:10:40.828 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:10:40.828 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:10:40.829 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:40.829 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.825+0000 7fe856ffd700 10 monclient: _finish_auth 0 2023-09-30T02:10:40.829 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.825+0000 7fe856ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:10:40.829 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.825+0000 7fe856ffd700 1 -- 172.21.15.59:0/2732506929 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe86401bbf0 con 0x7fe8700b1890 2023-09-30T02:10:40.829 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.825+0000 7fe856ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:10:40.829 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.825+0000 7fe856ffd700 1 -- 172.21.15.59:0/2732506929 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe864025420 con 0x7fe8700b1890 2023-09-30T02:10:40.829 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.825+0000 7fe856ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:10:40.829 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.825+0000 7fe856ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:10:40.829 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.825+0000 7fe856ffd700 10 monclient: dump: 2023-09-30T02:10:40.829 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:10:40.829 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:40.829 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:40.829 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:40.830 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:10:40.830 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:40.830 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:10:40.830 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:10:40.830 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:10:40.830 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:40.830 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.825+0000 7fe875582700 5 monclient: authenticate success, global_id 4670 2023-09-30T02:10:40.830 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.825+0000 7fe86d7fa700 4 set_mon_vals no callback set 2023-09-30T02:10:40.830 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.825+0000 7fe875582700 10 monclient: _renew_subs 2023-09-30T02:10:40.830 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.825+0000 7fe875582700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:10:40.830 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.825+0000 7fe875582700 1 -- 172.21.15.59:0/2732506929 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe870150750 con 0x7fe8700b1890 2023-09-30T02:10:40.831 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.825+0000 7fe875582700 10 client.4670.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:10:40.831 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.825+0000 7fe875582700 10 monclient: _renew_subs 2023-09-30T02:10:40.831 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.825+0000 7fe875582700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:10:40.831 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.825+0000 7fe875582700 1 -- 172.21.15.59:0/2732506929 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8700a9d40 con 0x7fe8700b1890 2023-09-30T02:10:40.831 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.825+0000 7fe875582700 1 librados: init done 2023-09-30T02:10:40.831 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.826+0000 7fe875582700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:10:40.831 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.826+0000 7fe875582700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:10:40.831 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.826+0000 7fe875582700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:10:40.831 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:40.826+0000 7fe875582700 1 -- 172.21.15.59:0/2732506929 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe870072b30 con 0x7fe8700b1890 2023-09-30T02:10:41.132 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:41.131+0000 7fe856ffd700 1 -- 172.21.15.59:0/2732506929 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fe864023070 con 0x7fe8700b1890 2023-09-30T02:10:41.132 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:41.131+0000 7fe856ffd700 1 --2- 172.21.15.59:0/2732506929 >> v2:172.21.15.110:6800/115870 conn(0x7fe85c068960 0x7fe85c06ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:41.132 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:41.131+0000 7fe856ffd700 1 -- 172.21.15.59:0/2732506929 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fe864085250 con 0x7fe8700b1890 2023-09-30T02:10:41.132 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:41.131+0000 7fe856ffd700 10 client.4670.objecter ms_dispatch 0x7fe870001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:10:41.132 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:41.131+0000 7fe856ffd700 3 client.4670.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:10:41.132 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:41.131+0000 7fe856ffd700 3 client.4670.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:10:41.132 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:41.131+0000 7fe856ffd700 20 client.4670.objecter dump_active .. 0 homeless 2023-09-30T02:10:41.133 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:41.131+0000 7fe856ffd700 1 -- 172.21.15.59:0/2732506929 <== mon.2 v2:172.21.15.59:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe8640857d0 con 0x7fe8700b1890 2023-09-30T02:10:41.133 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:41.131+0000 7fe856ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:10:41.133 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:41.131+0000 7fe856ffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:10:41.133 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:41.131+0000 7fe86effd700 1 --2- 172.21.15.59:0/2732506929 >> v2:172.21.15.110:6800/115870 conn(0x7fe85c068960 0x7fe85c06ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:41.133 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:41.131+0000 7fe86effd700 10 monclient: get_auth_request con 0x7fe85c068960 auth_method 0 2023-09-30T02:10:41.133 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:41.132+0000 7fe856ffd700 10 client.4670.objecter ms_handle_connect 0x7fe85c068960 2023-09-30T02:10:41.133 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:41.132+0000 7fe86effd700 1 --2- 172.21.15.59:0/2732506929 >> v2:172.21.15.110:6800/115870 conn(0x7fe85c068960 0x7fe85c06ae10 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe860009db0 tx=0x7fe8600091a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:10:41.388 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:41.388+0000 7fe875582700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:10:41.389 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:41.388+0000 7fe875582700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:10:41.389 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:41.388+0000 7fe875582700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:10:41.389 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:41.388+0000 7fe875582700 1 -- 172.21.15.59:0/2732506929 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe870150910 con 0x7fe8700b1890 2023-09-30T02:10:41.394 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:41.394+0000 7fe856ffd700 1 -- 172.21.15.59:0/2732506929 <== mon.2 v2:172.21.15.59:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fe864052c70 con 0x7fe8700b1890 2023-09-30T02:10:41.394 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:41.394+0000 7fe856ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:10:41.394 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:41.394+0000 7fe856ffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:10:41.394 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:10:41.394 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:10:41.399 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:41.397+0000 7fe875582700 10 librados: watch_flush enter 2023-09-30T02:10:41.400 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:41.397+0000 7fe875582700 10 librados: watch_flush exit 2023-09-30T02:10:41.400 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:41.397+0000 7fe875582700 20 client.4670.objecter shutdown clearing up homeless session... 2023-09-30T02:10:41.400 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:41.397+0000 7fe875582700 10 client.4670.objecter successfully canceled tick 2023-09-30T02:10:41.400 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:41.397+0000 7fe875582700 1 -- 172.21.15.59:0/2732506929 >> v2:172.21.15.110:6800/115870 conn(0x7fe85c068960 msgr2=0x7fe85c06ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:41.400 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:41.397+0000 7fe875582700 1 --2- 172.21.15.59:0/2732506929 >> v2:172.21.15.110:6800/115870 conn(0x7fe85c068960 0x7fe85c06ae10 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe860009db0 tx=0x7fe8600091a0 comp rx=0 tx=0).stop 2023-09-30T02:10:41.401 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:41.397+0000 7fe875582700 10 monclient: shutdown 2023-09-30T02:10:41.401 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:41.397+0000 7fe875582700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:10:41.401 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:41.397+0000 7fe875582700 1 -- 172.21.15.59:0/2732506929 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe8700b1890 msgr2=0x7fe870150fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:41.401 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:41.397+0000 7fe875582700 1 --2- 172.21.15.59:0/2732506929 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe8700b1890 0x7fe870150fa0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe8640093f0 tx=0x7fe864017000 comp rx=0 tx=0).stop 2023-09-30T02:10:41.401 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:41.398+0000 7fe875582700 1 -- 172.21.15.59:0/2732506929 shutdown_connections 2023-09-30T02:10:41.401 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:41.398+0000 7fe875582700 1 --2- 172.21.15.59:0/2732506929 >> v2:172.21.15.110:6800/115870 conn(0x7fe85c068960 0x7fe85c06ae10 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:41.402 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:41.398+0000 7fe875582700 1 --2- 172.21.15.59:0/2732506929 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe8700b10a0 0x7fe8700ab4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:41.402 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:41.398+0000 7fe875582700 1 --2- 172.21.15.59:0/2732506929 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8700b1c60 0x7fe87014f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:41.402 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:41.398+0000 7fe875582700 1 --2- 172.21.15.59:0/2732506929 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe8700b1890 0x7fe870150fa0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:41.402 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:41.398+0000 7fe875582700 1 -- 172.21.15.59:0/2732506929 >> 172.21.15.59:0/2732506929 conn(0x7fe8700631f0 msgr2=0x7fe8700b26a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:10:41.402 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:41.398+0000 7fe875582700 1 -- 172.21.15.59:0/2732506929 shutdown_connections 2023-09-30T02:10:41.402 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:41.398+0000 7fe875582700 1 -- 172.21.15.59:0/2732506929 wait complete. 2023-09-30T02:10:41.403 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:41.398+0000 7fe875582700 1 librados: shutdown 2023-09-30T02:10:44.413 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:10:44.570 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.570+0000 7f06207dd700 10 monclient: get_monmap_and_config 2023-09-30T02:10:44.571 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.570+0000 7f06207dd700 10 monclient: build_initial_monmap 2023-09-30T02:10:44.571 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.570+0000 7f06207dd700 10 monclient: monmap: 2023-09-30T02:10:44.571 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:10:44.571 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:44.571 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:10:44.571960+0000 2023-09-30T02:10:44.571 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:10:44.571960+0000 2023-09-30T02:10:44.571 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:10:44.571 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:44.571 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:10:44.571 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:10:44.571 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:10:44.572 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:10:44.572 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:10:44.572 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:44.573 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.571+0000 7f06207dd700 1 Processor -- start 2023-09-30T02:10:44.574 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.571+0000 7f06207dd700 1 -- start start 2023-09-30T02:10:44.574 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.571+0000 7f06207dd700 10 monclient: init 2023-09-30T02:10:44.574 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.572+0000 7f06207dd700 10 monclient: _reopen_session rank -1 2023-09-30T02:10:44.574 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.572+0000 7f06207dd700 10 monclient: _add_conns ranks=[3,1,0,2,4] 2023-09-30T02:10:44.574 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.572+0000 7f06207dd700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f06180a5b70 0x7f06180b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:44.574 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.572+0000 7f06207dd700 10 monclient(hunting): picked mon.noname-a con 0x7f06180a5b70 addr v2:172.21.15.59:3300/0 2023-09-30T02:10:44.574 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.572+0000 7f06207dd700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f06180a5f40 addr v1:172.21.15.110:6789/0 2023-09-30T02:10:44.574 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.572+0000 7f06207dd700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0618142290 addr v1:172.21.15.59:6789/0 2023-09-30T02:10:44.574 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.572+0000 7f06207dd700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06180a75a0 con 0x7f0618142290 2023-09-30T02:10:44.574 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.572+0000 7f06207dd700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06180af3f0 con 0x7f06180a5f40 2023-09-30T02:10:44.575 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.572+0000 7f06207dd700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:44.575 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.572+0000 7f06207dd700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06180a63e0 con 0x7f06180a5b70 2023-09-30T02:10:44.575 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.572+0000 7f06207dd700 10 monclient(hunting): _renew_subs 2023-09-30T02:10:44.575 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.572+0000 7f06207dd700 10 monclient(hunting): authenticate will time out at 2089.875264s 2023-09-30T02:10:44.575 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.572+0000 7f061e579700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f06180a5b70 0x7f06180b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:44.575 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.573+0000 7f061dd78700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f06180a5f40 0x7f0618141b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.59:34834/0 (socket says 172.21.15.59:34834) 2023-09-30T02:10:44.575 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.573+0000 7f061dd78700 1 -- 172.21.15.59:0/55544849 learned_addr learned my addr 172.21.15.59:0/55544849 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:10:44.575 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.573+0000 7f061e579700 10 monclient(hunting): get_auth_request con 0x7f06180a5b70 auth_method 0 2023-09-30T02:10:44.576 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.573+0000 7f061e579700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:10:44.576 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.573+0000 7f061e579700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:44.576 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.573+0000 7f061e579700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:44.576 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.574+0000 7f061e579700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:10:44.577 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.574+0000 7f061e579700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:10:44.577 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.574+0000 7f061e579700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:10:44.577 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.576+0000 7f061d577700 1 -- 172.21.15.59:0/55544849 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1594484787 0 0) 0x7f06180af3f0 con 0x7f06180a5f40 2023-09-30T02:10:44.577 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.576+0000 7f061d577700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:44.577 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.576+0000 7f061d577700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:44.577 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.576+0000 7f061d577700 10 monclient(hunting): my global_id is 4777 2023-09-30T02:10:44.577 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.576+0000 7f061d577700 1 -- 172.21.15.59:0/55544849 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0604003570 con 0x7f06180a5f40 2023-09-30T02:10:44.578 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.577+0000 7f061e579700 10 monclient(hunting): handle_auth_done global_id 4746 payload 274 2023-09-30T02:10:44.578 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.577+0000 7f061e579700 1 -- 172.21.15.59:0/55544849 >> v1:172.21.15.110:6789/0 conn(0x7f06180a5f40 legacy=0x7f0618141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:44.578 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.577+0000 7f061e579700 1 -- 172.21.15.59:0/55544849 >> v1:172.21.15.59:6789/0 conn(0x7f0618142290 legacy=0x7f061813af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:44.578 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.577+0000 7f061e579700 10 monclient: _finish_hunting 0 2023-09-30T02:10:44.578 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.577+0000 7f061e579700 1 monclient: found mon.noname-a 2023-09-30T02:10:44.578 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.577+0000 7f061e579700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:10:44.578 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.577+0000 7f061e579700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.59:3300/0 2023-09-30T02:10:44.578 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.577+0000 7f061e579700 1 -- 172.21.15.59:0/55544849 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f061813b640 con 0x7f06180a5b70 2023-09-30T02:10:44.583 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.582+0000 7f061e579700 1 --2- 172.21.15.59:0/55544849 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06180a5b70 0x7f06180b1500 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f06140098e0 tx=0x7f06140091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52e8de40a10c9bc9 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:10:44.590 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.589+0000 7f061d577700 1 -- 172.21.15.59:0/55544849 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f061400b640 con 0x7f06180a5b70 2023-09-30T02:10:44.591 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.589+0000 7f061d577700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:10:44.591 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.589+0000 7f061d577700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T02:10:44.591 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.589+0000 7f061d577700 10 monclient: dump: 2023-09-30T02:10:44.591 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:10:44.591 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:44.591 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:44.591 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:44.591 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:10:44.592 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:44.592 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:10:44.592 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:10:44.592 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:10:44.592 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:44.592 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.589+0000 7f061d577700 10 monclient: _finish_auth 0 2023-09-30T02:10:44.592 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.589+0000 7f061d577700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:10:44.592 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.589+0000 7f061d577700 1 -- 172.21.15.59:0/55544849 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f061400b7a0 con 0x7f06180a5b70 2023-09-30T02:10:44.592 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.589+0000 7f06207dd700 5 monclient: authenticate success, global_id 4746 2023-09-30T02:10:44.593 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.589+0000 7f06207dd700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:10:44.593 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.589+0000 7f061d577700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:10:44.593 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.589+0000 7f06207dd700 10 monclient: get_monmap_and_config success 2023-09-30T02:10:44.593 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.589+0000 7f061d577700 1 -- 172.21.15.59:0/55544849 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f061400bb40 con 0x7f06180a5b70 2023-09-30T02:10:44.593 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.589+0000 7f06207dd700 4 set_mon_vals no callback set 2023-09-30T02:10:44.593 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.589+0000 7f061d577700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:10:44.593 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.589+0000 7f061d577700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T02:10:44.593 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.589+0000 7f061d577700 10 monclient: dump: 2023-09-30T02:10:44.593 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:10:44.594 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:44.594 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:44.594 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:44.594 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:10:44.594 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:44.594 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:10:44.594 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:10:44.595 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:10:44.595 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:44.595 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.590+0000 7f06207dd700 10 monclient: shutdown 2023-09-30T02:10:44.595 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.590+0000 7f06207dd700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:10:44.595 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.590+0000 7f06207dd700 1 -- 172.21.15.59:0/55544849 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06180a5b70 msgr2=0x7f06180b1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:44.595 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.590+0000 7f06207dd700 1 --2- 172.21.15.59:0/55544849 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06180a5b70 0x7f06180b1500 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f06140098e0 tx=0x7f06140091a0 comp rx=0 tx=0).stop 2023-09-30T02:10:44.595 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.590+0000 7f06207dd700 1 -- 172.21.15.59:0/55544849 shutdown_connections 2023-09-30T02:10:44.595 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.590+0000 7f06207dd700 1 --2- 172.21.15.59:0/55544849 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06180a5b70 0x7f06180b1500 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:44.596 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.590+0000 7f06207dd700 1 -- 172.21.15.59:0/55544849 >> 172.21.15.59:0/55544849 conn(0x7f06180631f0 msgr2=0x7f06180635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:10:44.596 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.590+0000 7f06207dd700 1 -- 172.21.15.59:0/55544849 shutdown_connections 2023-09-30T02:10:44.596 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.590+0000 7f06207dd700 1 -- 172.21.15.59:0/55544849 wait complete. 2023-09-30T02:10:44.599 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.590+0000 7f06207dd700 10 monclient: build_initial_monmap 2023-09-30T02:10:44.599 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.591+0000 7f06207dd700 10 monclient: monmap: 2023-09-30T02:10:44.600 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:10:44.600 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:10:44.600 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:10:44.600 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:10:44.600 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:10:44.600 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:10:44.600 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:10:44.600 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:10:44.600 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:10:44.600 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:44.601 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.591+0000 7f06207dd700 1 librados: starting msgr at 2023-09-30T02:10:44.601 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.591+0000 7f06207dd700 1 librados: starting objecter 2023-09-30T02:10:44.601 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.591+0000 7f06207dd700 1 Processor -- start 2023-09-30T02:10:44.601 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.591+0000 7f06207dd700 1 -- start start 2023-09-30T02:10:44.601 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.591+0000 7f06207dd700 1 librados: setting wanted keys 2023-09-30T02:10:44.601 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.591+0000 7f06207dd700 1 librados: calling monclient init 2023-09-30T02:10:44.601 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.591+0000 7f06207dd700 10 monclient: init 2023-09-30T02:10:44.601 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.592+0000 7f06207dd700 10 monclient: _reopen_session rank -1 2023-09-30T02:10:44.601 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.592+0000 7f06207dd700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T02:10:44.601 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.592+0000 7f06207dd700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f06180a5b70 0x7f061813ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:44.602 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.592+0000 7f06207dd700 10 monclient(hunting): picked mon.noname-c con 0x7f06180a5b70 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:10:44.602 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.592+0000 7f06207dd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06180a5f40 0x7f06181408c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:44.602 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.592+0000 7f06207dd700 10 monclient(hunting): picked mon.noname-a con 0x7f06180a5f40 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:10:44.602 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.592+0000 7f06207dd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0618142290 0x7f061813c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:44.602 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.592+0000 7f06207dd700 10 monclient(hunting): picked mon.noname-b con 0x7f0618142290 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:10:44.602 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.592+0000 7f06207dd700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:44.602 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.592+0000 7f06207dd700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f061813c570 con 0x7f06180a5f40 2023-09-30T02:10:44.602 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.592+0000 7f06207dd700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:44.602 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.592+0000 7f06207dd700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f061813c820 con 0x7f0618142290 2023-09-30T02:10:44.603 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.592+0000 7f06207dd700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:44.603 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.592+0000 7f06207dd700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06180a85c0 con 0x7f06180a5b70 2023-09-30T02:10:44.603 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.592+0000 7f06207dd700 10 monclient(hunting): _renew_subs 2023-09-30T02:10:44.603 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.592+0000 7f06207dd700 10 monclient(hunting): authenticate will time out at 2089.895171s 2023-09-30T02:10:44.603 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.592+0000 7f061e579700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f06180a5b70 0x7f061813ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:44.603 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.593+0000 7f061ed7a700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0618142290 0x7f061813c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:44.603 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.593+0000 7f061e579700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f06180a5b70 0x7f061813ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:50824/0 (socket says 172.21.15.59:50824) 2023-09-30T02:10:44.603 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.593+0000 7f061e579700 1 -- 172.21.15.59:0/3655125148 learned_addr learned my addr 172.21.15.59:0/3655125148 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:10:44.603 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.593+0000 7f061e579700 10 monclient(hunting): get_auth_request con 0x7f06180a5b70 auth_method 0 2023-09-30T02:10:44.604 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.593+0000 7f061e579700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:10:44.604 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.593+0000 7f061e579700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:44.604 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.593+0000 7f061e579700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:44.604 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.593+0000 7f061ed7a700 10 monclient(hunting): get_auth_request con 0x7f0618142290 auth_method 0 2023-09-30T02:10:44.604 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.593+0000 7f061ed7a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:10:44.604 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.593+0000 7f061ed7a700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:44.604 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.593+0000 7f061ed7a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:44.604 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.594+0000 7f061ed7a700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:10:44.604 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.594+0000 7f061ed7a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:10:44.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.594+0000 7f061ed7a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:10:44.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.594+0000 7f061e579700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:10:44.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.594+0000 7f061e579700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:10:44.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.594+0000 7f061e579700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:10:44.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.594+0000 7f061dd78700 1 --2- 172.21.15.59:0/3655125148 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06180a5f40 0x7f06181408c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:44.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.595+0000 7f061dd78700 10 monclient(hunting): get_auth_request con 0x7f06180a5f40 auth_method 0 2023-09-30T02:10:44.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.595+0000 7f061dd78700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:10:44.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.595+0000 7f061dd78700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:44.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.595+0000 7f061dd78700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:44.606 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.595+0000 7f061dd78700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:10:44.606 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.595+0000 7f061dd78700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:10:44.606 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.595+0000 7f061dd78700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:10:44.606 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.597+0000 7f061ed7a700 10 monclient(hunting): handle_auth_done global_id 4780 payload 824 2023-09-30T02:10:44.606 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.597+0000 7f061ed7a700 1 -- 172.21.15.59:0/3655125148 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f06180a5b70 msgr2=0x7f061813ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:44.606 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.597+0000 7f061ed7a700 1 --2- 172.21.15.59:0/3655125148 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f06180a5b70 0x7f061813ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:44.606 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.597+0000 7f061ed7a700 1 -- 172.21.15.59:0/3655125148 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06180a5f40 msgr2=0x7f06181408c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:44.606 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.597+0000 7f061ed7a700 1 --2- 172.21.15.59:0/3655125148 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06180a5f40 0x7f06181408c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:44.606 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.597+0000 7f061ed7a700 10 monclient: _finish_hunting 0 2023-09-30T02:10:44.606 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.597+0000 7f061ed7a700 1 monclient: found mon.noname-b 2023-09-30T02:10:44.607 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.597+0000 7f061ed7a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:10:44.607 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.597+0000 7f061ed7a700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:10:44.607 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.597+0000 7f061ed7a700 1 -- 172.21.15.59:0/3655125148 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0614009420 con 0x7f0618142290 2023-09-30T02:10:44.608 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.600+0000 7f061ed7a700 1 --2- 172.21.15.59:0/3655125148 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0618142290 0x7f061813c030 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f061000c7c0 tx=0x7f06100015f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:10:44.609 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.600+0000 7f060e7fc700 10 client.?.objecter ms_handle_connect 0x7f0618142290 2023-09-30T02:10:44.609 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.600+0000 7f060e7fc700 10 client.?.objecter resend_mon_ops 2023-09-30T02:10:44.609 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.604+0000 7f060e7fc700 1 -- 172.21.15.59:0/3655125148 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0610011590 con 0x7f0618142290 2023-09-30T02:10:44.609 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.604+0000 7f060e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:10:44.609 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.604+0000 7f060e7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:10:44.609 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.604+0000 7f060e7fc700 10 monclient: dump: 2023-09-30T02:10:44.609 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:10:44.609 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:44.610 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:44.610 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:44.610 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:10:44.610 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:44.610 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:10:44.610 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:10:44.610 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:10:44.610 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:44.610 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.604+0000 7f060e7fc700 10 monclient: _finish_auth 0 2023-09-30T02:10:44.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.604+0000 7f060e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:10:44.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.604+0000 7f060e7fc700 1 -- 172.21.15.59:0/3655125148 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0610011710 con 0x7f0618142290 2023-09-30T02:10:44.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.604+0000 7f060e7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:10:44.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.604+0000 7f060e7fc700 1 -- 172.21.15.59:0/3655125148 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0610011ad0 con 0x7f0618142290 2023-09-30T02:10:44.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.604+0000 7f06207dd700 5 monclient: authenticate success, global_id 4780 2023-09-30T02:10:44.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.604+0000 7f06207dd700 10 monclient: _renew_subs 2023-09-30T02:10:44.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.604+0000 7f06207dd700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:10:44.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.604+0000 7f06207dd700 1 -- 172.21.15.59:0/3655125148 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06181414b0 con 0x7f0618142290 2023-09-30T02:10:44.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.604+0000 7f061cd76700 4 set_mon_vals no callback set 2023-09-30T02:10:44.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.604+0000 7f060e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:10:44.612 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.604+0000 7f060e7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:10:44.612 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.604+0000 7f060e7fc700 10 monclient: dump: 2023-09-30T02:10:44.612 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:10:44.612 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:44.612 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:44.612 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:44.612 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:10:44.612 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:44.612 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:10:44.613 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:10:44.613 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:10:44.613 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:44.613 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.604+0000 7f06207dd700 10 client.4780.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:10:44.613 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.604+0000 7f06207dd700 10 monclient: _renew_subs 2023-09-30T02:10:44.613 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.604+0000 7f06207dd700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:10:44.613 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.604+0000 7f06207dd700 1 -- 172.21.15.59:0/3655125148 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f061813dbd0 con 0x7f0618142290 2023-09-30T02:10:44.613 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.604+0000 7f06207dd700 1 librados: init done 2023-09-30T02:10:44.613 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.605+0000 7f06207dd700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:10:44.613 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.605+0000 7f06207dd700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:10:44.614 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.605+0000 7f06207dd700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:10:44.614 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.605+0000 7f06207dd700 1 -- 172.21.15.59:0/3655125148 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f061813c990 con 0x7f0618142290 2023-09-30T02:10:44.854 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.851+0000 7f060e7fc700 1 -- 172.21.15.59:0/3655125148 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f0610003030 con 0x7f0618142290 2023-09-30T02:10:44.855 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.852+0000 7f060e7fc700 1 --2- 172.21.15.59:0/3655125148 >> v2:172.21.15.110:6800/115870 conn(0x7f05fc068890 0x7f05fc06ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:44.855 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.852+0000 7f060e7fc700 1 -- 172.21.15.59:0/3655125148 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f0610084460 con 0x7f0618142290 2023-09-30T02:10:44.855 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.852+0000 7f060e7fc700 10 client.4780.objecter ms_dispatch 0x7f0618001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:10:44.855 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.852+0000 7f060e7fc700 3 client.4780.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:10:44.855 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.852+0000 7f060e7fc700 3 client.4780.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:10:44.855 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.852+0000 7f060e7fc700 20 client.4780.objecter dump_active .. 0 homeless 2023-09-30T02:10:44.855 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.852+0000 7f061e579700 1 --2- 172.21.15.59:0/3655125148 >> v2:172.21.15.110:6800/115870 conn(0x7f05fc068890 0x7f05fc06ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:44.855 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.852+0000 7f061e579700 10 monclient: get_auth_request con 0x7f05fc068890 auth_method 0 2023-09-30T02:10:44.855 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.852+0000 7f061e579700 1 --2- 172.21.15.59:0/3655125148 >> v2:172.21.15.110:6800/115870 conn(0x7f05fc068890 0x7f05fc06ad40 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f061400c010 tx=0x7f0614017000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:10:44.855 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.852+0000 7f060e7fc700 10 client.4780.objecter ms_handle_connect 0x7f05fc068890 2023-09-30T02:10:44.860 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.860+0000 7f060e7fc700 1 -- 172.21.15.59:0/3655125148 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0610051d00 con 0x7f0618142290 2023-09-30T02:10:44.860 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.860+0000 7f060e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:10:44.860 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:44.860+0000 7f060e7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:10:45.109 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:45.109+0000 7f06207dd700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:10:45.109 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:45.109+0000 7f06207dd700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:10:45.110 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:45.109+0000 7f06207dd700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:10:45.110 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:45.109+0000 7f06207dd700 1 -- 172.21.15.59:0/3655125148 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f0618064a00 con 0x7f0618142290 2023-09-30T02:10:45.113 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:45.113+0000 7f060e7fc700 1 -- 172.21.15.59:0/3655125148 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f0610055320 con 0x7f0618142290 2023-09-30T02:10:45.113 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:45.113+0000 7f060e7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:10:45.113 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:45.113+0000 7f060e7fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:10:45.114 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:10:45.114 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:10:45.116 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:45.115+0000 7f06207dd700 10 librados: watch_flush enter 2023-09-30T02:10:45.116 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:45.115+0000 7f06207dd700 10 librados: watch_flush exit 2023-09-30T02:10:45.117 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:45.115+0000 7f06207dd700 20 client.4780.objecter shutdown clearing up homeless session... 2023-09-30T02:10:45.117 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:45.115+0000 7f06207dd700 10 client.4780.objecter successfully canceled tick 2023-09-30T02:10:45.117 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:45.115+0000 7f06207dd700 1 -- 172.21.15.59:0/3655125148 >> v2:172.21.15.110:6800/115870 conn(0x7f05fc068890 msgr2=0x7f05fc06ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:45.117 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:45.115+0000 7f06207dd700 1 --2- 172.21.15.59:0/3655125148 >> v2:172.21.15.110:6800/115870 conn(0x7f05fc068890 0x7f05fc06ad40 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f061400c010 tx=0x7f0614017000 comp rx=0 tx=0).stop 2023-09-30T02:10:45.117 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:45.115+0000 7f06207dd700 10 monclient: shutdown 2023-09-30T02:10:45.117 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:45.115+0000 7f06207dd700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:10:45.117 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:45.115+0000 7f06207dd700 1 -- 172.21.15.59:0/3655125148 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0618142290 msgr2=0x7f061813c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:45.117 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:45.115+0000 7f06207dd700 1 --2- 172.21.15.59:0/3655125148 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0618142290 0x7f061813c030 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f061000c7c0 tx=0x7f06100015f0 comp rx=0 tx=0).stop 2023-09-30T02:10:45.117 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:45.115+0000 7f06207dd700 1 -- 172.21.15.59:0/3655125148 shutdown_connections 2023-09-30T02:10:45.117 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:45.115+0000 7f06207dd700 1 --2- 172.21.15.59:0/3655125148 >> v2:172.21.15.110:6800/115870 conn(0x7f05fc068890 0x7f05fc06ad40 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:45.117 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:45.115+0000 7f06207dd700 1 --2- 172.21.15.59:0/3655125148 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0618142290 0x7f061813c030 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:45.118 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:45.115+0000 7f06207dd700 1 --2- 172.21.15.59:0/3655125148 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f06180a5b70 0x7f061813ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:45.118 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:45.115+0000 7f06207dd700 1 --2- 172.21.15.59:0/3655125148 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06180a5f40 0x7f06181408c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:45.118 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:45.115+0000 7f06207dd700 1 -- 172.21.15.59:0/3655125148 >> 172.21.15.59:0/3655125148 conn(0x7f06180631f0 msgr2=0x7f06180b52f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:10:45.118 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:45.115+0000 7f06207dd700 1 -- 172.21.15.59:0/3655125148 shutdown_connections 2023-09-30T02:10:45.118 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:45.116+0000 7f06207dd700 1 -- 172.21.15.59:0/3655125148 wait complete. 2023-09-30T02:10:45.118 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:45.116+0000 7f06207dd700 1 librados: shutdown 2023-09-30T02:10:48.131 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:10:48.266 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.266+0000 7fe09f0c1700 10 monclient: get_monmap_and_config 2023-09-30T02:10:48.267 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.266+0000 7fe09f0c1700 10 monclient: build_initial_monmap 2023-09-30T02:10:48.267 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.266+0000 7fe09f0c1700 10 monclient: monmap: 2023-09-30T02:10:48.267 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:10:48.267 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:48.267 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:10:48.267598+0000 2023-09-30T02:10:48.267 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:10:48.267598+0000 2023-09-30T02:10:48.267 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:10:48.267 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:48.267 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:10:48.267 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:10:48.268 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:10:48.268 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:10:48.268 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:10:48.268 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:48.270 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.267+0000 7fe09f0c1700 1 Processor -- start 2023-09-30T02:10:48.270 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.267+0000 7fe09f0c1700 1 -- start start 2023-09-30T02:10:48.270 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.267+0000 7fe09f0c1700 10 monclient: init 2023-09-30T02:10:48.270 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.267+0000 7fe09f0c1700 10 monclient: _reopen_session rank -1 2023-09-30T02:10:48.271 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.267+0000 7fe09f0c1700 10 monclient: _add_conns ranks=[3,0,2,1,4] 2023-09-30T02:10:48.271 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.267+0000 7fe09f0c1700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fe0980a5b70 0x7fe0980b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:48.271 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.267+0000 7fe09f0c1700 10 monclient(hunting): picked mon.noname-a con 0x7fe0980a5b70 addr v2:172.21.15.59:3300/0 2023-09-30T02:10:48.271 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.267+0000 7fe09f0c1700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe0980a5f40 addr v1:172.21.15.59:6789/0 2023-09-30T02:10:48.271 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.267+0000 7fe09f0c1700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe098142290 0x7fe09813af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:48.271 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.267+0000 7fe09f0c1700 10 monclient(hunting): picked mon.noname-b con 0x7fe098142290 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:10:48.271 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.267+0000 7fe09f0c1700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0980a75a0 con 0x7fe0980a5f40 2023-09-30T02:10:48.271 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.267+0000 7fe09f0c1700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:48.272 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.267+0000 7fe09f0c1700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0981426d0 con 0x7fe0980a5b70 2023-09-30T02:10:48.272 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.267+0000 7fe09f0c1700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:48.272 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.267+0000 7fe09f0c1700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0980a63e0 con 0x7fe098142290 2023-09-30T02:10:48.272 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.267+0000 7fe09f0c1700 10 monclient(hunting): _renew_subs 2023-09-30T02:10:48.272 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.267+0000 7fe09f0c1700 10 monclient(hunting): authenticate will time out at 2093.570871s 2023-09-30T02:10:48.272 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.268+0000 7fe09d65e700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe098142290 0x7fe09813af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:48.272 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.268+0000 7fe09ce5d700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fe0980a5b70 0x7fe0980b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:48.272 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.268+0000 7fe09d65e700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe098142290 0x7fe09813af30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:48632/0 (socket says 172.21.15.59:48632) 2023-09-30T02:10:48.272 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.268+0000 7fe09d65e700 1 -- 172.21.15.59:0/1273942868 learned_addr learned my addr 172.21.15.59:0/1273942868 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:10:48.272 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.268+0000 7fe09d65e700 10 monclient(hunting): get_auth_request con 0x7fe098142290 auth_method 0 2023-09-30T02:10:48.272 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.268+0000 7fe09d65e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:10:48.272 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.268+0000 7fe09d65e700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:48.273 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.268+0000 7fe09d65e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:48.273 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.269+0000 7fe09ce5d700 10 monclient(hunting): get_auth_request con 0x7fe0980a5b70 auth_method 0 2023-09-30T02:10:48.273 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.269+0000 7fe09ce5d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:10:48.273 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.269+0000 7fe09ce5d700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:48.273 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.269+0000 7fe09ce5d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:48.273 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.269+0000 7fe09d65e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:10:48.273 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.269+0000 7fe09d65e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:10:48.273 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.269+0000 7fe09d65e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:10:48.273 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.270+0000 7fe09ce5d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:10:48.273 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.270+0000 7fe09ce5d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:10:48.273 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.270+0000 7fe09ce5d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:10:48.274 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.272+0000 7fe09d65e700 10 monclient(hunting): handle_auth_done global_id 4676 payload 274 2023-09-30T02:10:48.274 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.272+0000 7fe09d65e700 1 -- 172.21.15.59:0/1273942868 >> v2:172.21.15.59:3300/0 conn(0x7fe0980a5b70 msgr2=0x7fe0980b1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:10:48.274 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.272+0000 7fe09d65e700 1 --2- 172.21.15.59:0/1273942868 >> v2:172.21.15.59:3300/0 conn(0x7fe0980a5b70 0x7fe0980b1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:48.274 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.272+0000 7fe09d65e700 1 -- 172.21.15.59:0/1273942868 >> v1:172.21.15.59:6789/0 conn(0x7fe0980a5f40 legacy=0x7fe098141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:48.275 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.272+0000 7fe09d65e700 10 monclient: _finish_hunting 0 2023-09-30T02:10:48.275 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.272+0000 7fe09d65e700 1 monclient: found mon.noname-b 2023-09-30T02:10:48.275 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.272+0000 7fe09d65e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:10:48.275 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.272+0000 7fe09d65e700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.59:3301/0 2023-09-30T02:10:48.275 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.272+0000 7fe09d65e700 1 -- 172.21.15.59:0/1273942868 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0980af3f0 con 0x7fe098142290 2023-09-30T02:10:48.275 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.274+0000 7fe09d65e700 1 --2- 172.21.15.59:0/1273942868 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe098142290 0x7fe09813af30 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fe090009f20 tx=0x7fe090009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d8c90fa78d37e77 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:10:48.279 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.279+0000 7fe0977fe700 1 -- 172.21.15.59:0/1273942868 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe09000e070 con 0x7fe098142290 2023-09-30T02:10:48.280 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.279+0000 7fe0977fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:10:48.280 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.279+0000 7fe0977fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:10:48.280 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.279+0000 7fe0977fe700 10 monclient: dump: 2023-09-30T02:10:48.280 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:10:48.280 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:48.280 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:48.280 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:48.280 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:10:48.280 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:48.280 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:10:48.281 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:10:48.281 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:10:48.281 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:48.281 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.279+0000 7fe0977fe700 10 monclient: _finish_auth 0 2023-09-30T02:10:48.281 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.279+0000 7fe0977fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:10:48.281 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.279+0000 7fe0977fe700 1 -- 172.21.15.59:0/1273942868 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe09000b9d0 con 0x7fe098142290 2023-09-30T02:10:48.281 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.279+0000 7fe09f0c1700 5 monclient: authenticate success, global_id 4676 2023-09-30T02:10:48.281 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.279+0000 7fe09f0c1700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:10:48.281 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.279+0000 7fe0977fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:10:48.281 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.279+0000 7fe09f0c1700 10 monclient: get_monmap_and_config success 2023-09-30T02:10:48.281 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.279+0000 7fe09f0c1700 4 set_mon_vals no callback set 2023-09-30T02:10:48.281 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.279+0000 7fe0977fe700 1 -- 172.21.15.59:0/1273942868 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0900092e0 con 0x7fe098142290 2023-09-30T02:10:48.282 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.279+0000 7fe0977fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:10:48.282 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.279+0000 7fe0977fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:10:48.282 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.279+0000 7fe0977fe700 10 monclient: dump: 2023-09-30T02:10:48.282 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:10:48.282 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:48.282 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:48.282 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:48.282 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:10:48.282 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:48.282 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:10:48.282 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:10:48.282 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:10:48.283 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:48.286 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.280+0000 7fe09f0c1700 10 monclient: shutdown 2023-09-30T02:10:48.286 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.280+0000 7fe09f0c1700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:10:48.286 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.280+0000 7fe09f0c1700 1 -- 172.21.15.59:0/1273942868 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe098142290 msgr2=0x7fe09813af30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:48.286 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.280+0000 7fe09f0c1700 1 --2- 172.21.15.59:0/1273942868 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe098142290 0x7fe09813af30 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fe090009f20 tx=0x7fe090009840 comp rx=0 tx=0).stop 2023-09-30T02:10:48.286 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.280+0000 7fe09f0c1700 1 -- 172.21.15.59:0/1273942868 shutdown_connections 2023-09-30T02:10:48.286 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.280+0000 7fe09f0c1700 1 --2- 172.21.15.59:0/1273942868 >> v2:172.21.15.59:3300/0 conn(0x7fe0980a5b70 0x7fe0980b1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:48.286 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.280+0000 7fe09f0c1700 1 --2- 172.21.15.59:0/1273942868 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe098142290 0x7fe09813af30 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:48.286 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.280+0000 7fe09f0c1700 1 -- 172.21.15.59:0/1273942868 >> 172.21.15.59:0/1273942868 conn(0x7fe0980631f0 msgr2=0x7fe0980635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:10:48.286 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.280+0000 7fe09f0c1700 1 -- 172.21.15.59:0/1273942868 shutdown_connections 2023-09-30T02:10:48.287 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.280+0000 7fe09f0c1700 1 -- 172.21.15.59:0/1273942868 wait complete. 2023-09-30T02:10:48.287 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.281+0000 7fe09f0c1700 10 monclient: build_initial_monmap 2023-09-30T02:10:48.287 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.281+0000 7fe09f0c1700 10 monclient: monmap: 2023-09-30T02:10:48.287 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:10:48.287 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:10:48.287 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:10:48.287 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:10:48.287 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:10:48.287 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:10:48.287 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:10:48.287 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:10:48.287 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:10:48.288 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:48.288 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.281+0000 7fe09f0c1700 1 librados: starting msgr at 2023-09-30T02:10:48.288 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.281+0000 7fe09f0c1700 1 librados: starting objecter 2023-09-30T02:10:48.288 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.281+0000 7fe09f0c1700 1 Processor -- start 2023-09-30T02:10:48.288 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.281+0000 7fe09f0c1700 1 -- start start 2023-09-30T02:10:48.288 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.281+0000 7fe09f0c1700 1 librados: setting wanted keys 2023-09-30T02:10:48.288 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.281+0000 7fe09f0c1700 1 librados: calling monclient init 2023-09-30T02:10:48.288 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.281+0000 7fe09f0c1700 10 monclient: init 2023-09-30T02:10:48.288 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.282+0000 7fe09f0c1700 10 monclient: _reopen_session rank -1 2023-09-30T02:10:48.288 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.282+0000 7fe09f0c1700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T02:10:48.288 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.282+0000 7fe09f0c1700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe0980a5b70 0x7fe0980aabd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:48.288 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.282+0000 7fe09f0c1700 10 monclient(hunting): picked mon.noname-c con 0x7fe0980a5b70 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:10:48.289 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.282+0000 7fe09f0c1700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0980a5f40 0x7fe09813e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:48.289 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.282+0000 7fe09f0c1700 10 monclient(hunting): picked mon.noname-a con 0x7fe0980a5f40 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:10:48.289 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.282+0000 7fe09f0c1700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe098142290 0x7fe09814dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:48.289 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.282+0000 7fe09f0c1700 10 monclient(hunting): picked mon.noname-b con 0x7fe098142290 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:10:48.289 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.282+0000 7fe09f0c1700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:48.289 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.282+0000 7fe09f0c1700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0980abdd0 con 0x7fe0980a5f40 2023-09-30T02:10:48.289 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.282+0000 7fe09f0c1700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:48.289 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.282+0000 7fe09f0c1700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0980ac040 con 0x7fe098142290 2023-09-30T02:10:48.289 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.282+0000 7fe09f0c1700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:48.289 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.282+0000 7fe09f0c1700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0980aa6b0 con 0x7fe0980a5b70 2023-09-30T02:10:48.289 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.282+0000 7fe09f0c1700 10 monclient(hunting): _renew_subs 2023-09-30T02:10:48.289 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.282+0000 7fe09f0c1700 10 monclient(hunting): authenticate will time out at 2093.585379s 2023-09-30T02:10:48.290 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.283+0000 7fe09ce5d700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe0980a5b70 0x7fe0980aabd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:48.290 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.283+0000 7fe09d65e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe098142290 0x7fe09814dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:48.290 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.283+0000 7fe09ce5d700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe0980a5b70 0x7fe0980aabd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:48638/0 (socket says 172.21.15.59:48638) 2023-09-30T02:10:48.290 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.283+0000 7fe09ce5d700 1 -- 172.21.15.59:0/3697954046 learned_addr learned my addr 172.21.15.59:0/3697954046 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:10:48.290 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.283+0000 7fe09ce5d700 10 monclient(hunting): get_auth_request con 0x7fe0980a5b70 auth_method 0 2023-09-30T02:10:48.290 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.283+0000 7fe09ce5d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:10:48.290 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.283+0000 7fe09ce5d700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:48.290 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.283+0000 7fe09ce5d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:48.290 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.283+0000 7fe09d65e700 10 monclient(hunting): get_auth_request con 0x7fe098142290 auth_method 0 2023-09-30T02:10:48.290 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.283+0000 7fe09d65e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:10:48.290 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.283+0000 7fe09d65e700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:48.291 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.283+0000 7fe09d65e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:48.291 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.283+0000 7fe09ce5d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:10:48.291 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.283+0000 7fe09ce5d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:10:48.291 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.283+0000 7fe09ce5d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:10:48.291 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.284+0000 7fe09d65e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:10:48.291 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.284+0000 7fe09d65e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:10:48.291 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.284+0000 7fe09d65e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:10:48.291 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.284+0000 7fe097fff700 1 --2- 172.21.15.59:0/3697954046 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0980a5f40 0x7fe09813e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:48.291 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.285+0000 7fe097fff700 10 monclient(hunting): get_auth_request con 0x7fe0980a5f40 auth_method 0 2023-09-30T02:10:48.291 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.285+0000 7fe097fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:10:48.291 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.285+0000 7fe097fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:48.292 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.285+0000 7fe097fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:48.292 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.285+0000 7fe097fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:10:48.292 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.285+0000 7fe097fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:10:48.292 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.285+0000 7fe097fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:10:48.293 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.288+0000 7fe09ce5d700 10 monclient(hunting): handle_auth_done global_id 4679 payload 824 2023-09-30T02:10:48.293 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.288+0000 7fe09ce5d700 1 -- 172.21.15.59:0/3697954046 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe098142290 msgr2=0x7fe09814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:48.293 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.288+0000 7fe09ce5d700 1 --2- 172.21.15.59:0/3697954046 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe098142290 0x7fe09814dc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:48.293 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.288+0000 7fe09ce5d700 1 -- 172.21.15.59:0/3697954046 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0980a5f40 msgr2=0x7fe09813e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:48.293 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.288+0000 7fe09ce5d700 1 --2- 172.21.15.59:0/3697954046 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0980a5f40 0x7fe09813e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:48.293 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.288+0000 7fe09ce5d700 10 monclient: _finish_hunting 0 2023-09-30T02:10:48.293 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.288+0000 7fe09ce5d700 1 monclient: found mon.noname-c 2023-09-30T02:10:48.293 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.288+0000 7fe09ce5d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:10:48.293 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.288+0000 7fe09ce5d700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.59:3301/0 2023-09-30T02:10:48.293 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.288+0000 7fe09ce5d700 1 -- 172.21.15.59:0/3697954046 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe090009ae0 con 0x7fe0980a5b70 2023-09-30T02:10:48.293 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.290+0000 7fe09ce5d700 1 --2- 172.21.15.59:0/3697954046 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe0980a5b70 0x7fe0980aabd0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fe08800ec80 tx=0x7fe0880091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:10:48.294 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.290+0000 7fe094ff9700 10 client.?.objecter ms_handle_connect 0x7fe0980a5b70 2023-09-30T02:10:48.294 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.290+0000 7fe094ff9700 10 client.?.objecter resend_mon_ops 2023-09-30T02:10:48.295 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.295+0000 7fe094ff9700 1 -- 172.21.15.59:0/3697954046 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe088009690 con 0x7fe0980a5b70 2023-09-30T02:10:48.296 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.295+0000 7fe094ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:10:48.296 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.295+0000 7fe094ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:10:48.296 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.295+0000 7fe094ff9700 10 monclient: dump: 2023-09-30T02:10:48.296 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:10:48.296 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:48.296 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:48.296 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:48.296 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:10:48.296 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:48.296 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:10:48.296 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:10:48.297 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:10:48.297 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:48.297 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.295+0000 7fe094ff9700 10 monclient: _finish_auth 0 2023-09-30T02:10:48.297 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.295+0000 7fe094ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:10:48.297 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.295+0000 7fe094ff9700 1 -- 172.21.15.59:0/3697954046 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe08800c850 con 0x7fe0980a5b70 2023-09-30T02:10:48.297 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.295+0000 7fe094ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:10:48.297 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.295+0000 7fe094ff9700 1 -- 172.21.15.59:0/3697954046 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe08800cc10 con 0x7fe0980a5b70 2023-09-30T02:10:48.297 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.295+0000 7fe094ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:10:48.297 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.295+0000 7fe094ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:10:48.298 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.295+0000 7fe094ff9700 10 monclient: dump: 2023-09-30T02:10:48.298 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:10:48.298 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:48.298 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:48.298 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:48.298 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:10:48.298 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:48.298 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:10:48.298 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:10:48.298 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:10:48.298 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:48.299 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.295+0000 7fe09f0c1700 5 monclient: authenticate success, global_id 4679 2023-09-30T02:10:48.299 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.295+0000 7fe09f0c1700 10 monclient: _renew_subs 2023-09-30T02:10:48.299 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.295+0000 7fe09f0c1700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:10:48.299 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.295+0000 7fe09f0c1700 1 -- 172.21.15.59:0/3697954046 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0980a8dc0 con 0x7fe0980a5b70 2023-09-30T02:10:48.299 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.295+0000 7fe096ffd700 4 set_mon_vals no callback set 2023-09-30T02:10:48.299 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.295+0000 7fe09f0c1700 10 client.4679.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:10:48.299 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.295+0000 7fe09f0c1700 10 monclient: _renew_subs 2023-09-30T02:10:48.299 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.295+0000 7fe09f0c1700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:10:48.299 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.295+0000 7fe09f0c1700 1 -- 172.21.15.59:0/3697954046 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0980aa130 con 0x7fe0980a5b70 2023-09-30T02:10:48.299 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.295+0000 7fe09f0c1700 1 librados: init done 2023-09-30T02:10:48.300 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.296+0000 7fe09f0c1700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:10:48.300 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.296+0000 7fe09f0c1700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:10:48.300 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.296+0000 7fe09f0c1700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:10:48.300 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.296+0000 7fe09f0c1700 1 -- 172.21.15.59:0/3697954046 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0980b1800 con 0x7fe0980a5b70 2023-09-30T02:10:48.316 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.315+0000 7fe094ff9700 1 -- 172.21.15.59:0/3697954046 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fe088013470 con 0x7fe0980a5b70 2023-09-30T02:10:48.316 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.316+0000 7fe094ff9700 1 --2- 172.21.15.59:0/3697954046 >> v2:172.21.15.110:6800/115870 conn(0x7fe078068890 0x7fe07806ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:48.316 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.316+0000 7fe097fff700 1 --2- 172.21.15.59:0/3697954046 >> v2:172.21.15.110:6800/115870 conn(0x7fe078068890 0x7fe07806ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:48.317 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.316+0000 7fe097fff700 10 monclient: get_auth_request con 0x7fe078068890 auth_method 0 2023-09-30T02:10:48.317 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.317+0000 7fe094ff9700 10 client.4679.objecter ms_handle_connect 0x7fe078068890 2023-09-30T02:10:48.317 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.317+0000 7fe097fff700 1 --2- 172.21.15.59:0/3697954046 >> v2:172.21.15.110:6800/115870 conn(0x7fe078068890 0x7fe07806ad40 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe0980ab490 tx=0x7fe080000bc0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:10:48.593 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.593+0000 7fe094ff9700 1 -- 172.21.15.59:0/3697954046 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fe088056510 con 0x7fe0980a5b70 2023-09-30T02:10:48.594 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.593+0000 7fe094ff9700 10 client.4679.objecter ms_dispatch 0x7fe098001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:10:48.594 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.593+0000 7fe094ff9700 3 client.4679.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:10:48.594 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.593+0000 7fe094ff9700 3 client.4679.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:10:48.594 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.593+0000 7fe094ff9700 20 client.4679.objecter dump_active .. 0 homeless 2023-09-30T02:10:48.594 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.593+0000 7fe094ff9700 1 -- 172.21.15.59:0/3697954046 <== mon.2 v2:172.21.15.59:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe088052150 con 0x7fe0980a5b70 2023-09-30T02:10:48.594 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.593+0000 7fe094ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:10:48.594 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.593+0000 7fe094ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:10:48.848 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.847+0000 7fe09f0c1700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:10:48.848 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.847+0000 7fe09f0c1700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:10:48.848 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.847+0000 7fe09f0c1700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:10:48.848 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.847+0000 7fe09f0c1700 1 -- 172.21.15.59:0/3697954046 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe09813b1b0 con 0x7fe0980a5b70 2023-09-30T02:10:48.853 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.853+0000 7fe094ff9700 1 -- 172.21.15.59:0/3697954046 <== mon.2 v2:172.21.15.59:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fe088017100 con 0x7fe0980a5b70 2023-09-30T02:10:48.853 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.853+0000 7fe094ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:10:48.853 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.853+0000 7fe094ff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:10:48.853 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:10:48.853 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:10:48.856 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.855+0000 7fe09f0c1700 10 librados: watch_flush enter 2023-09-30T02:10:48.856 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.855+0000 7fe09f0c1700 10 librados: watch_flush exit 2023-09-30T02:10:48.856 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.855+0000 7fe09f0c1700 20 client.4679.objecter shutdown clearing up homeless session... 2023-09-30T02:10:48.856 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.855+0000 7fe09f0c1700 10 client.4679.objecter successfully canceled tick 2023-09-30T02:10:48.856 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.855+0000 7fe09f0c1700 1 -- 172.21.15.59:0/3697954046 >> v2:172.21.15.110:6800/115870 conn(0x7fe078068890 msgr2=0x7fe07806ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:48.856 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.855+0000 7fe09f0c1700 1 --2- 172.21.15.59:0/3697954046 >> v2:172.21.15.110:6800/115870 conn(0x7fe078068890 0x7fe07806ad40 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe0980ab490 tx=0x7fe080000bc0 comp rx=0 tx=0).stop 2023-09-30T02:10:48.857 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.855+0000 7fe09f0c1700 10 monclient: shutdown 2023-09-30T02:10:48.857 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.855+0000 7fe09f0c1700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:10:48.857 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.855+0000 7fe09f0c1700 1 -- 172.21.15.59:0/3697954046 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe0980a5b70 msgr2=0x7fe0980aabd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:48.857 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.855+0000 7fe09f0c1700 1 --2- 172.21.15.59:0/3697954046 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe0980a5b70 0x7fe0980aabd0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fe08800ec80 tx=0x7fe0880091a0 comp rx=0 tx=0).stop 2023-09-30T02:10:48.857 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.855+0000 7fe09f0c1700 1 -- 172.21.15.59:0/3697954046 shutdown_connections 2023-09-30T02:10:48.857 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.855+0000 7fe09f0c1700 1 --2- 172.21.15.59:0/3697954046 >> v2:172.21.15.110:6800/115870 conn(0x7fe078068890 0x7fe07806ad40 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:48.857 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.855+0000 7fe09f0c1700 1 --2- 172.21.15.59:0/3697954046 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe098142290 0x7fe09814dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:48.857 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.855+0000 7fe09f0c1700 1 --2- 172.21.15.59:0/3697954046 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe0980a5b70 0x7fe0980aabd0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:48.858 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.855+0000 7fe09f0c1700 1 --2- 172.21.15.59:0/3697954046 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0980a5f40 0x7fe09813e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:48.858 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.855+0000 7fe09f0c1700 1 -- 172.21.15.59:0/3697954046 >> 172.21.15.59:0/3697954046 conn(0x7fe0980631f0 msgr2=0x7fe0980b0be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:10:48.858 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.855+0000 7fe09f0c1700 1 -- 172.21.15.59:0/3697954046 shutdown_connections 2023-09-30T02:10:48.858 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.855+0000 7fe09f0c1700 1 -- 172.21.15.59:0/3697954046 wait complete. 2023-09-30T02:10:48.860 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:48.855+0000 7fe09f0c1700 1 librados: shutdown 2023-09-30T02:10:51.870 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:10:52.003 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.003+0000 7f5afc7c5700 10 monclient: get_monmap_and_config 2023-09-30T02:10:52.003 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.003+0000 7f5afc7c5700 10 monclient: build_initial_monmap 2023-09-30T02:10:52.004 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.003+0000 7f5afc7c5700 10 monclient: monmap: 2023-09-30T02:10:52.004 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:10:52.004 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:52.004 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:10:52.004731+0000 2023-09-30T02:10:52.004 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:10:52.004731+0000 2023-09-30T02:10:52.004 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:10:52.004 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:52.004 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:10:52.004 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:10:52.004 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:10:52.004 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:10:52.004 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:10:52.004 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:52.006 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.004+0000 7f5afc7c5700 1 Processor -- start 2023-09-30T02:10:52.006 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.004+0000 7f5afc7c5700 1 -- start start 2023-09-30T02:10:52.006 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.004+0000 7f5afc7c5700 10 monclient: init 2023-09-30T02:10:52.006 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.005+0000 7f5afc7c5700 10 monclient: _reopen_session rank -1 2023-09-30T02:10:52.006 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.005+0000 7f5afc7c5700 10 monclient: _add_conns ranks=[0,1,2,4,3] 2023-09-30T02:10:52.006 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.005+0000 7f5afc7c5700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5af406e610 addr v1:172.21.15.59:6789/0 2023-09-30T02:10:52.006 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.005+0000 7f5afc7c5700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5af406e9e0 addr v1:172.21.15.110:6789/0 2023-09-30T02:10:52.006 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.005+0000 7f5afc7c5700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5af40ab1c0 0x7f5af40ab940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:52.006 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.005+0000 7f5afc7c5700 10 monclient(hunting): picked mon.noname-b con 0x7f5af40ab1c0 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:10:52.007 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.005+0000 7f5afc7c5700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5af413c700 con 0x7f5af406e610 2023-09-30T02:10:52.007 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.005+0000 7f5afc7c5700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5af40add30 con 0x7f5af406e9e0 2023-09-30T02:10:52.007 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.005+0000 7f5afc7c5700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:52.007 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.005+0000 7f5afc7c5700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5af40ac150 con 0x7f5af40ab1c0 2023-09-30T02:10:52.007 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.005+0000 7f5afc7c5700 10 monclient(hunting): _renew_subs 2023-09-30T02:10:52.007 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.005+0000 7f5afc7c5700 10 monclient(hunting): authenticate will time out at 2097.308182s 2023-09-30T02:10:52.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.007+0000 7f5af9d60700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f5af406e9e0 0x7f5af4144940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.59:57238/0 (socket says 172.21.15.59:57238) 2023-09-30T02:10:52.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.007+0000 7f5af9d60700 1 -- 172.21.15.59:0/1622688750 learned_addr learned my addr 172.21.15.59:0/1622688750 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:10:52.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.007+0000 7f5afad62700 1 --2- 172.21.15.59:0/1622688750 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5af40ab1c0 0x7f5af40ab940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:52.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.008+0000 7f5afad62700 10 monclient(hunting): get_auth_request con 0x7f5af40ab1c0 auth_method 0 2023-09-30T02:10:52.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.008+0000 7f5afad62700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:10:52.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.008+0000 7f5afad62700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:52.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.008+0000 7f5afad62700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:52.009 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.009+0000 7f5afad62700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:10:52.009 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.009+0000 7f5afad62700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:10:52.009 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.009+0000 7f5afad62700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:10:52.011 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.010+0000 7f5af955f700 1 -- 172.21.15.59:0/1622688750 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3824265619 0 0) 0x7f5af40add30 con 0x7f5af406e9e0 2023-09-30T02:10:52.011 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.011+0000 7f5af955f700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:52.011 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.011+0000 7f5af955f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:52.011 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.011+0000 7f5af955f700 10 monclient(hunting): my global_id is 4786 2023-09-30T02:10:52.011 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.011+0000 7f5af955f700 1 -- 172.21.15.59:0/1622688750 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ae0003570 con 0x7f5af406e9e0 2023-09-30T02:10:52.012 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.012+0000 7f5afad62700 10 monclient(hunting): handle_auth_done global_id 4682 payload 274 2023-09-30T02:10:52.012 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.012+0000 7f5afad62700 1 -- 172.21.15.59:0/1622688750 >> v1:172.21.15.110:6789/0 conn(0x7f5af406e9e0 legacy=0x7f5af4144940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:52.012 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.012+0000 7f5afad62700 1 -- 172.21.15.59:0/1622688750 >> v1:172.21.15.59:6789/0 conn(0x7f5af406e610 legacy=0x7f5af40aaab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:10:52.012 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.012+0000 7f5afad62700 10 monclient: _finish_hunting 0 2023-09-30T02:10:52.012 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.012+0000 7f5afad62700 1 monclient: found mon.noname-b 2023-09-30T02:10:52.012 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.012+0000 7f5afad62700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:10:52.013 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.012+0000 7f5afad62700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.59:3301/0 2023-09-30T02:10:52.013 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.012+0000 7f5afad62700 1 -- 172.21.15.59:0/1622688750 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5af40683a0 con 0x7f5af40ab1c0 2023-09-30T02:10:52.014 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.014+0000 7f5afad62700 1 --2- 172.21.15.59:0/1622688750 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5af40ab1c0 0x7f5af40ab940 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f5aec009e90 tx=0x7f5aec009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db6f57b5e814ee2f server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:10:52.020 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.019+0000 7f5af955f700 1 -- 172.21.15.59:0/1622688750 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aec00e070 con 0x7f5af40ab1c0 2023-09-30T02:10:52.020 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.019+0000 7f5af955f700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:10:52.020 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.019+0000 7f5af955f700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:10:52.020 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.019+0000 7f5af955f700 10 monclient: dump: 2023-09-30T02:10:52.020 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:10:52.020 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:52.020 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:52.020 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:52.020 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:10:52.021 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:52.021 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:10:52.021 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:10:52.021 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:10:52.021 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:52.021 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.019+0000 7f5af955f700 10 monclient: _finish_auth 0 2023-09-30T02:10:52.021 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.019+0000 7f5af955f700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:10:52.021 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.019+0000 7f5af955f700 1 -- 172.21.15.59:0/1622688750 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5aec00bbf0 con 0x7f5af40ab1c0 2023-09-30T02:10:52.021 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.019+0000 7f5afc7c5700 5 monclient: authenticate success, global_id 4682 2023-09-30T02:10:52.021 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.019+0000 7f5afc7c5700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:10:52.021 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.019+0000 7f5af955f700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:10:52.021 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.019+0000 7f5afc7c5700 10 monclient: get_monmap_and_config success 2023-09-30T02:10:52.021 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.019+0000 7f5afc7c5700 4 set_mon_vals no callback set 2023-09-30T02:10:52.022 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.019+0000 7f5af955f700 1 -- 172.21.15.59:0/1622688750 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aec012430 con 0x7f5af40ab1c0 2023-09-30T02:10:52.022 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.019+0000 7f5af955f700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:10:52.022 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.019+0000 7f5af955f700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:10:52.022 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.019+0000 7f5af955f700 10 monclient: dump: 2023-09-30T02:10:52.022 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:10:52.022 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:52.022 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:52.022 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:52.022 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:10:52.022 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:52.022 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:10:52.023 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:10:52.023 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:10:52.023 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:52.025 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.020+0000 7f5afc7c5700 10 monclient: shutdown 2023-09-30T02:10:52.025 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.020+0000 7f5afc7c5700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:10:52.025 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.020+0000 7f5afc7c5700 1 -- 172.21.15.59:0/1622688750 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5af40ab1c0 msgr2=0x7f5af40ab940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:52.026 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.020+0000 7f5afc7c5700 1 --2- 172.21.15.59:0/1622688750 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5af40ab1c0 0x7f5af40ab940 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f5aec009e90 tx=0x7f5aec009780 comp rx=0 tx=0).stop 2023-09-30T02:10:52.026 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.020+0000 7f5afc7c5700 1 -- 172.21.15.59:0/1622688750 shutdown_connections 2023-09-30T02:10:52.026 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.020+0000 7f5afc7c5700 1 --2- 172.21.15.59:0/1622688750 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5af40ab1c0 0x7f5af40ab940 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:52.026 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.020+0000 7f5afc7c5700 1 -- 172.21.15.59:0/1622688750 >> 172.21.15.59:0/1622688750 conn(0x7f5af406a030 msgr2=0x7f5af4068730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:10:52.026 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.020+0000 7f5afc7c5700 1 -- 172.21.15.59:0/1622688750 shutdown_connections 2023-09-30T02:10:52.026 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.020+0000 7f5afc7c5700 1 -- 172.21.15.59:0/1622688750 wait complete. 2023-09-30T02:10:52.026 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.021+0000 7f5afc7c5700 10 monclient: build_initial_monmap 2023-09-30T02:10:52.026 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.021+0000 7f5afc7c5700 10 monclient: monmap: 2023-09-30T02:10:52.026 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:10:52.026 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:10:52.026 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:10:52.027 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:10:52.027 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:10:52.027 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:10:52.027 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:10:52.027 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:10:52.027 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:10:52.027 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:52.027 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.021+0000 7f5afc7c5700 1 librados: starting msgr at 2023-09-30T02:10:52.027 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.021+0000 7f5afc7c5700 1 librados: starting objecter 2023-09-30T02:10:52.027 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.021+0000 7f5afc7c5700 1 Processor -- start 2023-09-30T02:10:52.027 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.021+0000 7f5afc7c5700 1 -- start start 2023-09-30T02:10:52.027 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.021+0000 7f5afc7c5700 1 librados: setting wanted keys 2023-09-30T02:10:52.027 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.021+0000 7f5afc7c5700 1 librados: calling monclient init 2023-09-30T02:10:52.028 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.021+0000 7f5afc7c5700 10 monclient: init 2023-09-30T02:10:52.028 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.022+0000 7f5afc7c5700 10 monclient: _reopen_session rank -1 2023-09-30T02:10:52.028 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.022+0000 7f5afc7c5700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T02:10:52.028 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.022+0000 7f5afc7c5700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5af406e610 0x7f5af413eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:52.028 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.022+0000 7f5afc7c5700 10 monclient(hunting): picked mon.noname-b con 0x7f5af406e610 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:10:52.028 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.022+0000 7f5afc7c5700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5af406e9e0 0x7f5af41422b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:52.028 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.022+0000 7f5afc7c5700 10 monclient(hunting): picked mon.noname-a con 0x7f5af406e9e0 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:10:52.028 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.022+0000 7f5afc7c5700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5af40ab1c0 0x7f5af41427f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:52.028 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.022+0000 7f5afc7c5700 10 monclient(hunting): picked mon.noname-c con 0x7f5af40ab1c0 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:10:52.028 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.022+0000 7f5afc7c5700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:52.028 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.022+0000 7f5afc7c5700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5af4141e80 con 0x7f5af406e9e0 2023-09-30T02:10:52.028 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.022+0000 7f5afc7c5700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:52.029 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.022+0000 7f5afc7c5700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5af413f410 con 0x7f5af406e610 2023-09-30T02:10:52.029 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.022+0000 7f5afc7c5700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:52.029 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.022+0000 7f5afc7c5700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5af41401c0 con 0x7f5af40ab1c0 2023-09-30T02:10:52.029 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.022+0000 7f5afc7c5700 10 monclient(hunting): _renew_subs 2023-09-30T02:10:52.029 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.022+0000 7f5afc7c5700 10 monclient(hunting): authenticate will time out at 2097.325450s 2023-09-30T02:10:52.029 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.023+0000 7f5af9d60700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5af406e9e0 0x7f5af41422b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:52.029 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.023+0000 7f5afa561700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5af406e610 0x7f5af413eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:52.029 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.023+0000 7f5afa561700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5af406e610 0x7f5af413eed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:59482/0 (socket says 172.21.15.59:59482) 2023-09-30T02:10:52.029 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.023+0000 7f5afa561700 1 -- 172.21.15.59:0/975703153 learned_addr learned my addr 172.21.15.59:0/975703153 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:10:52.029 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.023+0000 7f5afa561700 10 monclient(hunting): get_auth_request con 0x7f5af406e610 auth_method 0 2023-09-30T02:10:52.029 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.023+0000 7f5afa561700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:10:52.030 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.023+0000 7f5afa561700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:52.030 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.023+0000 7f5afa561700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:52.030 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.023+0000 7f5af9d60700 10 monclient(hunting): get_auth_request con 0x7f5af406e9e0 auth_method 0 2023-09-30T02:10:52.030 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.023+0000 7f5af9d60700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:10:52.030 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.023+0000 7f5af9d60700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:52.030 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.023+0000 7f5af9d60700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:52.030 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.024+0000 7f5afad62700 1 --2- 172.21.15.59:0/975703153 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5af40ab1c0 0x7f5af41427f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:52.030 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.024+0000 7f5afa561700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:10:52.030 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.024+0000 7f5afa561700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:10:52.030 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.024+0000 7f5afa561700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:10:52.030 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.024+0000 7f5af9d60700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:10:52.031 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.024+0000 7f5af9d60700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:10:52.031 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.024+0000 7f5af9d60700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:10:52.031 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.024+0000 7f5afad62700 10 monclient(hunting): get_auth_request con 0x7f5af40ab1c0 auth_method 0 2023-09-30T02:10:52.031 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.024+0000 7f5afad62700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:10:52.031 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.024+0000 7f5afad62700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:52.031 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.024+0000 7f5afad62700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:52.031 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.025+0000 7f5afad62700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:10:52.031 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.025+0000 7f5afad62700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:10:52.031 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.025+0000 7f5afad62700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:10:52.032 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.027+0000 7f5afa561700 10 monclient(hunting): handle_auth_done global_id 4789 payload 824 2023-09-30T02:10:52.032 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.027+0000 7f5afa561700 1 -- 172.21.15.59:0/975703153 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5af40ab1c0 msgr2=0x7f5af41427f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:52.032 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.027+0000 7f5afa561700 1 --2- 172.21.15.59:0/975703153 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5af40ab1c0 0x7f5af41427f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:52.032 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.027+0000 7f5afa561700 1 -- 172.21.15.59:0/975703153 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5af406e9e0 msgr2=0x7f5af41422b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:52.032 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.027+0000 7f5afa561700 1 --2- 172.21.15.59:0/975703153 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5af406e9e0 0x7f5af41422b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:52.032 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.027+0000 7f5afa561700 10 monclient: _finish_hunting 0 2023-09-30T02:10:52.032 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.027+0000 7f5afa561700 1 monclient: found mon.noname-b 2023-09-30T02:10:52.032 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.027+0000 7f5afa561700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:10:52.032 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.027+0000 7f5afa561700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:10:52.033 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.027+0000 7f5afa561700 1 -- 172.21.15.59:0/975703153 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aec009a20 con 0x7f5af406e610 2023-09-30T02:10:52.033 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.030+0000 7f5afa561700 1 --2- 172.21.15.59:0/975703153 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5af406e610 0x7f5af413eed0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f5af000b5e0 tx=0x7f5af0009120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:10:52.033 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.030+0000 7f5aea7fc700 10 client.?.objecter ms_handle_connect 0x7f5af406e610 2023-09-30T02:10:52.033 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.030+0000 7f5aea7fc700 10 client.?.objecter resend_mon_ops 2023-09-30T02:10:52.034 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.034+0000 7f5aea7fc700 1 -- 172.21.15.59:0/975703153 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5af000f510 con 0x7f5af406e610 2023-09-30T02:10:52.035 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.034+0000 7f5aea7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:10:52.035 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.034+0000 7f5aea7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:10:52.035 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.034+0000 7f5aea7fc700 10 monclient: dump: 2023-09-30T02:10:52.035 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:10:52.035 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:52.035 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:52.035 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:52.035 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:10:52.035 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:52.035 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:10:52.036 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:10:52.036 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:10:52.036 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:52.036 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.034+0000 7f5aea7fc700 10 monclient: _finish_auth 0 2023-09-30T02:10:52.036 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.034+0000 7f5aea7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:10:52.036 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.034+0000 7f5aea7fc700 1 -- 172.21.15.59:0/975703153 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5af000f670 con 0x7f5af406e610 2023-09-30T02:10:52.036 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.034+0000 7f5aea7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:10:52.036 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.034+0000 7f5aea7fc700 1 -- 172.21.15.59:0/975703153 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5af000fa30 con 0x7f5af406e610 2023-09-30T02:10:52.036 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.034+0000 7f5afc7c5700 5 monclient: authenticate success, global_id 4789 2023-09-30T02:10:52.036 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.034+0000 7f5afc7c5700 10 monclient: _renew_subs 2023-09-30T02:10:52.036 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.034+0000 7f5afc7c5700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:10:52.036 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.034+0000 7f5afc7c5700 1 -- 172.21.15.59:0/975703153 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5af40a5820 con 0x7f5af406e610 2023-09-30T02:10:52.037 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.034+0000 7f5af8d5e700 4 set_mon_vals no callback set 2023-09-30T02:10:52.037 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.034+0000 7f5aea7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:10:52.037 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.034+0000 7f5aea7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:10:52.037 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.034+0000 7f5aea7fc700 10 monclient: dump: 2023-09-30T02:10:52.037 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:10:52.037 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:52.037 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:52.037 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:52.037 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:10:52.037 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:52.037 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:10:52.037 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:10:52.038 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:10:52.038 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:52.038 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.034+0000 7f5afc7c5700 10 client.4789.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:10:52.038 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.034+0000 7f5afc7c5700 10 monclient: _renew_subs 2023-09-30T02:10:52.038 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.034+0000 7f5afc7c5700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:10:52.038 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.034+0000 7f5afc7c5700 1 -- 172.21.15.59:0/975703153 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5af4142d30 con 0x7f5af406e610 2023-09-30T02:10:52.038 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.034+0000 7f5afc7c5700 1 librados: init done 2023-09-30T02:10:52.041 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.035+0000 7f5afc7c5700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:10:52.042 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.035+0000 7f5afc7c5700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:10:52.042 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.035+0000 7f5afc7c5700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:10:52.042 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.035+0000 7f5afc7c5700 1 -- 172.21.15.59:0/975703153 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5af4142fb0 con 0x7f5af406e610 2023-09-30T02:10:52.288 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.287+0000 7f5aea7fc700 1 -- 172.21.15.59:0/975703153 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f5af0024430 con 0x7f5af406e610 2023-09-30T02:10:52.288 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.288+0000 7f5aea7fc700 1 --2- 172.21.15.59:0/975703153 >> v2:172.21.15.110:6800/115870 conn(0x7f5ad8068890 0x7f5ad806ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:52.288 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.288+0000 7f5aea7fc700 1 -- 172.21.15.59:0/975703153 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f5af0084470 con 0x7f5af406e610 2023-09-30T02:10:52.289 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.288+0000 7f5aea7fc700 10 client.4789.objecter ms_dispatch 0x7f5af4001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:10:52.289 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.288+0000 7f5aea7fc700 3 client.4789.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:10:52.289 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.288+0000 7f5aea7fc700 3 client.4789.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:10:52.289 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.288+0000 7f5aea7fc700 20 client.4789.objecter dump_active .. 0 homeless 2023-09-30T02:10:52.289 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.288+0000 7f5af9d60700 1 --2- 172.21.15.59:0/975703153 >> v2:172.21.15.110:6800/115870 conn(0x7f5ad8068890 0x7f5ad806ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:52.289 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.288+0000 7f5af9d60700 10 monclient: get_auth_request con 0x7f5ad8068890 auth_method 0 2023-09-30T02:10:52.289 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.288+0000 7f5aea7fc700 10 client.4789.objecter ms_handle_connect 0x7f5ad8068890 2023-09-30T02:10:52.290 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.288+0000 7f5af9d60700 1 --2- 172.21.15.59:0/975703153 >> v2:172.21.15.110:6800/115870 conn(0x7f5ad8068890 0x7f5ad806ad40 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f5af413f760 tx=0x7f5ae4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:10:52.299 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.298+0000 7f5aea7fc700 1 -- 172.21.15.59:0/975703153 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5af0051d90 con 0x7f5af406e610 2023-09-30T02:10:52.299 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.298+0000 7f5aea7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:10:52.299 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.298+0000 7f5aea7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:10:52.552 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.551+0000 7f5afc7c5700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:10:52.552 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.551+0000 7f5afc7c5700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:10:52.552 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.551+0000 7f5afc7c5700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:10:52.552 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.551+0000 7f5afc7c5700 1 -- 172.21.15.59:0/975703153 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f5af4064a00 con 0x7f5af406e610 2023-09-30T02:10:52.556 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.556+0000 7f5aea7fc700 1 -- 172.21.15.59:0/975703153 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f5af00553b0 con 0x7f5af406e610 2023-09-30T02:10:52.556 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.556+0000 7f5aea7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:10:52.556 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.556+0000 7f5aea7fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:10:52.557 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:10:52.557 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:10:52.559 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.558+0000 7f5afc7c5700 10 librados: watch_flush enter 2023-09-30T02:10:52.559 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.558+0000 7f5afc7c5700 10 librados: watch_flush exit 2023-09-30T02:10:52.559 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.558+0000 7f5afc7c5700 20 client.4789.objecter shutdown clearing up homeless session... 2023-09-30T02:10:52.559 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.558+0000 7f5afc7c5700 10 client.4789.objecter successfully canceled tick 2023-09-30T02:10:52.559 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.558+0000 7f5afc7c5700 1 -- 172.21.15.59:0/975703153 >> v2:172.21.15.110:6800/115870 conn(0x7f5ad8068890 msgr2=0x7f5ad806ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:52.560 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.558+0000 7f5afc7c5700 1 --2- 172.21.15.59:0/975703153 >> v2:172.21.15.110:6800/115870 conn(0x7f5ad8068890 0x7f5ad806ad40 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f5af413f760 tx=0x7f5ae4009040 comp rx=0 tx=0).stop 2023-09-30T02:10:52.560 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.558+0000 7f5afc7c5700 10 monclient: shutdown 2023-09-30T02:10:52.560 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.558+0000 7f5afc7c5700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:10:52.560 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.558+0000 7f5afc7c5700 1 -- 172.21.15.59:0/975703153 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5af406e610 msgr2=0x7f5af413eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:52.560 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.558+0000 7f5afc7c5700 1 --2- 172.21.15.59:0/975703153 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5af406e610 0x7f5af413eed0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f5af000b5e0 tx=0x7f5af0009120 comp rx=0 tx=0).stop 2023-09-30T02:10:52.560 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.558+0000 7f5afc7c5700 1 -- 172.21.15.59:0/975703153 shutdown_connections 2023-09-30T02:10:52.560 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.558+0000 7f5afc7c5700 1 --2- 172.21.15.59:0/975703153 >> v2:172.21.15.110:6800/115870 conn(0x7f5ad8068890 0x7f5ad806ad40 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:52.560 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.558+0000 7f5afc7c5700 1 --2- 172.21.15.59:0/975703153 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f5af406e610 0x7f5af413eed0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:52.560 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.558+0000 7f5afc7c5700 1 --2- 172.21.15.59:0/975703153 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5af40ab1c0 0x7f5af41427f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:52.560 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.558+0000 7f5afc7c5700 1 --2- 172.21.15.59:0/975703153 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5af406e9e0 0x7f5af41422b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:52.560 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.558+0000 7f5afc7c5700 1 -- 172.21.15.59:0/975703153 >> 172.21.15.59:0/975703153 conn(0x7f5af406a030 msgr2=0x7f5af40b03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:10:52.561 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.558+0000 7f5afc7c5700 1 -- 172.21.15.59:0/975703153 shutdown_connections 2023-09-30T02:10:52.561 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.558+0000 7f5afc7c5700 1 -- 172.21.15.59:0/975703153 wait complete. 2023-09-30T02:10:52.561 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:52.558+0000 7f5afc7c5700 1 librados: shutdown 2023-09-30T02:10:54.364 INFO:tasks.ceph.mon.a.smithi059.stderr:==00:00:06:04.229 116256== Warning: unimplemented fcntl command: 1036 2023-09-30T02:10:54.368 INFO:tasks.ceph.mon.a.smithi059.stderr:==00:00:06:04.233 116256== Warning: unimplemented fcntl command: 1036 2023-09-30T02:10:54.396 INFO:tasks.ceph.mon.a.smithi059.stderr:==00:00:06:04.261 116256== Warning: unimplemented fcntl command: 1036 2023-09-30T02:10:54.419 INFO:tasks.ceph.mon.b.smithi110.stderr:==00:00:06:04.267 115869== Warning: unimplemented fcntl command: 1036 2023-09-30T02:10:54.424 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:06:04.289 116257== Warning: unimplemented fcntl command: 1036 2023-09-30T02:10:54.424 INFO:tasks.ceph.mon.b.smithi110.stderr:==00:00:06:04.271 115869== Warning: unimplemented fcntl command: 1036 2023-09-30T02:10:54.429 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:06:04.294 116257== Warning: unimplemented fcntl command: 1036 2023-09-30T02:10:54.446 INFO:tasks.ceph.mon.b.smithi110.stderr:==00:00:06:04.294 115869== Warning: unimplemented fcntl command: 1036 2023-09-30T02:10:54.454 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:06:04.320 116257== Warning: unimplemented fcntl command: 1036 2023-09-30T02:10:55.573 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:10:55.712 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.712+0000 7fcc428f9700 10 monclient: get_monmap_and_config 2023-09-30T02:10:55.712 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.712+0000 7fcc428f9700 10 monclient: build_initial_monmap 2023-09-30T02:10:55.713 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.712+0000 7fcc428f9700 10 monclient: monmap: 2023-09-30T02:10:55.713 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:10:55.713 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:55.713 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:10:55.713439+0000 2023-09-30T02:10:55.713 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:10:55.713439+0000 2023-09-30T02:10:55.713 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:10:55.713 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:55.713 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:10:55.713 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:10:55.713 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:10:55.713 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:10:55.714 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:10:55.714 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:55.716 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.712+0000 7fcc428f9700 1 Processor -- start 2023-09-30T02:10:55.716 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.712+0000 7fcc428f9700 1 -- start start 2023-09-30T02:10:55.716 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.712+0000 7fcc428f9700 10 monclient: init 2023-09-30T02:10:55.716 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.713+0000 7fcc428f9700 10 monclient: _reopen_session rank -1 2023-09-30T02:10:55.716 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.713+0000 7fcc428f9700 10 monclient: _add_conns ranks=[0,2,4,3,1] 2023-09-30T02:10:55.716 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.713+0000 7fcc428f9700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcc3c0a5b70 addr v1:172.21.15.59:6789/0 2023-09-30T02:10:55.716 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.713+0000 7fcc428f9700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fcc3c142290 0x7fcc3c0b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:55.716 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.713+0000 7fcc428f9700 10 monclient(hunting): picked mon.noname-b con 0x7fcc3c142290 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:10:55.716 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.713+0000 7fcc428f9700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fcc3c0a5f40 0x7fcc3c13af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:55.716 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.713+0000 7fcc428f9700 10 monclient(hunting): picked mon.noname-c con 0x7fcc3c0a5f40 addr v2:172.21.15.110:3300/0 2023-09-30T02:10:55.717 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.713+0000 7fcc428f9700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc3c0a6380 con 0x7fcc3c0a5b70 2023-09-30T02:10:55.717 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.713+0000 7fcc428f9700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:55.717 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.713+0000 7fcc428f9700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc3c0a75a0 con 0x7fcc3c0a5f40 2023-09-30T02:10:55.717 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.713+0000 7fcc428f9700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:55.717 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.713+0000 7fcc428f9700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3c0af3f0 con 0x7fcc3c142290 2023-09-30T02:10:55.717 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.713+0000 7fcc428f9700 10 monclient(hunting): _renew_subs 2023-09-30T02:10:55.717 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.713+0000 7fcc428f9700 10 monclient(hunting): authenticate will time out at 2101.016573s 2023-09-30T02:10:55.717 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.714+0000 7fcc3b7fe700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fcc3c142290 0x7fcc3c0b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:55.717 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.714+0000 7fcc3bfff700 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fcc3c0a5b70 0x7fcc3c141b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:52346/0 (socket says 172.21.15.59:52346) 2023-09-30T02:10:55.717 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.714+0000 7fcc3bfff700 1 -- 172.21.15.59:0/1157455642 learned_addr learned my addr 172.21.15.59:0/1157455642 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:10:55.717 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.714+0000 7fcc40e96700 1 --2- 172.21.15.59:0/1157455642 >> v2:172.21.15.110:3300/0 conn(0x7fcc3c0a5f40 0x7fcc3c13af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:55.718 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.714+0000 7fcc3b7fe700 10 monclient(hunting): get_auth_request con 0x7fcc3c142290 auth_method 0 2023-09-30T02:10:55.718 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.714+0000 7fcc3b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:10:55.718 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.714+0000 7fcc3b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:55.718 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.714+0000 7fcc3b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:55.718 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.715+0000 7fcc40e96700 10 monclient(hunting): get_auth_request con 0x7fcc3c0a5f40 auth_method 0 2023-09-30T02:10:55.718 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.715+0000 7fcc40e96700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:10:55.718 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.715+0000 7fcc40e96700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:55.718 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.715+0000 7fcc40e96700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:55.718 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.715+0000 7fcc3b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:10:55.718 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.715+0000 7fcc3b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:10:55.719 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.715+0000 7fcc3b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:10:55.719 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.715+0000 7fcc40e96700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:10:55.719 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.715+0000 7fcc40e96700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:10:55.719 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.715+0000 7fcc40e96700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:10:55.720 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.717+0000 7fcc3affd700 1 -- 172.21.15.59:0/1157455642 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3014087619 0 0) 0x7fcc3c0a6380 con 0x7fcc3c0a5b70 2023-09-30T02:10:55.720 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.717+0000 7fcc3affd700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:55.720 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.717+0000 7fcc3affd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:55.720 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.717+0000 7fcc3affd700 10 monclient(hunting): my global_id is 4764 2023-09-30T02:10:55.720 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.717+0000 7fcc3affd700 1 -- 172.21.15.59:0/1157455642 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc24003570 con 0x7fcc3c0a5b70 2023-09-30T02:10:55.720 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.718+0000 7fcc3b7fe700 10 monclient(hunting): handle_auth_done global_id 4688 payload 274 2023-09-30T02:10:55.720 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.718+0000 7fcc3b7fe700 1 -- 172.21.15.59:0/1157455642 >> v2:172.21.15.110:3300/0 conn(0x7fcc3c0a5f40 msgr2=0x7fcc3c13af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:10:55.720 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.718+0000 7fcc3b7fe700 1 --2- 172.21.15.59:0/1157455642 >> v2:172.21.15.110:3300/0 conn(0x7fcc3c0a5f40 0x7fcc3c13af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:55.720 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.718+0000 7fcc3b7fe700 1 -- 172.21.15.59:0/1157455642 >> v1:172.21.15.59:6789/0 conn(0x7fcc3c0a5b70 legacy=0x7fcc3c141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:55.721 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.718+0000 7fcc3b7fe700 10 monclient: _finish_hunting 0 2023-09-30T02:10:55.721 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.718+0000 7fcc3b7fe700 1 monclient: found mon.noname-b 2023-09-30T02:10:55.721 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.718+0000 7fcc3b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:10:55.721 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.718+0000 7fcc3b7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.59:3301/0 2023-09-30T02:10:55.721 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.718+0000 7fcc3b7fe700 1 -- 172.21.15.59:0/1157455642 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc3c13b470 con 0x7fcc3c142290 2023-09-30T02:10:55.721 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.721+0000 7fcc3b7fe700 1 --2- 172.21.15.59:0/1157455642 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fcc3c142290 0x7fcc3c0b1500 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fcc300098e0 tx=0x7fcc300091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=104a5c41c2b4c371 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:10:55.727 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.725+0000 7fcc3affd700 1 -- 172.21.15.59:0/1157455642 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc30011630 con 0x7fcc3c142290 2023-09-30T02:10:55.727 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.725+0000 7fcc3affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:10:55.727 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.726+0000 7fcc3affd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:10:55.727 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.726+0000 7fcc3affd700 10 monclient: dump: 2023-09-30T02:10:55.727 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:10:55.727 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:55.727 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:55.728 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:55.728 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:10:55.728 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:55.728 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:10:55.728 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:10:55.728 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:10:55.728 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:55.728 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.726+0000 7fcc3affd700 10 monclient: _finish_auth 0 2023-09-30T02:10:55.728 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.726+0000 7fcc3affd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:10:55.728 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.726+0000 7fcc3affd700 1 -- 172.21.15.59:0/1157455642 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc30011790 con 0x7fcc3c142290 2023-09-30T02:10:55.728 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.726+0000 7fcc428f9700 5 monclient: authenticate success, global_id 4688 2023-09-30T02:10:55.729 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.726+0000 7fcc428f9700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:10:55.729 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.726+0000 7fcc3affd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:10:55.729 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.726+0000 7fcc428f9700 10 monclient: get_monmap_and_config success 2023-09-30T02:10:55.729 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.726+0000 7fcc3affd700 1 -- 172.21.15.59:0/1157455642 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc30011b30 con 0x7fcc3c142290 2023-09-30T02:10:55.729 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.726+0000 7fcc428f9700 4 set_mon_vals no callback set 2023-09-30T02:10:55.729 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.726+0000 7fcc3affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:10:55.729 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.726+0000 7fcc3affd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:10:55.729 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.726+0000 7fcc3affd700 10 monclient: dump: 2023-09-30T02:10:55.729 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:10:55.729 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:55.729 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:55.729 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:55.730 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:10:55.730 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:55.730 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:10:55.730 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:10:55.730 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:10:55.730 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:55.730 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.726+0000 7fcc428f9700 10 monclient: shutdown 2023-09-30T02:10:55.730 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.726+0000 7fcc428f9700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:10:55.730 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.726+0000 7fcc428f9700 1 -- 172.21.15.59:0/1157455642 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fcc3c142290 msgr2=0x7fcc3c0b1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:55.730 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.726+0000 7fcc428f9700 1 --2- 172.21.15.59:0/1157455642 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fcc3c142290 0x7fcc3c0b1500 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fcc300098e0 tx=0x7fcc300091a0 comp rx=0 tx=0).stop 2023-09-30T02:10:55.730 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.726+0000 7fcc428f9700 1 -- 172.21.15.59:0/1157455642 shutdown_connections 2023-09-30T02:10:55.731 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.726+0000 7fcc428f9700 1 --2- 172.21.15.59:0/1157455642 >> v2:172.21.15.110:3300/0 conn(0x7fcc3c0a5f40 0x7fcc3c13af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:55.731 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.726+0000 7fcc428f9700 1 --2- 172.21.15.59:0/1157455642 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fcc3c142290 0x7fcc3c0b1500 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:55.731 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.726+0000 7fcc428f9700 1 -- 172.21.15.59:0/1157455642 >> 172.21.15.59:0/1157455642 conn(0x7fcc3c0631f0 msgr2=0x7fcc3c0635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:10:55.731 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.726+0000 7fcc428f9700 1 -- 172.21.15.59:0/1157455642 shutdown_connections 2023-09-30T02:10:55.731 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.726+0000 7fcc428f9700 1 -- 172.21.15.59:0/1157455642 wait complete. 2023-09-30T02:10:55.733 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.727+0000 7fcc428f9700 10 monclient: build_initial_monmap 2023-09-30T02:10:55.733 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.727+0000 7fcc428f9700 10 monclient: monmap: 2023-09-30T02:10:55.734 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:10:55.734 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:10:55.734 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:10:55.734 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:10:55.734 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:10:55.734 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:10:55.734 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:10:55.734 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:10:55.734 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:10:55.734 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:55.735 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.727+0000 7fcc428f9700 1 librados: starting msgr at 2023-09-30T02:10:55.735 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.727+0000 7fcc428f9700 1 librados: starting objecter 2023-09-30T02:10:55.735 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.728+0000 7fcc428f9700 1 Processor -- start 2023-09-30T02:10:55.735 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.728+0000 7fcc428f9700 1 -- start start 2023-09-30T02:10:55.735 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.728+0000 7fcc428f9700 1 librados: setting wanted keys 2023-09-30T02:10:55.735 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.728+0000 7fcc428f9700 1 librados: calling monclient init 2023-09-30T02:10:55.735 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.728+0000 7fcc428f9700 10 monclient: init 2023-09-30T02:10:55.735 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.728+0000 7fcc428f9700 10 monclient: _reopen_session rank -1 2023-09-30T02:10:55.735 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.728+0000 7fcc428f9700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T02:10:55.735 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.728+0000 7fcc428f9700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcc3c0a5b70 0x7fcc3c13eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:55.735 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.728+0000 7fcc428f9700 10 monclient(hunting): picked mon.noname-b con 0x7fcc3c0a5b70 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:10:55.736 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.728+0000 7fcc428f9700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fcc3c0a5f40 0x7fcc3c0a7d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:55.736 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.728+0000 7fcc428f9700 10 monclient(hunting): picked mon.noname-c con 0x7fcc3c0a5f40 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:10:55.736 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.728+0000 7fcc428f9700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc3c142290 0x7fcc3c13c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:55.736 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.728+0000 7fcc428f9700 10 monclient(hunting): picked mon.noname-a con 0x7fcc3c142290 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:10:55.736 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.728+0000 7fcc428f9700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:55.736 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.728+0000 7fcc428f9700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3c13c1f0 con 0x7fcc3c142290 2023-09-30T02:10:55.736 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.728+0000 7fcc428f9700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:55.736 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.728+0000 7fcc428f9700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3c13dcc0 con 0x7fcc3c0a5b70 2023-09-30T02:10:55.736 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.728+0000 7fcc428f9700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:55.736 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.728+0000 7fcc428f9700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3c13de30 con 0x7fcc3c0a5f40 2023-09-30T02:10:55.736 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.728+0000 7fcc428f9700 10 monclient(hunting): _renew_subs 2023-09-30T02:10:55.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.728+0000 7fcc428f9700 10 monclient(hunting): authenticate will time out at 2101.031996s 2023-09-30T02:10:55.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.729+0000 7fcc3b7fe700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fcc3c0a5f40 0x7fcc3c0a7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:55.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.729+0000 7fcc40e96700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc3c142290 0x7fcc3c13c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:55.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.729+0000 7fcc3bfff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcc3c0a5b70 0x7fcc3c13eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:55.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.729+0000 7fcc3b7fe700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fcc3c0a5f40 0x7fcc3c0a7d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:48674/0 (socket says 172.21.15.59:48674) 2023-09-30T02:10:55.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.729+0000 7fcc3b7fe700 1 -- 172.21.15.59:0/1693631007 learned_addr learned my addr 172.21.15.59:0/1693631007 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:10:55.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.729+0000 7fcc3b7fe700 10 monclient(hunting): get_auth_request con 0x7fcc3c0a5f40 auth_method 0 2023-09-30T02:10:55.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.729+0000 7fcc3b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:10:55.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.729+0000 7fcc3b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:55.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.729+0000 7fcc3b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:55.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.730+0000 7fcc3bfff700 10 monclient(hunting): get_auth_request con 0x7fcc3c0a5b70 auth_method 0 2023-09-30T02:10:55.738 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.730+0000 7fcc3bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:10:55.738 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.730+0000 7fcc3bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:55.738 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.730+0000 7fcc3bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:55.738 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.730+0000 7fcc40e96700 10 monclient(hunting): get_auth_request con 0x7fcc3c142290 auth_method 0 2023-09-30T02:10:55.738 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.730+0000 7fcc40e96700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:10:55.738 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.730+0000 7fcc40e96700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:55.738 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.730+0000 7fcc40e96700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:55.738 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.730+0000 7fcc3b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:10:55.738 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.730+0000 7fcc3b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:10:55.738 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.730+0000 7fcc3b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:10:55.738 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.730+0000 7fcc3bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:10:55.739 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.730+0000 7fcc3bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:10:55.739 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.730+0000 7fcc3bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:10:55.739 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.731+0000 7fcc40e96700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:10:55.739 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.731+0000 7fcc40e96700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:10:55.739 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.731+0000 7fcc40e96700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:10:55.740 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.734+0000 7fcc3b7fe700 10 monclient(hunting): handle_auth_done global_id 4691 payload 824 2023-09-30T02:10:55.740 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.734+0000 7fcc3b7fe700 1 -- 172.21.15.59:0/1693631007 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcc3c0a5b70 msgr2=0x7fcc3c13eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:55.740 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.734+0000 7fcc3b7fe700 1 --2- 172.21.15.59:0/1693631007 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcc3c0a5b70 0x7fcc3c13eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:55.740 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.734+0000 7fcc3b7fe700 1 -- 172.21.15.59:0/1693631007 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc3c142290 msgr2=0x7fcc3c13c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:55.740 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.734+0000 7fcc3b7fe700 1 --2- 172.21.15.59:0/1693631007 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc3c142290 0x7fcc3c13c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:55.740 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.734+0000 7fcc3b7fe700 10 monclient: _finish_hunting 0 2023-09-30T02:10:55.740 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.734+0000 7fcc3b7fe700 1 monclient: found mon.noname-c 2023-09-30T02:10:55.740 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.734+0000 7fcc3b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:10:55.740 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.734+0000 7fcc3b7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.59:3301/0 2023-09-30T02:10:55.740 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.734+0000 7fcc3b7fe700 1 -- 172.21.15.59:0/1693631007 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc30009420 con 0x7fcc3c0a5f40 2023-09-30T02:10:55.741 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.736+0000 7fcc3b7fe700 1 --2- 172.21.15.59:0/1693631007 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fcc3c0a5f40 0x7fcc3c0a7d70 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fcc30009ed0 tx=0x7fcc30017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:10:55.741 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.736+0000 7fcc23fff700 10 client.?.objecter ms_handle_connect 0x7fcc3c0a5f40 2023-09-30T02:10:55.741 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.736+0000 7fcc23fff700 10 client.?.objecter resend_mon_ops 2023-09-30T02:10:55.742 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.740+0000 7fcc23fff700 1 -- 172.21.15.59:0/1693631007 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc30011900 con 0x7fcc3c0a5f40 2023-09-30T02:10:55.742 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.740+0000 7fcc23fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:10:55.742 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.740+0000 7fcc23fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:10:55.742 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.740+0000 7fcc23fff700 10 monclient: dump: 2023-09-30T02:10:55.742 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:10:55.742 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:55.742 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:55.743 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:55.743 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:10:55.743 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:55.743 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:10:55.743 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:10:55.743 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:10:55.743 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:55.743 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.740+0000 7fcc23fff700 10 monclient: _finish_auth 0 2023-09-30T02:10:55.743 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.740+0000 7fcc23fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:10:55.743 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.740+0000 7fcc23fff700 1 -- 172.21.15.59:0/1693631007 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc30011a80 con 0x7fcc3c0a5f40 2023-09-30T02:10:55.743 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.740+0000 7fcc23fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:10:55.744 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.741+0000 7fcc23fff700 1 -- 172.21.15.59:0/1693631007 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc30025640 con 0x7fcc3c0a5f40 2023-09-30T02:10:55.744 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.741+0000 7fcc428f9700 5 monclient: authenticate success, global_id 4691 2023-09-30T02:10:55.744 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.741+0000 7fcc428f9700 10 monclient: _renew_subs 2023-09-30T02:10:55.744 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.741+0000 7fcc428f9700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:10:55.744 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.741+0000 7fcc428f9700 1 -- 172.21.15.59:0/1693631007 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc3c0a82b0 con 0x7fcc3c0a5f40 2023-09-30T02:10:55.744 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.741+0000 7fcc23fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:10:55.744 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.741+0000 7fcc23fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:10:55.744 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.741+0000 7fcc23fff700 10 monclient: dump: 2023-09-30T02:10:55.744 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:10:55.744 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:55.744 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:55.745 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:55.745 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:10:55.745 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:55.745 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:10:55.745 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:10:55.745 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:10:55.745 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:55.745 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.741+0000 7fcc3a7fc700 4 set_mon_vals no callback set 2023-09-30T02:10:55.745 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.741+0000 7fcc428f9700 10 client.4691.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:10:55.745 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.741+0000 7fcc428f9700 10 monclient: _renew_subs 2023-09-30T02:10:55.745 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.741+0000 7fcc428f9700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:10:55.745 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.741+0000 7fcc428f9700 1 -- 172.21.15.59:0/1693631007 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc3c13cd80 con 0x7fcc3c0a5f40 2023-09-30T02:10:55.746 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.741+0000 7fcc428f9700 1 librados: init done 2023-09-30T02:10:55.746 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.742+0000 7fcc428f9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:10:55.746 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.742+0000 7fcc428f9700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:10:55.746 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.742+0000 7fcc428f9700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:10:55.746 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.742+0000 7fcc428f9700 1 -- 172.21.15.59:0/1693631007 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc3c13c3a0 con 0x7fcc3c0a5f40 2023-09-30T02:10:55.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.968+0000 7fcc23fff700 1 -- 172.21.15.59:0/1693631007 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fcc300259f0 con 0x7fcc3c0a5f40 2023-09-30T02:10:55.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.969+0000 7fcc23fff700 1 --2- 172.21.15.59:0/1693631007 >> v2:172.21.15.110:6800/115870 conn(0x7fcc280688c0 0x7fcc2806ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:55.970 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.969+0000 7fcc3bfff700 1 --2- 172.21.15.59:0/1693631007 >> v2:172.21.15.110:6800/115870 conn(0x7fcc280688c0 0x7fcc2806ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:55.970 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.969+0000 7fcc3bfff700 10 monclient: get_auth_request con 0x7fcc280688c0 auth_method 0 2023-09-30T02:10:55.970 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.969+0000 7fcc23fff700 10 client.4691.objecter ms_handle_connect 0x7fcc280688c0 2023-09-30T02:10:55.970 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.969+0000 7fcc3bfff700 1 --2- 172.21.15.59:0/1693631007 >> v2:172.21.15.110:6800/115870 conn(0x7fcc280688c0 0x7fcc2806ad70 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fcc2c007fd0 tx=0x7fcc2c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:10:55.999 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.998+0000 7fcc23fff700 1 -- 172.21.15.59:0/1693631007 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fcc300577e0 con 0x7fcc3c0a5f40 2023-09-30T02:10:55.999 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.998+0000 7fcc23fff700 10 client.4691.objecter ms_dispatch 0x7fcc3c001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:10:55.999 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.998+0000 7fcc23fff700 3 client.4691.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:10:55.999 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.998+0000 7fcc23fff700 3 client.4691.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:10:55.999 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:55.998+0000 7fcc23fff700 20 client.4691.objecter dump_active .. 0 homeless 2023-09-30T02:10:56.053 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:56.052+0000 7fcc23fff700 1 -- 172.21.15.59:0/1693631007 <== mon.2 v2:172.21.15.59:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcc30053420 con 0x7fcc3c0a5f40 2023-09-30T02:10:56.053 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:56.052+0000 7fcc23fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:10:56.053 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:56.052+0000 7fcc23fff700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:10:56.337 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:56.335+0000 7fcc428f9700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:10:56.337 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:56.335+0000 7fcc428f9700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:10:56.337 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:56.335+0000 7fcc428f9700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:10:56.338 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:56.335+0000 7fcc428f9700 1 -- 172.21.15.59:0/1693631007 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fcc3c13c3a0 con 0x7fcc3c0a5f40 2023-09-30T02:10:56.342 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:56.341+0000 7fcc23fff700 1 -- 172.21.15.59:0/1693631007 <== mon.2 v2:172.21.15.59:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fcc30033100 con 0x7fcc3c0a5f40 2023-09-30T02:10:56.342 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:56.341+0000 7fcc23fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:10:56.342 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:56.341+0000 7fcc23fff700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:10:56.342 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:10:56.342 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:10:56.345 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:56.344+0000 7fcc428f9700 10 librados: watch_flush enter 2023-09-30T02:10:56.345 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:56.344+0000 7fcc428f9700 10 librados: watch_flush exit 2023-09-30T02:10:56.345 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:56.344+0000 7fcc428f9700 20 client.4691.objecter shutdown clearing up homeless session... 2023-09-30T02:10:56.345 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:56.344+0000 7fcc428f9700 10 client.4691.objecter successfully canceled tick 2023-09-30T02:10:56.345 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:56.344+0000 7fcc428f9700 1 -- 172.21.15.59:0/1693631007 >> v2:172.21.15.110:6800/115870 conn(0x7fcc280688c0 msgr2=0x7fcc2806ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:56.345 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:56.344+0000 7fcc428f9700 1 --2- 172.21.15.59:0/1693631007 >> v2:172.21.15.110:6800/115870 conn(0x7fcc280688c0 0x7fcc2806ad70 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fcc2c007fd0 tx=0x7fcc2c009040 comp rx=0 tx=0).stop 2023-09-30T02:10:56.345 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:56.344+0000 7fcc428f9700 10 monclient: shutdown 2023-09-30T02:10:56.345 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:56.344+0000 7fcc428f9700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:10:56.345 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:56.344+0000 7fcc428f9700 1 -- 172.21.15.59:0/1693631007 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fcc3c0a5f40 msgr2=0x7fcc3c0a7d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:56.345 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:56.344+0000 7fcc428f9700 1 --2- 172.21.15.59:0/1693631007 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fcc3c0a5f40 0x7fcc3c0a7d70 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fcc30009ed0 tx=0x7fcc30017000 comp rx=0 tx=0).stop 2023-09-30T02:10:56.346 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:56.344+0000 7fcc428f9700 1 -- 172.21.15.59:0/1693631007 shutdown_connections 2023-09-30T02:10:56.346 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:56.344+0000 7fcc428f9700 1 --2- 172.21.15.59:0/1693631007 >> v2:172.21.15.110:6800/115870 conn(0x7fcc280688c0 0x7fcc2806ad70 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:56.346 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:56.344+0000 7fcc428f9700 1 --2- 172.21.15.59:0/1693631007 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fcc3c0a5b70 0x7fcc3c13eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:56.346 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:56.344+0000 7fcc428f9700 1 --2- 172.21.15.59:0/1693631007 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcc3c142290 0x7fcc3c13c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:56.346 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:56.344+0000 7fcc428f9700 1 --2- 172.21.15.59:0/1693631007 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fcc3c0a5f40 0x7fcc3c0a7d70 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:56.346 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:56.344+0000 7fcc428f9700 1 -- 172.21.15.59:0/1693631007 >> 172.21.15.59:0/1693631007 conn(0x7fcc3c0631f0 msgr2=0x7fcc3c141b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:10:56.346 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:56.344+0000 7fcc428f9700 1 -- 172.21.15.59:0/1693631007 shutdown_connections 2023-09-30T02:10:56.346 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:56.344+0000 7fcc428f9700 1 -- 172.21.15.59:0/1693631007 wait complete. 2023-09-30T02:10:56.346 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:56.344+0000 7fcc428f9700 1 librados: shutdown 2023-09-30T02:10:59.359 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:10:59.489 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.488+0000 7f6715014700 10 monclient: get_monmap_and_config 2023-09-30T02:10:59.489 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.488+0000 7f6715014700 10 monclient: build_initial_monmap 2023-09-30T02:10:59.489 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.488+0000 7f6715014700 10 monclient: monmap: 2023-09-30T02:10:59.489 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:10:59.489 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:59.489 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:10:59.490202+0000 2023-09-30T02:10:59.490 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:10:59.490202+0000 2023-09-30T02:10:59.490 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:10:59.490 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:59.490 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:10:59.490 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:10:59.490 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:10:59.490 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:10:59.490 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:10:59.490 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:59.493 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.489+0000 7f6715014700 1 Processor -- start 2023-09-30T02:10:59.493 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.489+0000 7f6715014700 1 -- start start 2023-09-30T02:10:59.493 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.489+0000 7f6715014700 10 monclient: init 2023-09-30T02:10:59.493 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.490+0000 7f6715014700 10 monclient: _reopen_session rank -1 2023-09-30T02:10:59.493 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.490+0000 7f6715014700 10 monclient: _add_conns ranks=[2,3,4,0,1] 2023-09-30T02:10:59.493 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.490+0000 7f6715014700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f67100a5b70 0x7f67100b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:59.493 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.490+0000 7f6715014700 10 monclient(hunting): picked mon.noname-b con 0x7f67100a5b70 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:10:59.493 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.490+0000 7f6715014700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f67100a5f40 0x7f6710141b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:59.493 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.490+0000 7f6715014700 10 monclient(hunting): picked mon.noname-a con 0x7f67100a5f40 addr v2:172.21.15.59:3300/0 2023-09-30T02:10:59.493 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.490+0000 7f6715014700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f67101420c0 0x7f671013e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:59.494 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.490+0000 7f6715014700 10 monclient(hunting): picked mon.noname-c con 0x7f67101420c0 addr v2:172.21.15.110:3300/0 2023-09-30T02:10:59.494 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.490+0000 7f6715014700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:59.494 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.490+0000 7f6715014700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67100a63b0 con 0x7f67100a5f40 2023-09-30T02:10:59.494 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.490+0000 7f6715014700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:59.494 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.490+0000 7f6715014700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67100a75a0 con 0x7f67101420c0 2023-09-30T02:10:59.494 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.490+0000 7f6715014700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:59.494 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.490+0000 7f6715014700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67100af3f0 con 0x7f67100a5b70 2023-09-30T02:10:59.494 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.490+0000 7f6715014700 10 monclient(hunting): _renew_subs 2023-09-30T02:10:59.494 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.490+0000 7f6715014700 10 monclient(hunting): authenticate will time out at 2104.793264s 2023-09-30T02:10:59.494 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.490+0000 7f670ed9d700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f67100a5b70 0x7f67100b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:59.494 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.490+0000 7f670e59c700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f67100a5f40 0x7f6710141b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:59.495 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.491+0000 7f670f59e700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f67101420c0 0x7f671013e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:59.495 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.491+0000 7f670ed9d700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f67100a5b70 0x7f67100b1500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:52890/0 (socket says 172.21.15.59:52890) 2023-09-30T02:10:59.495 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.491+0000 7f670ed9d700 1 -- 172.21.15.59:0/410171670 learned_addr learned my addr 172.21.15.59:0/410171670 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:10:59.495 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.491+0000 7f670f59e700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f67101420c0 0x7f671013e230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:46008/0 (socket says 172.21.15.59:46008) 2023-09-30T02:10:59.495 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.491+0000 7f670ed9d700 10 monclient(hunting): get_auth_request con 0x7f67100a5b70 auth_method 0 2023-09-30T02:10:59.495 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.491+0000 7f670ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:10:59.495 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.491+0000 7f670ed9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:59.495 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.491+0000 7f670ed9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:59.495 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.491+0000 7f670f59e700 10 monclient(hunting): get_auth_request con 0x7f67101420c0 auth_method 0 2023-09-30T02:10:59.495 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.491+0000 7f670f59e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:10:59.495 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.491+0000 7f670f59e700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:59.495 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.491+0000 7f670f59e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:59.496 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.491+0000 7f670e59c700 10 monclient(hunting): get_auth_request con 0x7f67100a5f40 auth_method 0 2023-09-30T02:10:59.496 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.491+0000 7f670e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:10:59.496 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.491+0000 7f670e59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:59.496 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.491+0000 7f670e59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:59.496 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.492+0000 7f670f59e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:10:59.496 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.492+0000 7f670f59e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:10:59.496 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.492+0000 7f670f59e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:10:59.496 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.492+0000 7f670ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:10:59.496 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.492+0000 7f670ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:10:59.496 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.492+0000 7f670ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:10:59.496 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.492+0000 7f670e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:10:59.496 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.492+0000 7f670e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:10:59.497 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.492+0000 7f670e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:10:59.497 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.494+0000 7f670f59e700 10 monclient(hunting): handle_auth_done global_id 4798 payload 274 2023-09-30T02:10:59.497 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.494+0000 7f670f59e700 1 -- 172.21.15.59:0/410171670 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f67100a5b70 msgr2=0x7f67100b1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:10:59.498 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.494+0000 7f670f59e700 1 --2- 172.21.15.59:0/410171670 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f67100a5b70 0x7f67100b1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:59.498 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.494+0000 7f670f59e700 1 -- 172.21.15.59:0/410171670 >> v2:172.21.15.59:3300/0 conn(0x7f67100a5f40 msgr2=0x7f6710141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:10:59.498 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.494+0000 7f670f59e700 1 --2- 172.21.15.59:0/410171670 >> v2:172.21.15.59:3300/0 conn(0x7f67100a5f40 0x7f6710141b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:59.498 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.494+0000 7f670f59e700 10 monclient: _finish_hunting 0 2023-09-30T02:10:59.498 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.494+0000 7f670f59e700 1 monclient: found mon.noname-c 2023-09-30T02:10:59.498 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.494+0000 7f670f59e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:10:59.498 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.494+0000 7f670f59e700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3300/0 2023-09-30T02:10:59.498 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.494+0000 7f670f59e700 1 -- 172.21.15.59:0/410171670 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6710142500 con 0x7f67101420c0 2023-09-30T02:10:59.498 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.496+0000 7f670f59e700 1 --2- 172.21.15.59:0/410171670 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f67101420c0 0x7f671013e230 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f67100b0970 tx=0x7f6708009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=76713ae5ade07a39 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:10:59.501 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.500+0000 7f670dd9b700 1 -- 172.21.15.59:0/410171670 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f670800e070 con 0x7f67101420c0 2023-09-30T02:10:59.501 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.500+0000 7f670dd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:10:59.501 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.500+0000 7f670dd9b700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:10:59.501 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.500+0000 7f670dd9b700 10 monclient: dump: 2023-09-30T02:10:59.501 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:10:59.501 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:59.501 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:59.501 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:59.501 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:10:59.502 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:59.502 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:10:59.502 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:10:59.502 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:10:59.502 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:59.502 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.500+0000 7f670dd9b700 10 monclient: _finish_auth 0 2023-09-30T02:10:59.502 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.500+0000 7f670dd9b700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:10:59.502 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.500+0000 7f670dd9b700 1 -- 172.21.15.59:0/410171670 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f670800bab0 con 0x7f67101420c0 2023-09-30T02:10:59.502 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.500+0000 7f670dd9b700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:10:59.502 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.500+0000 7f670dd9b700 1 -- 172.21.15.59:0/410171670 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f670800be70 con 0x7f67101420c0 2023-09-30T02:10:59.502 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.500+0000 7f670dd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:10:59.502 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.500+0000 7f670dd9b700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:10:59.503 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.500+0000 7f670dd9b700 10 monclient: dump: 2023-09-30T02:10:59.503 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:10:59.503 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:59.503 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:59.503 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:59.503 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:10:59.503 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:59.503 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:10:59.503 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:10:59.503 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:10:59.503 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:59.503 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.500+0000 7f6715014700 5 monclient: authenticate success, global_id 4798 2023-09-30T02:10:59.503 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.500+0000 7f6715014700 10 monclient: get_monmap_and_config success 2023-09-30T02:10:59.504 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.500+0000 7f6715014700 4 set_mon_vals no callback set 2023-09-30T02:10:59.507 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.501+0000 7f6715014700 10 monclient: shutdown 2023-09-30T02:10:59.507 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.501+0000 7f6715014700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:10:59.507 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.501+0000 7f6715014700 1 -- 172.21.15.59:0/410171670 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f67101420c0 msgr2=0x7f671013e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:59.507 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.501+0000 7f6715014700 1 --2- 172.21.15.59:0/410171670 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f67101420c0 0x7f671013e230 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f67100b0970 tx=0x7f6708009780 comp rx=0 tx=0).stop 2023-09-30T02:10:59.507 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.501+0000 7f6715014700 1 -- 172.21.15.59:0/410171670 shutdown_connections 2023-09-30T02:10:59.507 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.501+0000 7f6715014700 1 --2- 172.21.15.59:0/410171670 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f67100a5b70 0x7f67100b1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:59.507 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.501+0000 7f6715014700 1 --2- 172.21.15.59:0/410171670 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f67101420c0 0x7f671013e230 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:59.507 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.501+0000 7f6715014700 1 --2- 172.21.15.59:0/410171670 >> v2:172.21.15.59:3300/0 conn(0x7f67100a5f40 0x7f6710141b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:59.507 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.501+0000 7f6715014700 1 -- 172.21.15.59:0/410171670 >> 172.21.15.59:0/410171670 conn(0x7f67100631f0 msgr2=0x7f67100635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:10:59.508 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.501+0000 7f6715014700 1 -- 172.21.15.59:0/410171670 shutdown_connections 2023-09-30T02:10:59.508 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.501+0000 7f6715014700 1 -- 172.21.15.59:0/410171670 wait complete. 2023-09-30T02:10:59.508 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.502+0000 7f6715014700 10 monclient: build_initial_monmap 2023-09-30T02:10:59.508 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.502+0000 7f6715014700 10 monclient: monmap: 2023-09-30T02:10:59.508 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:10:59.508 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:10:59.508 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:10:59.508 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:10:59.508 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:10:59.508 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:10:59.508 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:10:59.508 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:10:59.508 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:10:59.509 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:59.509 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.502+0000 7f6715014700 1 librados: starting msgr at 2023-09-30T02:10:59.509 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.502+0000 7f6715014700 1 librados: starting objecter 2023-09-30T02:10:59.509 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.502+0000 7f6715014700 1 Processor -- start 2023-09-30T02:10:59.509 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.503+0000 7f6715014700 1 -- start start 2023-09-30T02:10:59.509 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.503+0000 7f6715014700 1 librados: setting wanted keys 2023-09-30T02:10:59.509 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.503+0000 7f6715014700 1 librados: calling monclient init 2023-09-30T02:10:59.509 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.503+0000 7f6715014700 10 monclient: init 2023-09-30T02:10:59.509 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.503+0000 7f6715014700 10 monclient: _reopen_session rank -1 2023-09-30T02:10:59.509 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.503+0000 7f6715014700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T02:10:59.509 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.503+0000 7f6715014700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f67100a5b70 0x7f671013acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:59.509 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.503+0000 7f6715014700 10 monclient(hunting): picked mon.noname-c con 0x7f67100a5b70 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:10:59.509 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.503+0000 7f6715014700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f67100a5f40 0x7f671014d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:59.510 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.503+0000 7f6715014700 10 monclient(hunting): picked mon.noname-b con 0x7f67100a5f40 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:10:59.510 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.503+0000 7f6715014700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67101420c0 0x7f671014dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:59.510 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.503+0000 7f6715014700 10 monclient(hunting): picked mon.noname-a con 0x7f67101420c0 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:10:59.510 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.503+0000 7f6715014700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:59.510 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.503+0000 7f6715014700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67100aa0c0 con 0x7f67101420c0 2023-09-30T02:10:59.510 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.503+0000 7f6715014700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:59.510 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.503+0000 7f6715014700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f671013ba20 con 0x7f67100a5f40 2023-09-30T02:10:59.510 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.503+0000 7f6715014700 10 monclient(hunting): start opening mon connection 2023-09-30T02:10:59.510 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.503+0000 7f6715014700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67100a9ba0 con 0x7f67100a5b70 2023-09-30T02:10:59.510 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.503+0000 7f6715014700 10 monclient(hunting): _renew_subs 2023-09-30T02:10:59.510 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.503+0000 7f6715014700 10 monclient(hunting): authenticate will time out at 2104.806689s 2023-09-30T02:10:59.510 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.504+0000 7f670e59c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f67100a5f40 0x7f671014d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:59.511 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.504+0000 7f670ed9d700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f67100a5b70 0x7f671013acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:59.511 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.504+0000 7f670f59e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67101420c0 0x7f671014dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:59.511 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.504+0000 7f670e59c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f67100a5f40 0x7f671014d780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:46012/0 (socket says 172.21.15.59:46012) 2023-09-30T02:10:59.511 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.504+0000 7f670e59c700 1 -- 172.21.15.59:0/2497445228 learned_addr learned my addr 172.21.15.59:0/2497445228 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:10:59.511 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.504+0000 7f670e59c700 10 monclient(hunting): get_auth_request con 0x7f67100a5f40 auth_method 0 2023-09-30T02:10:59.511 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.504+0000 7f670e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:10:59.511 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.504+0000 7f670e59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:59.511 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.504+0000 7f670e59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:59.511 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.504+0000 7f670ed9d700 10 monclient(hunting): get_auth_request con 0x7f67100a5b70 auth_method 0 2023-09-30T02:10:59.511 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.504+0000 7f670ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:10:59.511 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.504+0000 7f670ed9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:59.511 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.504+0000 7f670ed9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:59.512 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.504+0000 7f670f59e700 10 monclient(hunting): get_auth_request con 0x7f67101420c0 auth_method 0 2023-09-30T02:10:59.512 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.504+0000 7f670f59e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:10:59.512 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.504+0000 7f670f59e700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:10:59.512 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.504+0000 7f670f59e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:10:59.512 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.505+0000 7f670e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:10:59.512 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.505+0000 7f670e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:10:59.512 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.505+0000 7f670e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:10:59.512 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.505+0000 7f670ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:10:59.512 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.505+0000 7f670ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:10:59.512 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.505+0000 7f670ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:10:59.512 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.505+0000 7f670f59e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:10:59.512 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.505+0000 7f670f59e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:10:59.513 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.505+0000 7f670f59e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:10:59.513 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.507+0000 7f670e59c700 10 monclient(hunting): handle_auth_done global_id 4801 payload 824 2023-09-30T02:10:59.513 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.508+0000 7f670e59c700 1 -- 172.21.15.59:0/2497445228 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f67100a5b70 msgr2=0x7f671013acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:59.513 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.508+0000 7f670e59c700 1 --2- 172.21.15.59:0/2497445228 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f67100a5b70 0x7f671013acf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:59.513 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.508+0000 7f670e59c700 1 -- 172.21.15.59:0/2497445228 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67101420c0 msgr2=0x7f671014dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:10:59.514 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.508+0000 7f670e59c700 1 --2- 172.21.15.59:0/2497445228 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67101420c0 0x7f671014dcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:10:59.514 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.508+0000 7f670e59c700 10 monclient: _finish_hunting 0 2023-09-30T02:10:59.514 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.508+0000 7f670e59c700 1 monclient: found mon.noname-b 2023-09-30T02:10:59.514 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.508+0000 7f670e59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:10:59.514 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.508+0000 7f670e59c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:10:59.514 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.508+0000 7f670e59c700 1 -- 172.21.15.59:0/2497445228 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6708009a20 con 0x7f67100a5f40 2023-09-30T02:10:59.514 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.510+0000 7f670e59c700 1 --2- 172.21.15.59:0/2497445228 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f67100a5f40 0x7f671014d780 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f670400ecb0 tx=0x7f67040091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:10:59.514 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.510+0000 7f66f6ffd700 10 client.?.objecter ms_handle_connect 0x7f67100a5f40 2023-09-30T02:10:59.514 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.510+0000 7f66f6ffd700 10 client.?.objecter resend_mon_ops 2023-09-30T02:10:59.515 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.514+0000 7f66f6ffd700 1 -- 172.21.15.59:0/2497445228 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f670400c850 con 0x7f67100a5f40 2023-09-30T02:10:59.516 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.514+0000 7f66f6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:10:59.516 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.514+0000 7f66f6ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:10:59.516 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.514+0000 7f66f6ffd700 10 monclient: dump: 2023-09-30T02:10:59.516 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:10:59.516 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:59.516 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:59.516 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:59.516 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:10:59.516 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:59.516 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:10:59.516 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:10:59.516 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:10:59.517 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:59.517 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.514+0000 7f66f6ffd700 10 monclient: _finish_auth 0 2023-09-30T02:10:59.517 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.514+0000 7f66f6ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:10:59.517 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.514+0000 7f66f6ffd700 1 -- 172.21.15.59:0/2497445228 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f670400c9b0 con 0x7f67100a5f40 2023-09-30T02:10:59.517 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.514+0000 7f66f6ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:10:59.517 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.514+0000 7f66f6ffd700 1 -- 172.21.15.59:0/2497445228 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f670400cd50 con 0x7f67100a5f40 2023-09-30T02:10:59.517 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.514+0000 7f6715014700 5 monclient: authenticate success, global_id 4801 2023-09-30T02:10:59.517 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.514+0000 7f6715014700 10 monclient: _renew_subs 2023-09-30T02:10:59.517 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.514+0000 7f6715014700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:10:59.517 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.514+0000 7f6715014700 1 -- 172.21.15.59:0/2497445228 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67100a8e80 con 0x7f67100a5f40 2023-09-30T02:10:59.517 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.514+0000 7f66f6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:10:59.517 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.514+0000 7f66f6ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:10:59.517 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.514+0000 7f66f6ffd700 10 monclient: dump: 2023-09-30T02:10:59.518 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:10:59.518 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:10:59.518 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:59.518 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:10:59.518 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:10:59.518 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:10:59.518 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:10:59.518 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:10:59.518 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:10:59.518 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:10:59.518 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.514+0000 7f670d59a700 4 set_mon_vals no callback set 2023-09-30T02:10:59.518 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.514+0000 7f6715014700 10 client.4801.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:10:59.519 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.514+0000 7f6715014700 10 monclient: _renew_subs 2023-09-30T02:10:59.519 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.514+0000 7f6715014700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:10:59.519 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.514+0000 7f6715014700 1 -- 172.21.15.59:0/2497445228 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67100aa290 con 0x7f67100a5f40 2023-09-30T02:10:59.519 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.514+0000 7f6715014700 1 librados: init done 2023-09-30T02:10:59.519 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.515+0000 7f6715014700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:10:59.519 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.515+0000 7f6715014700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:10:59.519 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.515+0000 7f6715014700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:10:59.519 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.515+0000 7f6715014700 1 -- 172.21.15.59:0/2497445228 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67100a9dc0 con 0x7f67100a5f40 2023-09-30T02:10:59.581 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.581+0000 7f66f6ffd700 1 -- 172.21.15.59:0/2497445228 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f6704013680 con 0x7f67100a5f40 2023-09-30T02:10:59.583 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.581+0000 7f66f6ffd700 1 --2- 172.21.15.59:0/2497445228 >> v2:172.21.15.110:6800/115870 conn(0x7f66fc068890 0x7f66fc06ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:10:59.583 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.582+0000 7f66f6ffd700 1 -- 172.21.15.59:0/2497445228 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f6704084900 con 0x7f67100a5f40 2023-09-30T02:10:59.583 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.582+0000 7f66f6ffd700 10 client.4801.objecter ms_dispatch 0x7f6710001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:10:59.583 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.582+0000 7f66f6ffd700 3 client.4801.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:10:59.583 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.582+0000 7f66f6ffd700 3 client.4801.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:10:59.583 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.582+0000 7f66f6ffd700 20 client.4801.objecter dump_active .. 0 homeless 2023-09-30T02:10:59.583 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.582+0000 7f670ed9d700 1 --2- 172.21.15.59:0/2497445228 >> v2:172.21.15.110:6800/115870 conn(0x7f66fc068890 0x7f66fc06ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:10:59.583 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.582+0000 7f670ed9d700 10 monclient: get_auth_request con 0x7f66fc068890 auth_method 0 2023-09-30T02:10:59.583 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.582+0000 7f670ed9d700 1 --2- 172.21.15.59:0/2497445228 >> v2:172.21.15.110:6800/115870 conn(0x7f66fc068890 0x7f66fc06ad40 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f6700006d40 tx=0x7f6700009250 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:10:59.583 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.582+0000 7f66f6ffd700 10 client.4801.objecter ms_handle_connect 0x7f66fc068890 2023-09-30T02:10:59.782 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.781+0000 7f66f6ffd700 1 -- 172.21.15.59:0/2497445228 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f67040521a0 con 0x7f67100a5f40 2023-09-30T02:10:59.782 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.782+0000 7f66f6ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:10:59.782 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:10:59.782+0000 7f66f6ffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:11:00.039 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:00.038+0000 7f6715014700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:11:00.039 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:00.038+0000 7f6715014700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:11:00.039 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:00.038+0000 7f6715014700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:11:00.039 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:00.038+0000 7f6715014700 1 -- 172.21.15.59:0/2497445228 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f6710064a00 con 0x7f67100a5f40 2023-09-30T02:11:00.044 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:00.044+0000 7f66f6ffd700 1 -- 172.21.15.59:0/2497445228 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f67040557c0 con 0x7f67100a5f40 2023-09-30T02:11:00.044 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:00.044+0000 7f66f6ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:11:00.044 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:00.044+0000 7f66f6ffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:11:00.044 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:11:00.045 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:11:00.047 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:00.046+0000 7f6715014700 10 librados: watch_flush enter 2023-09-30T02:11:00.047 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:00.046+0000 7f6715014700 10 librados: watch_flush exit 2023-09-30T02:11:00.047 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:00.046+0000 7f6715014700 20 client.4801.objecter shutdown clearing up homeless session... 2023-09-30T02:11:00.047 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:00.046+0000 7f6715014700 10 client.4801.objecter successfully canceled tick 2023-09-30T02:11:00.047 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:00.046+0000 7f6715014700 1 -- 172.21.15.59:0/2497445228 >> v2:172.21.15.110:6800/115870 conn(0x7f66fc068890 msgr2=0x7f66fc06ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:00.047 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:00.046+0000 7f6715014700 1 --2- 172.21.15.59:0/2497445228 >> v2:172.21.15.110:6800/115870 conn(0x7f66fc068890 0x7f66fc06ad40 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f6700006d40 tx=0x7f6700009250 comp rx=0 tx=0).stop 2023-09-30T02:11:00.047 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:00.046+0000 7f6715014700 10 monclient: shutdown 2023-09-30T02:11:00.047 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:00.046+0000 7f6715014700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:11:00.047 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:00.046+0000 7f6715014700 1 -- 172.21.15.59:0/2497445228 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f67100a5f40 msgr2=0x7f671014d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:00.048 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:00.046+0000 7f6715014700 1 --2- 172.21.15.59:0/2497445228 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f67100a5f40 0x7f671014d780 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f670400ecb0 tx=0x7f67040091a0 comp rx=0 tx=0).stop 2023-09-30T02:11:00.048 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:00.047+0000 7f6715014700 1 -- 172.21.15.59:0/2497445228 shutdown_connections 2023-09-30T02:11:00.048 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:00.047+0000 7f6715014700 1 --2- 172.21.15.59:0/2497445228 >> v2:172.21.15.110:6800/115870 conn(0x7f66fc068890 0x7f66fc06ad40 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:00.049 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:00.047+0000 7f6715014700 1 --2- 172.21.15.59:0/2497445228 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67101420c0 0x7f671014dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:00.049 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:00.047+0000 7f6715014700 1 --2- 172.21.15.59:0/2497445228 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f67100a5b70 0x7f671013acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:00.049 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:00.047+0000 7f6715014700 1 --2- 172.21.15.59:0/2497445228 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f67100a5f40 0x7f671014d780 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:00.049 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:00.047+0000 7f6715014700 1 -- 172.21.15.59:0/2497445228 >> 172.21.15.59:0/2497445228 conn(0x7f67100631f0 msgr2=0x7f67100b0be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:11:00.049 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:00.047+0000 7f6715014700 1 -- 172.21.15.59:0/2497445228 shutdown_connections 2023-09-30T02:11:00.049 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:00.047+0000 7f6715014700 1 -- 172.21.15.59:0/2497445228 wait complete. 2023-09-30T02:11:00.049 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:00.047+0000 7f6715014700 1 librados: shutdown 2023-09-30T02:11:03.062 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:11:03.201 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.200+0000 7f45d7c22700 10 monclient: get_monmap_and_config 2023-09-30T02:11:03.201 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.200+0000 7f45d7c22700 10 monclient: build_initial_monmap 2023-09-30T02:11:03.201 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.200+0000 7f45d7c22700 10 monclient: monmap: 2023-09-30T02:11:03.201 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:11:03.201 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:03.201 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:11:03.202091+0000 2023-09-30T02:11:03.202 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:11:03.202091+0000 2023-09-30T02:11:03.202 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:11:03.202 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:03.202 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:11:03.202 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:11:03.202 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:11:03.202 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:11:03.203 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:11:03.203 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:03.205 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.201+0000 7f45d7c22700 1 Processor -- start 2023-09-30T02:11:03.205 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.201+0000 7f45d7c22700 1 -- start start 2023-09-30T02:11:03.205 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.201+0000 7f45d7c22700 10 monclient: init 2023-09-30T02:11:03.205 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.202+0000 7f45d7c22700 10 monclient: _reopen_session rank -1 2023-09-30T02:11:03.205 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.202+0000 7f45d7c22700 10 monclient: _add_conns ranks=[0,1,2,4,3] 2023-09-30T02:11:03.205 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.202+0000 7f45d7c22700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f45d00b16e0 addr v1:172.21.15.59:6789/0 2023-09-30T02:11:03.205 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.202+0000 7f45d7c22700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f45d00a7650 addr v1:172.21.15.110:6789/0 2023-09-30T02:11:03.206 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.202+0000 7f45d7c22700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f45d0146d30 0x7f45d0149110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:03.206 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.202+0000 7f45d7c22700 10 monclient(hunting): picked mon.noname-b con 0x7f45d0146d30 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:11:03.206 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.202+0000 7f45d7c22700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45d00a5d40 con 0x7f45d00b16e0 2023-09-30T02:11:03.206 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.202+0000 7f45d7c22700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45d00b1ab0 con 0x7f45d00a7650 2023-09-30T02:11:03.206 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.202+0000 7f45d7c22700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:03.206 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.202+0000 7f45d7c22700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45d00a6c70 con 0x7f45d0146d30 2023-09-30T02:11:03.206 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.202+0000 7f45d7c22700 10 monclient(hunting): _renew_subs 2023-09-30T02:11:03.206 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.202+0000 7f45d7c22700 10 monclient(hunting): authenticate will time out at 2108.505390s 2023-09-30T02:11:03.207 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.202+0000 7f45d61bf700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f45d0146d30 0x7f45d0149110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:03.207 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.203+0000 7f45d59be700 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f45d00b16e0 0x7f45d00a54e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:54040/0 (socket says 172.21.15.59:54040) 2023-09-30T02:11:03.207 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.203+0000 7f45d59be700 1 -- 172.21.15.59:0/78416013 learned_addr learned my addr 172.21.15.59:0/78416013 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:11:03.207 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.203+0000 7f45d61bf700 10 monclient(hunting): get_auth_request con 0x7f45d0146d30 auth_method 0 2023-09-30T02:11:03.207 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.203+0000 7f45d61bf700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:03.207 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.203+0000 7f45d61bf700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:03.208 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.203+0000 7f45d61bf700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:03.208 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.204+0000 7f45d61bf700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:03.208 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.204+0000 7f45d61bf700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:03.208 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.204+0000 7f45d61bf700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:03.209 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.205+0000 7f45d49bc700 1 -- 172.21.15.59:0/78416013 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1548652392 0 0) 0x7f45d00a5d40 con 0x7f45d00b16e0 2023-09-30T02:11:03.209 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.205+0000 7f45d49bc700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:03.209 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.205+0000 7f45d49bc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:03.210 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.205+0000 7f45d49bc700 10 monclient(hunting): my global_id is 4776 2023-09-30T02:11:03.210 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.205+0000 7f45d49bc700 1 -- 172.21.15.59:0/78416013 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45bc003570 con 0x7f45d00b16e0 2023-09-30T02:11:03.210 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.206+0000 7f45d61bf700 10 monclient(hunting): handle_auth_done global_id 4700 payload 274 2023-09-30T02:11:03.210 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.206+0000 7f45d49bc700 1 -- 172.21.15.59:0/78416013 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 679580815 0 0) 0x7f45d00b1ab0 con 0x7f45d00a7650 2023-09-30T02:11:03.210 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.206+0000 7f45d61bf700 1 -- 172.21.15.59:0/78416013 >> v1:172.21.15.110:6789/0 conn(0x7f45d00a7650 legacy=0x7f45d00a7a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:03.210 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.206+0000 7f45d61bf700 1 -- 172.21.15.59:0/78416013 >> v1:172.21.15.59:6789/0 conn(0x7f45d00b16e0 legacy=0x7f45d00a54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:03.211 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.207+0000 7f45d61bf700 10 monclient: _finish_hunting 0 2023-09-30T02:11:03.211 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.207+0000 7f45d61bf700 1 monclient: found mon.noname-b 2023-09-30T02:11:03.211 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.207+0000 7f45d61bf700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:11:03.211 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.207+0000 7f45d61bf700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.59:3301/0 2023-09-30T02:11:03.211 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.207+0000 7f45d61bf700 1 -- 172.21.15.59:0/78416013 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45d00a6f50 con 0x7f45d0146d30 2023-09-30T02:11:03.211 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.207+0000 7f45d49bc700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 2023-09-30T02:11:03.211 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.209+0000 7f45d61bf700 1 --2- 172.21.15.59:0/78416013 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f45d0146d30 0x7f45d0149110 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f45c8009e90 tx=0x7f45c8009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b431bbcc897cb4a3 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:03.215 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.213+0000 7f45d49bc700 1 -- 172.21.15.59:0/78416013 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45c800e070 con 0x7f45d0146d30 2023-09-30T02:11:03.215 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.213+0000 7f45d49bc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:03.215 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.213+0000 7f45d49bc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:11:03.215 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.213+0000 7f45d49bc700 10 monclient: dump: 2023-09-30T02:11:03.215 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:03.215 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:03.215 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:03.216 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:03.216 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:03.216 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:03.216 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:03.216 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:03.216 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:03.216 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:03.216 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.213+0000 7f45d49bc700 10 monclient: _finish_auth 0 2023-09-30T02:11:03.217 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.213+0000 7f45d49bc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:11:03.217 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.213+0000 7f45d49bc700 1 -- 172.21.15.59:0/78416013 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45c800bc30 con 0x7f45d0146d30 2023-09-30T02:11:03.217 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.213+0000 7f45d7c22700 5 monclient: authenticate success, global_id 4700 2023-09-30T02:11:03.217 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.213+0000 7f45d7c22700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:11:03.217 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.213+0000 7f45d49bc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:11:03.217 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.213+0000 7f45d7c22700 10 monclient: get_monmap_and_config success 2023-09-30T02:11:03.217 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.213+0000 7f45d7c22700 4 set_mon_vals no callback set 2023-09-30T02:11:03.217 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.213+0000 7f45d49bc700 1 -- 172.21.15.59:0/78416013 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45c8012430 con 0x7f45d0146d30 2023-09-30T02:11:03.218 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.213+0000 7f45d49bc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:03.218 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.213+0000 7f45d49bc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:11:03.218 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.213+0000 7f45d49bc700 10 monclient: dump: 2023-09-30T02:11:03.218 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:03.218 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:03.218 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:03.218 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:03.218 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:03.219 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:03.219 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:03.219 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:03.219 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:03.219 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:03.219 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.214+0000 7f45d7c22700 10 monclient: shutdown 2023-09-30T02:11:03.219 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.214+0000 7f45d7c22700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:11:03.219 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.214+0000 7f45d7c22700 1 -- 172.21.15.59:0/78416013 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f45d0146d30 msgr2=0x7f45d0149110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:03.220 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.214+0000 7f45d7c22700 1 --2- 172.21.15.59:0/78416013 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f45d0146d30 0x7f45d0149110 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f45c8009e90 tx=0x7f45c8009780 comp rx=0 tx=0).stop 2023-09-30T02:11:03.220 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.214+0000 7f45d7c22700 1 -- 172.21.15.59:0/78416013 shutdown_connections 2023-09-30T02:11:03.220 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.214+0000 7f45d7c22700 1 --2- 172.21.15.59:0/78416013 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f45d0146d30 0x7f45d0149110 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:03.220 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.214+0000 7f45d7c22700 1 -- 172.21.15.59:0/78416013 >> 172.21.15.59:0/78416013 conn(0x7f45d0068fe0 msgr2=0x7f45d00631f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:11:03.220 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.214+0000 7f45d7c22700 1 -- 172.21.15.59:0/78416013 shutdown_connections 2023-09-30T02:11:03.220 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.214+0000 7f45d7c22700 1 -- 172.21.15.59:0/78416013 wait complete. 2023-09-30T02:11:03.224 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.215+0000 7f45d7c22700 10 monclient: build_initial_monmap 2023-09-30T02:11:03.224 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.215+0000 7f45d7c22700 10 monclient: monmap: 2023-09-30T02:11:03.224 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:11:03.224 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:11:03.224 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:11:03.224 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:11:03.225 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:11:03.225 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:11:03.225 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:11:03.225 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:11:03.225 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:11:03.225 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:03.225 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.215+0000 7f45d7c22700 1 librados: starting msgr at 2023-09-30T02:11:03.225 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.215+0000 7f45d7c22700 1 librados: starting objecter 2023-09-30T02:11:03.226 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.215+0000 7f45d7c22700 1 Processor -- start 2023-09-30T02:11:03.226 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.215+0000 7f45d7c22700 1 -- start start 2023-09-30T02:11:03.226 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.216+0000 7f45d7c22700 1 librados: setting wanted keys 2023-09-30T02:11:03.226 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.216+0000 7f45d7c22700 1 librados: calling monclient init 2023-09-30T02:11:03.226 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.216+0000 7f45d7c22700 10 monclient: init 2023-09-30T02:11:03.226 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.216+0000 7f45d7c22700 10 monclient: _reopen_session rank -1 2023-09-30T02:11:03.226 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.216+0000 7f45d7c22700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T02:11:03.226 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.216+0000 7f45d7c22700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f45d00a7650 0x7f45d00aaab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:03.227 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.216+0000 7f45d7c22700 10 monclient(hunting): picked mon.noname-c con 0x7f45d00a7650 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:11:03.227 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.216+0000 7f45d7c22700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45d00b16e0 0x7f45d0138280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:03.227 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.216+0000 7f45d7c22700 10 monclient(hunting): picked mon.noname-a con 0x7f45d00b16e0 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:11:03.227 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.216+0000 7f45d7c22700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f45d0146d30 0x7f45d01387c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:03.227 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.216+0000 7f45d7c22700 10 monclient(hunting): picked mon.noname-b con 0x7f45d0146d30 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:11:03.227 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.216+0000 7f45d7c22700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:03.227 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.216+0000 7f45d7c22700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45d00b2d40 con 0x7f45d00b16e0 2023-09-30T02:11:03.227 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.216+0000 7f45d7c22700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:03.228 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.216+0000 7f45d7c22700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45d013e200 con 0x7f45d0146d30 2023-09-30T02:11:03.228 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.216+0000 7f45d7c22700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:03.228 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.216+0000 7f45d7c22700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45d013d450 con 0x7f45d00a7650 2023-09-30T02:11:03.228 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.216+0000 7f45d7c22700 10 monclient(hunting): _renew_subs 2023-09-30T02:11:03.228 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.216+0000 7f45d7c22700 10 monclient(hunting): authenticate will time out at 2108.519727s 2023-09-30T02:11:03.228 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.217+0000 7f45d59be700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f45d00a7650 0x7f45d00aaab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:03.228 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.217+0000 7f45d51bd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45d00b16e0 0x7f45d0138280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:03.228 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.217+0000 7f45d59be700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f45d00a7650 0x7f45d00aaab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:52926/0 (socket says 172.21.15.59:52926) 2023-09-30T02:11:03.229 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.217+0000 7f45d59be700 1 -- 172.21.15.59:0/455399021 learned_addr learned my addr 172.21.15.59:0/455399021 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:11:03.229 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.217+0000 7f45d59be700 10 monclient(hunting): get_auth_request con 0x7f45d00a7650 auth_method 0 2023-09-30T02:11:03.229 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.217+0000 7f45d59be700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:03.229 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.217+0000 7f45d59be700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:03.229 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.217+0000 7f45d59be700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:03.229 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.217+0000 7f45d61bf700 1 --2- 172.21.15.59:0/455399021 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f45d0146d30 0x7f45d01387c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:03.229 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.217+0000 7f45d51bd700 10 monclient(hunting): get_auth_request con 0x7f45d00b16e0 auth_method 0 2023-09-30T02:11:03.230 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.217+0000 7f45d51bd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:03.230 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.217+0000 7f45d51bd700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:03.230 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.217+0000 7f45d51bd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:03.230 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.218+0000 7f45d61bf700 10 monclient(hunting): get_auth_request con 0x7f45d0146d30 auth_method 0 2023-09-30T02:11:03.230 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.218+0000 7f45d61bf700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:03.230 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.218+0000 7f45d61bf700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:03.230 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.218+0000 7f45d61bf700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:03.230 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.218+0000 7f45d59be700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:03.231 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.218+0000 7f45d59be700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:03.231 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.218+0000 7f45d59be700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:03.231 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.218+0000 7f45d51bd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:03.231 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.218+0000 7f45d51bd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:03.231 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.218+0000 7f45d51bd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:03.231 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.218+0000 7f45d61bf700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:03.231 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.218+0000 7f45d61bf700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:03.232 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.218+0000 7f45d61bf700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:03.232 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.222+0000 7f45d59be700 10 monclient(hunting): handle_auth_done global_id 4703 payload 824 2023-09-30T02:11:03.232 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.222+0000 7f45d59be700 1 -- 172.21.15.59:0/455399021 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f45d0146d30 msgr2=0x7f45d01387c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:03.232 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.222+0000 7f45d59be700 1 --2- 172.21.15.59:0/455399021 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f45d0146d30 0x7f45d01387c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:03.232 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.222+0000 7f45d59be700 1 -- 172.21.15.59:0/455399021 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45d00b16e0 msgr2=0x7f45d0138280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:03.233 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.222+0000 7f45d59be700 1 --2- 172.21.15.59:0/455399021 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45d00b16e0 0x7f45d0138280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:03.233 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.222+0000 7f45d59be700 10 monclient: _finish_hunting 0 2023-09-30T02:11:03.233 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.222+0000 7f45d59be700 1 monclient: found mon.noname-c 2023-09-30T02:11:03.233 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.222+0000 7f45d59be700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:11:03.233 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.222+0000 7f45d59be700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.59:3301/0 2023-09-30T02:11:03.233 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.222+0000 7f45d59be700 1 -- 172.21.15.59:0/455399021 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45c8009a20 con 0x7f45d00a7650 2023-09-30T02:11:03.235 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.224+0000 7f45d59be700 1 --2- 172.21.15.59:0/455399021 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f45d00a7650 0x7f45d00aaab0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f45cc00c7c0 tx=0x7f45cc007e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:03.235 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.224+0000 7f45c1ffb700 10 client.?.objecter ms_handle_connect 0x7f45d00a7650 2023-09-30T02:11:03.235 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.224+0000 7f45c1ffb700 10 client.?.objecter resend_mon_ops 2023-09-30T02:11:03.235 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.228+0000 7f45c1ffb700 1 -- 172.21.15.59:0/455399021 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45cc005b30 con 0x7f45d00a7650 2023-09-30T02:11:03.235 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.228+0000 7f45c1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:03.236 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.228+0000 7f45c1ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:11:03.236 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.228+0000 7f45c1ffb700 10 monclient: dump: 2023-09-30T02:11:03.236 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:03.236 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:03.236 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:03.236 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:03.236 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:03.236 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:03.237 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:03.237 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:03.237 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:03.237 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:03.237 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.228+0000 7f45c1ffb700 10 monclient: _finish_auth 0 2023-09-30T02:11:03.237 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.228+0000 7f45c1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:11:03.237 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.228+0000 7f45c1ffb700 1 -- 172.21.15.59:0/455399021 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45cc005c90 con 0x7f45d00a7650 2023-09-30T02:11:03.238 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.228+0000 7f45d7c22700 5 monclient: authenticate success, global_id 4703 2023-09-30T02:11:03.238 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.228+0000 7f45d7c22700 10 monclient: _renew_subs 2023-09-30T02:11:03.238 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.228+0000 7f45d7c22700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:11:03.238 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.228+0000 7f45d7c22700 1 -- 172.21.15.59:0/455399021 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45d00aaff0 con 0x7f45d00a7650 2023-09-30T02:11:03.238 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.228+0000 7f45c1ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:11:03.238 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.228+0000 7f45c1ffb700 1 -- 172.21.15.59:0/455399021 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45cc007300 con 0x7f45d00a7650 2023-09-30T02:11:03.238 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.228+0000 7f45c1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:03.239 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.228+0000 7f45c1ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:11:03.239 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.228+0000 7f45c1ffb700 10 monclient: dump: 2023-09-30T02:11:03.239 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:03.239 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:03.239 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:03.239 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:03.239 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:03.239 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:03.240 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:03.240 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:03.240 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:03.240 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:03.240 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.228+0000 7f45c3fff700 4 set_mon_vals no callback set 2023-09-30T02:11:03.240 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.229+0000 7f45d7c22700 10 client.4703.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:11:03.240 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.229+0000 7f45d7c22700 10 monclient: _renew_subs 2023-09-30T02:11:03.241 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.229+0000 7f45d7c22700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:11:03.241 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.229+0000 7f45d7c22700 1 -- 172.21.15.59:0/455399021 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45d013ef60 con 0x7f45d00a7650 2023-09-30T02:11:03.241 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.229+0000 7f45d7c22700 1 librados: init done 2023-09-30T02:11:03.241 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.230+0000 7f45d7c22700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:11:03.241 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.230+0000 7f45d7c22700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:11:03.241 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.230+0000 7f45d7c22700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:11:03.241 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.230+0000 7f45d7c22700 1 -- 172.21.15.59:0/455399021 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45d013f150 con 0x7f45d00a7650 2023-09-30T02:11:03.369 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.369+0000 7f45c1ffb700 1 -- 172.21.15.59:0/455399021 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f45cc01b3b0 con 0x7f45d00a7650 2023-09-30T02:11:03.370 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.369+0000 7f45c1ffb700 1 --2- 172.21.15.59:0/455399021 >> v2:172.21.15.110:6800/115870 conn(0x7f45b4068960 0x7f45b406ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:03.370 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.369+0000 7f45c1ffb700 1 -- 172.21.15.59:0/455399021 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f45cc083760 con 0x7f45d00a7650 2023-09-30T02:11:03.370 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.369+0000 7f45c1ffb700 10 client.4703.objecter ms_dispatch 0x7f45d0001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:11:03.370 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.369+0000 7f45c1ffb700 3 client.4703.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:11:03.370 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.369+0000 7f45c1ffb700 3 client.4703.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:11:03.370 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.369+0000 7f45c1ffb700 20 client.4703.objecter dump_active .. 0 homeless 2023-09-30T02:11:03.370 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.369+0000 7f45d51bd700 1 --2- 172.21.15.59:0/455399021 >> v2:172.21.15.110:6800/115870 conn(0x7f45b4068960 0x7f45b406ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:03.370 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.369+0000 7f45d51bd700 10 monclient: get_auth_request con 0x7f45b4068960 auth_method 0 2023-09-30T02:11:03.371 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.370+0000 7f45d51bd700 1 --2- 172.21.15.59:0/455399021 >> v2:172.21.15.110:6800/115870 conn(0x7f45b4068960 0x7f45b406ae10 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f45d013ec90 tx=0x7f45c4007ec0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:03.371 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.370+0000 7f45c1ffb700 10 client.4703.objecter ms_handle_connect 0x7f45b4068960 2023-09-30T02:11:03.526 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.525+0000 7f45c1ffb700 1 -- 172.21.15.59:0/455399021 <== mon.2 v2:172.21.15.59:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f45cc051180 con 0x7f45d00a7650 2023-09-30T02:11:03.526 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.525+0000 7f45c1ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:11:03.526 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.525+0000 7f45c1ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:11:03.779 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.778+0000 7f45d7c22700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:11:03.779 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.778+0000 7f45d7c22700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:11:03.779 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.778+0000 7f45d7c22700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:11:03.779 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.778+0000 7f45d7c22700 1 -- 172.21.15.59:0/455399021 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f45d0072b30 con 0x7f45d00a7650 2023-09-30T02:11:03.821 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.820+0000 7f45c1ffb700 1 -- 172.21.15.59:0/455399021 <== mon.2 v2:172.21.15.59:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f45cc0547a0 con 0x7f45d00a7650 2023-09-30T02:11:03.821 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.820+0000 7f45c1ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:11:03.821 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.820+0000 7f45c1ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:11:03.821 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:11:03.822 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:11:03.824 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.823+0000 7f45d7c22700 10 librados: watch_flush enter 2023-09-30T02:11:03.824 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.823+0000 7f45d7c22700 10 librados: watch_flush exit 2023-09-30T02:11:03.824 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.823+0000 7f45d7c22700 20 client.4703.objecter shutdown clearing up homeless session... 2023-09-30T02:11:03.824 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.823+0000 7f45d7c22700 10 client.4703.objecter successfully canceled tick 2023-09-30T02:11:03.824 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.823+0000 7f45d7c22700 1 -- 172.21.15.59:0/455399021 >> v2:172.21.15.110:6800/115870 conn(0x7f45b4068960 msgr2=0x7f45b406ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:03.824 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.823+0000 7f45d7c22700 1 --2- 172.21.15.59:0/455399021 >> v2:172.21.15.110:6800/115870 conn(0x7f45b4068960 0x7f45b406ae10 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f45d013ec90 tx=0x7f45c4007ec0 comp rx=0 tx=0).stop 2023-09-30T02:11:03.824 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.823+0000 7f45d7c22700 10 monclient: shutdown 2023-09-30T02:11:03.824 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.823+0000 7f45d7c22700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:11:03.824 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.823+0000 7f45d7c22700 1 -- 172.21.15.59:0/455399021 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f45d00a7650 msgr2=0x7f45d00aaab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:03.824 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.823+0000 7f45d7c22700 1 --2- 172.21.15.59:0/455399021 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f45d00a7650 0x7f45d00aaab0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f45cc00c7c0 tx=0x7f45cc007e00 comp rx=0 tx=0).stop 2023-09-30T02:11:03.825 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.823+0000 7f45d7c22700 1 -- 172.21.15.59:0/455399021 shutdown_connections 2023-09-30T02:11:03.825 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.823+0000 7f45d7c22700 1 --2- 172.21.15.59:0/455399021 >> v2:172.21.15.110:6800/115870 conn(0x7f45b4068960 0x7f45b406ae10 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:03.825 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.823+0000 7f45d7c22700 1 --2- 172.21.15.59:0/455399021 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f45d0146d30 0x7f45d01387c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:03.825 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.823+0000 7f45d7c22700 1 --2- 172.21.15.59:0/455399021 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f45d00a7650 0x7f45d00aaab0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:03.825 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.823+0000 7f45d7c22700 1 --2- 172.21.15.59:0/455399021 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45d00b16e0 0x7f45d0138280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:03.825 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.823+0000 7f45d7c22700 1 -- 172.21.15.59:0/455399021 >> 172.21.15.59:0/455399021 conn(0x7f45d0068fe0 msgr2=0x7f45d013b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:11:03.825 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.823+0000 7f45d7c22700 1 -- 172.21.15.59:0/455399021 shutdown_connections 2023-09-30T02:11:03.825 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.823+0000 7f45d7c22700 1 -- 172.21.15.59:0/455399021 wait complete. 2023-09-30T02:11:03.825 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:03.823+0000 7f45d7c22700 1 librados: shutdown 2023-09-30T02:11:06.839 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:11:06.975 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.974+0000 7f6033759700 10 monclient: get_monmap_and_config 2023-09-30T02:11:06.975 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.974+0000 7f6033759700 10 monclient: build_initial_monmap 2023-09-30T02:11:06.975 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.974+0000 7f6033759700 10 monclient: monmap: 2023-09-30T02:11:06.975 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:11:06.975 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:06.975 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:11:06.975687+0000 2023-09-30T02:11:06.976 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:11:06.975687+0000 2023-09-30T02:11:06.976 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:11:06.976 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:06.976 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:11:06.976 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:11:06.976 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:11:06.976 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:11:06.976 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:11:06.977 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:06.977 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.974+0000 7f6033759700 1 Processor -- start 2023-09-30T02:11:06.978 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.975+0000 7f6033759700 1 -- start start 2023-09-30T02:11:06.979 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.975+0000 7f6033759700 10 monclient: init 2023-09-30T02:11:06.979 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.975+0000 7f6033759700 10 monclient: _reopen_session rank -1 2023-09-30T02:11:06.979 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.975+0000 7f6033759700 10 monclient: _add_conns ranks=[4,3,1,2,0] 2023-09-30T02:11:06.979 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.975+0000 7f6033759700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f602c0a5b70 0x7f602c0b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:06.979 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.975+0000 7f6033759700 10 monclient(hunting): picked mon.noname-c con 0x7f602c0a5b70 addr v2:172.21.15.110:3300/0 2023-09-30T02:11:06.979 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.975+0000 7f6033759700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f602c0a5f40 0x7f602c141b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:06.980 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.975+0000 7f6033759700 10 monclient(hunting): picked mon.noname-a con 0x7f602c0a5f40 addr v2:172.21.15.59:3300/0 2023-09-30T02:11:06.980 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.975+0000 7f6033759700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f602c1420c0 addr v1:172.21.15.110:6789/0 2023-09-30T02:11:06.980 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.975+0000 7f6033759700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f602c0a75a0 con 0x7f602c1420c0 2023-09-30T02:11:06.980 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.975+0000 7f6033759700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:06.980 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.975+0000 7f6033759700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f602c0a63e0 con 0x7f602c0a5f40 2023-09-30T02:11:06.980 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.975+0000 7f6033759700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:06.981 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.975+0000 7f6033759700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f602c0af3f0 con 0x7f602c0a5b70 2023-09-30T02:11:06.981 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.975+0000 7f6033759700 10 monclient(hunting): _renew_subs 2023-09-30T02:11:06.981 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.975+0000 7f6033759700 10 monclient(hunting): authenticate will time out at 2112.278979s 2023-09-30T02:11:06.981 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.976+0000 7f6030cf4700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f602c0a5f40 0x7f602c141b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:06.981 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.976+0000 7f60314f5700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f602c0a5b70 0x7f602c0b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:06.981 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.977+0000 7f6030cf4700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f602c0a5f40 0x7f602c141b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44014/0 (socket says 172.21.15.59:44014) 2023-09-30T02:11:06.981 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.977+0000 7f6030cf4700 1 -- 172.21.15.59:0/167851154 learned_addr learned my addr 172.21.15.59:0/167851154 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:11:06.982 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.977+0000 7f6030cf4700 10 monclient(hunting): get_auth_request con 0x7f602c0a5f40 auth_method 0 2023-09-30T02:11:06.982 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.977+0000 7f6030cf4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:06.982 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.977+0000 7f6030cf4700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:06.982 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.977+0000 7f6030cf4700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:06.982 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.977+0000 7f60314f5700 10 monclient(hunting): get_auth_request con 0x7f602c0a5b70 auth_method 0 2023-09-30T02:11:06.982 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.977+0000 7f60314f5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:06.983 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.977+0000 7f60314f5700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:06.983 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.977+0000 7f60314f5700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:06.983 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.977+0000 7f6030cf4700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:06.983 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.977+0000 7f6030cf4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:06.983 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.977+0000 7f6030cf4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:06.984 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.980+0000 7f60314f5700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:06.984 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.980+0000 7f60314f5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:06.984 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.980+0000 7f60314f5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:06.985 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.980+0000 7f6030cf4700 10 monclient(hunting): handle_auth_done global_id 4782 payload 274 2023-09-30T02:11:06.985 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.980+0000 7f6030cf4700 1 -- 172.21.15.59:0/167851154 >> v2:172.21.15.110:3300/0 conn(0x7f602c0a5b70 msgr2=0x7f602c0b1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:11:06.985 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.980+0000 7f6030cf4700 1 --2- 172.21.15.59:0/167851154 >> v2:172.21.15.110:3300/0 conn(0x7f602c0a5b70 0x7f602c0b1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:06.985 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.980+0000 7f6030cf4700 1 -- 172.21.15.59:0/167851154 >> v1:172.21.15.110:6789/0 conn(0x7f602c1420c0 legacy=0x7f602c13e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:06.985 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.980+0000 7f6030cf4700 10 monclient: _finish_hunting 0 2023-09-30T02:11:06.985 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.980+0000 7f6030cf4700 1 monclient: found mon.noname-a 2023-09-30T02:11:06.985 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.980+0000 7f6030cf4700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:11:06.986 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.980+0000 7f6030cf4700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.59:3300/0 2023-09-30T02:11:06.986 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.980+0000 7f6030cf4700 1 -- 172.21.15.59:0/167851154 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f602c1424e0 con 0x7f602c0a5f40 2023-09-30T02:11:06.986 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.983+0000 7f6030cf4700 1 --2- 172.21.15.59:0/167851154 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f602c0a5f40 0x7f602c141b80 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f602c0b49e0 tx=0x7f601c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=447babf794c0a8e3 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:06.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.987+0000 7f6023fff700 1 -- 172.21.15.59:0/167851154 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f601c009e70 con 0x7f602c0a5f40 2023-09-30T02:11:06.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.987+0000 7f6023fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:06.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.987+0000 7f6023fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T02:11:06.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.987+0000 7f6023fff700 10 monclient: dump: 2023-09-30T02:11:06.988 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:06.989 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:06.989 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:06.989 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:06.989 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:06.989 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:06.989 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:06.989 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:06.990 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:06.990 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:06.990 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.987+0000 7f6023fff700 10 monclient: _finish_auth 0 2023-09-30T02:11:06.990 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.987+0000 7f6023fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:11:06.990 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.987+0000 7f6023fff700 1 -- 172.21.15.59:0/167851154 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f601c00b860 con 0x7f602c0a5f40 2023-09-30T02:11:06.990 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.987+0000 7f6033759700 5 monclient: authenticate success, global_id 4782 2023-09-30T02:11:06.991 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.987+0000 7f6033759700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:11:06.991 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.987+0000 7f6023fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:11:06.991 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.987+0000 7f6033759700 10 monclient: get_monmap_and_config success 2023-09-30T02:11:06.991 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.987+0000 7f6033759700 4 set_mon_vals no callback set 2023-09-30T02:11:06.991 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.987+0000 7f6023fff700 1 -- 172.21.15.59:0/167851154 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f601c00bc00 con 0x7f602c0a5f40 2023-09-30T02:11:06.991 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.987+0000 7f6023fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:06.992 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.987+0000 7f6023fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T02:11:06.992 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.987+0000 7f6023fff700 10 monclient: dump: 2023-09-30T02:11:06.992 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:06.992 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:06.992 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:06.992 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:06.992 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:06.993 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:06.993 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:06.993 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:06.993 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:06.993 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:06.997 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.988+0000 7f6033759700 10 monclient: shutdown 2023-09-30T02:11:06.997 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.988+0000 7f6033759700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:11:06.997 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.988+0000 7f6033759700 1 -- 172.21.15.59:0/167851154 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f602c0a5f40 msgr2=0x7f602c141b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:06.997 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.988+0000 7f6033759700 1 --2- 172.21.15.59:0/167851154 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f602c0a5f40 0x7f602c141b80 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f602c0b49e0 tx=0x7f601c009250 comp rx=0 tx=0).stop 2023-09-30T02:11:06.997 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.988+0000 7f6033759700 1 -- 172.21.15.59:0/167851154 shutdown_connections 2023-09-30T02:11:06.998 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.988+0000 7f6033759700 1 --2- 172.21.15.59:0/167851154 >> v2:172.21.15.110:3300/0 conn(0x7f602c0a5b70 0x7f602c0b1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:06.998 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.988+0000 7f6033759700 1 --2- 172.21.15.59:0/167851154 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f602c0a5f40 0x7f602c141b80 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:06.998 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.988+0000 7f6033759700 1 -- 172.21.15.59:0/167851154 >> 172.21.15.59:0/167851154 conn(0x7f602c0631f0 msgr2=0x7f602c0635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:11:06.998 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.988+0000 7f6033759700 1 -- 172.21.15.59:0/167851154 shutdown_connections 2023-09-30T02:11:06.998 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.988+0000 7f6033759700 1 -- 172.21.15.59:0/167851154 wait complete. 2023-09-30T02:11:06.998 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.989+0000 7f6033759700 10 monclient: build_initial_monmap 2023-09-30T02:11:06.999 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.989+0000 7f6033759700 10 monclient: monmap: 2023-09-30T02:11:06.999 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:11:06.999 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:11:06.999 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:11:06.999 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:11:06.999 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:11:06.999 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:11:07.000 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:11:07.000 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:11:07.000 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:11:07.000 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:07.000 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.990+0000 7f6033759700 1 librados: starting msgr at 2023-09-30T02:11:07.000 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.990+0000 7f6033759700 1 librados: starting objecter 2023-09-30T02:11:07.001 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.990+0000 7f6033759700 1 Processor -- start 2023-09-30T02:11:07.001 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.990+0000 7f6033759700 1 -- start start 2023-09-30T02:11:07.001 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.990+0000 7f6033759700 1 librados: setting wanted keys 2023-09-30T02:11:07.001 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.990+0000 7f6033759700 1 librados: calling monclient init 2023-09-30T02:11:07.001 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.990+0000 7f6033759700 10 monclient: init 2023-09-30T02:11:07.001 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.991+0000 7f6033759700 10 monclient: _reopen_session rank -1 2023-09-30T02:11:07.001 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.991+0000 7f6033759700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T02:11:07.001 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.991+0000 7f6033759700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f602c0a5b70 0x7f602c13cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:07.002 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.991+0000 7f6033759700 10 monclient(hunting): picked mon.noname-c con 0x7f602c0a5b70 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:11:07.002 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.991+0000 7f6033759700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f602c0a5f40 0x7f602c136450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:07.002 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.991+0000 7f6033759700 10 monclient(hunting): picked mon.noname-a con 0x7f602c0a5f40 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:11:07.002 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.991+0000 7f6033759700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f602c1420c0 0x7f602c13fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:07.002 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.991+0000 7f6033759700 10 monclient(hunting): picked mon.noname-b con 0x7f602c1420c0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:11:07.002 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.991+0000 7f6033759700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:07.003 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.991+0000 7f6033759700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f602c13caf0 con 0x7f602c0a5f40 2023-09-30T02:11:07.003 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.991+0000 7f6033759700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:07.003 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.991+0000 7f6033759700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f602c13e0e0 con 0x7f602c1420c0 2023-09-30T02:11:07.003 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.991+0000 7f6033759700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:07.003 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.991+0000 7f6033759700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f602c13aac0 con 0x7f602c0a5b70 2023-09-30T02:11:07.003 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.991+0000 7f6033759700 10 monclient(hunting): _renew_subs 2023-09-30T02:11:07.004 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.991+0000 7f6033759700 10 monclient(hunting): authenticate will time out at 2112.294167s 2023-09-30T02:11:07.004 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.991+0000 7f6031cf6700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f602c1420c0 0x7f602c13fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:07.004 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.992+0000 7f60314f5700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f602c0a5b70 0x7f602c13cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:07.004 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.992+0000 7f6030cf4700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f602c0a5f40 0x7f602c136450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:07.004 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.992+0000 7f6031cf6700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f602c1420c0 0x7f602c13fa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:46026/0 (socket says 172.21.15.59:46026) 2023-09-30T02:11:07.004 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.992+0000 7f6031cf6700 1 -- 172.21.15.59:0/1475429703 learned_addr learned my addr 172.21.15.59:0/1475429703 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:11:07.004 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.992+0000 7f6031cf6700 10 monclient(hunting): get_auth_request con 0x7f602c1420c0 auth_method 0 2023-09-30T02:11:07.005 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.992+0000 7f6031cf6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:07.005 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.992+0000 7f6031cf6700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:07.005 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.992+0000 7f6031cf6700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:07.005 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.992+0000 7f60314f5700 10 monclient(hunting): get_auth_request con 0x7f602c0a5b70 auth_method 0 2023-09-30T02:11:07.005 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.992+0000 7f60314f5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:07.005 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.992+0000 7f60314f5700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:07.006 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.992+0000 7f60314f5700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:07.006 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.992+0000 7f6030cf4700 10 monclient(hunting): get_auth_request con 0x7f602c0a5f40 auth_method 0 2023-09-30T02:11:07.006 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.992+0000 7f6030cf4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:07.006 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.992+0000 7f6030cf4700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:07.006 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.992+0000 7f6030cf4700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:07.006 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.992+0000 7f6031cf6700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:07.006 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.992+0000 7f6031cf6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:07.007 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.992+0000 7f6031cf6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:07.007 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.993+0000 7f60314f5700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:07.007 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.993+0000 7f60314f5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:07.007 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.993+0000 7f60314f5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:07.007 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.993+0000 7f6030cf4700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:07.007 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.993+0000 7f6030cf4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:07.007 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.993+0000 7f6030cf4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:07.007 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.995+0000 7f6031cf6700 10 monclient(hunting): handle_auth_done global_id 4816 payload 824 2023-09-30T02:11:07.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.995+0000 7f6031cf6700 1 -- 172.21.15.59:0/1475429703 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f602c0a5b70 msgr2=0x7f602c13cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:07.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.996+0000 7f6031cf6700 1 --2- 172.21.15.59:0/1475429703 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f602c0a5b70 0x7f602c13cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:07.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.996+0000 7f6031cf6700 1 -- 172.21.15.59:0/1475429703 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f602c0a5f40 msgr2=0x7f602c136450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:07.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.996+0000 7f6031cf6700 1 --2- 172.21.15.59:0/1475429703 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f602c0a5f40 0x7f602c136450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:07.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.996+0000 7f6031cf6700 10 monclient: _finish_hunting 0 2023-09-30T02:11:07.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.996+0000 7f6031cf6700 1 monclient: found mon.noname-b 2023-09-30T02:11:07.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.996+0000 7f6031cf6700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:11:07.009 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.996+0000 7f6031cf6700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:11:07.009 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.996+0000 7f6031cf6700 1 -- 172.21.15.59:0/1475429703 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f601c0094f0 con 0x7f602c1420c0 2023-09-30T02:11:07.011 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.998+0000 7f6031cf6700 1 --2- 172.21.15.59:0/1475429703 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f602c1420c0 0x7f602c13fa70 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f602400c840 tx=0x7f6024001670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:07.014 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.998+0000 7f60217fa700 10 client.?.objecter ms_handle_connect 0x7f602c1420c0 2023-09-30T02:11:07.014 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:06.998+0000 7f60217fa700 10 client.?.objecter resend_mon_ops 2023-09-30T02:11:07.014 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:07.002+0000 7f60217fa700 1 -- 172.21.15.59:0/1475429703 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6024012630 con 0x7f602c1420c0 2023-09-30T02:11:07.014 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:07.002+0000 7f60217fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:07.015 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:07.002+0000 7f60217fa700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:11:07.015 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:07.002+0000 7f60217fa700 10 monclient: dump: 2023-09-30T02:11:07.015 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:07.015 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:07.015 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:07.015 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:07.015 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:07.015 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:07.016 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:07.016 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:07.016 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:07.016 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:07.016 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:07.002+0000 7f60217fa700 10 monclient: _finish_auth 0 2023-09-30T02:11:07.016 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:07.003+0000 7f60217fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:11:07.016 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:07.003+0000 7f60217fa700 1 -- 172.21.15.59:0/1475429703 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6024012790 con 0x7f602c1420c0 2023-09-30T02:11:07.016 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:07.003+0000 7f60217fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:11:07.017 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:07.003+0000 7f60217fa700 1 -- 172.21.15.59:0/1475429703 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6024012b50 con 0x7f602c1420c0 2023-09-30T02:11:07.017 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:07.003+0000 7f60217fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:07.017 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:07.003+0000 7f60217fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:11:07.017 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:07.003+0000 7f60217fa700 10 monclient: dump: 2023-09-30T02:11:07.017 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:07.017 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:07.017 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:07.017 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:07.018 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:07.018 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:07.018 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:07.018 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:07.018 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:07.018 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:07.018 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:07.003+0000 7f6033759700 5 monclient: authenticate success, global_id 4816 2023-09-30T02:11:07.019 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:07.003+0000 7f6033759700 10 monclient: _renew_subs 2023-09-30T02:11:07.019 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:07.003+0000 7f6033759700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:11:07.019 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:07.003+0000 7f6033759700 1 -- 172.21.15.59:0/1475429703 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f602c13ffb0 con 0x7f602c1420c0 2023-09-30T02:11:07.019 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:07.003+0000 7f60237fe700 4 set_mon_vals no callback set 2023-09-30T02:11:07.019 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:07.003+0000 7f6033759700 10 client.4816.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:11:07.019 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:07.003+0000 7f6033759700 10 monclient: _renew_subs 2023-09-30T02:11:07.019 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:07.003+0000 7f6033759700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:11:07.020 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:07.003+0000 7f6033759700 1 -- 172.21.15.59:0/1475429703 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f602c13c500 con 0x7f602c1420c0 2023-09-30T02:11:07.020 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:07.003+0000 7f6033759700 1 librados: init done 2023-09-30T02:11:07.020 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:07.004+0000 7f6033759700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:11:07.020 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:07.004+0000 7f6033759700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:11:07.020 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:07.004+0000 7f6033759700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:11:07.020 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:07.004+0000 7f6033759700 1 -- 172.21.15.59:0/1475429703 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f602c13cc60 con 0x7f602c1420c0 2023-09-30T02:11:07.217 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:07.216+0000 7f60217fa700 1 -- 172.21.15.59:0/1475429703 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f6024003030 con 0x7f602c1420c0 2023-09-30T02:11:07.217 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:07.217+0000 7f60217fa700 1 --2- 172.21.15.59:0/1475429703 >> v2:172.21.15.110:6800/115870 conn(0x7f6010068960 0x7f601006ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:07.218 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:07.217+0000 7f60314f5700 1 --2- 172.21.15.59:0/1475429703 >> v2:172.21.15.110:6800/115870 conn(0x7f6010068960 0x7f601006ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:07.218 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:07.217+0000 7f60314f5700 10 monclient: get_auth_request con 0x7f6010068960 auth_method 0 2023-09-30T02:11:07.218 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:07.218+0000 7f60314f5700 1 --2- 172.21.15.59:0/1475429703 >> v2:172.21.15.110:6800/115870 conn(0x7f6010068960 0x7f601006ae10 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f6028009db0 tx=0x7f60280091a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:07.218 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:07.218+0000 7f60217fa700 10 client.4816.objecter ms_handle_connect 0x7f6010068960 2023-09-30T02:11:07.222 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:07.223+0000 7f60217fa700 1 -- 172.21.15.59:0/1475429703 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f60240560d0 con 0x7f602c1420c0 2023-09-30T02:11:07.223 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:07.223+0000 7f60217fa700 10 client.4816.objecter ms_dispatch 0x7f602c001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:11:07.223 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:07.223+0000 7f60217fa700 3 client.4816.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:11:07.223 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:07.223+0000 7f60217fa700 3 client.4816.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:11:07.223 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:07.223+0000 7f60217fa700 20 client.4816.objecter dump_active .. 0 homeless 2023-09-30T02:11:07.268 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:07.268+0000 7f60217fa700 1 -- 172.21.15.59:0/1475429703 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6024051d10 con 0x7f602c1420c0 2023-09-30T02:11:07.268 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:07.268+0000 7f60217fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:11:07.269 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:07.268+0000 7f60217fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:11:07.518 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:07.518+0000 7f6033759700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:11:07.518 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:07.518+0000 7f6033759700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:11:07.518 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:07.518+0000 7f6033759700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:11:07.518 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:07.518+0000 7f6033759700 1 -- 172.21.15.59:0/1475429703 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f602c064a00 con 0x7f602c1420c0 2023-09-30T02:11:07.523 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:07.523+0000 7f60217fa700 1 -- 172.21.15.59:0/1475429703 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f6024055330 con 0x7f602c1420c0 2023-09-30T02:11:07.523 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:07.523+0000 7f60217fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:11:07.523 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:07.523+0000 7f60217fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:11:07.523 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:11:07.523 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:11:07.526 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:07.524+0000 7f6033759700 10 librados: watch_flush enter 2023-09-30T02:11:07.526 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:07.525+0000 7f6033759700 10 librados: watch_flush exit 2023-09-30T02:11:07.526 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:07.525+0000 7f6033759700 20 client.4816.objecter shutdown clearing up homeless session... 2023-09-30T02:11:07.526 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:07.525+0000 7f6033759700 10 client.4816.objecter successfully canceled tick 2023-09-30T02:11:07.526 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:07.525+0000 7f6033759700 1 -- 172.21.15.59:0/1475429703 >> v2:172.21.15.110:6800/115870 conn(0x7f6010068960 msgr2=0x7f601006ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:07.526 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:07.525+0000 7f6033759700 1 --2- 172.21.15.59:0/1475429703 >> v2:172.21.15.110:6800/115870 conn(0x7f6010068960 0x7f601006ae10 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f6028009db0 tx=0x7f60280091a0 comp rx=0 tx=0).stop 2023-09-30T02:11:07.526 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:07.525+0000 7f6033759700 10 monclient: shutdown 2023-09-30T02:11:07.526 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:07.525+0000 7f6033759700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:11:07.526 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:07.525+0000 7f6033759700 1 -- 172.21.15.59:0/1475429703 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f602c1420c0 msgr2=0x7f602c13fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:07.526 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:07.525+0000 7f6033759700 1 --2- 172.21.15.59:0/1475429703 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f602c1420c0 0x7f602c13fa70 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f602400c840 tx=0x7f6024001670 comp rx=0 tx=0).stop 2023-09-30T02:11:07.527 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:07.525+0000 7f6033759700 1 -- 172.21.15.59:0/1475429703 shutdown_connections 2023-09-30T02:11:07.527 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:07.525+0000 7f6033759700 1 --2- 172.21.15.59:0/1475429703 >> v2:172.21.15.110:6800/115870 conn(0x7f6010068960 0x7f601006ae10 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:07.527 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:07.525+0000 7f6033759700 1 --2- 172.21.15.59:0/1475429703 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f602c1420c0 0x7f602c13fa70 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:07.527 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:07.525+0000 7f6033759700 1 --2- 172.21.15.59:0/1475429703 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f602c0a5b70 0x7f602c13cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:07.527 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:07.525+0000 7f6033759700 1 --2- 172.21.15.59:0/1475429703 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f602c0a5f40 0x7f602c136450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:07.527 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:07.525+0000 7f6033759700 1 -- 172.21.15.59:0/1475429703 >> 172.21.15.59:0/1475429703 conn(0x7f602c0631f0 msgr2=0x7f602c0af560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:11:07.527 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:07.525+0000 7f6033759700 1 -- 172.21.15.59:0/1475429703 shutdown_connections 2023-09-30T02:11:07.527 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:07.525+0000 7f6033759700 1 -- 172.21.15.59:0/1475429703 wait complete. 2023-09-30T02:11:07.527 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:07.525+0000 7f6033759700 1 librados: shutdown 2023-09-30T02:11:10.540 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:11:10.675 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.674+0000 7f58b51b6700 10 monclient: get_monmap_and_config 2023-09-30T02:11:10.675 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.674+0000 7f58b51b6700 10 monclient: build_initial_monmap 2023-09-30T02:11:10.675 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.674+0000 7f58b51b6700 10 monclient: monmap: 2023-09-30T02:11:10.675 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:11:10.675 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:10.675 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:11:10.675868+0000 2023-09-30T02:11:10.675 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:11:10.675868+0000 2023-09-30T02:11:10.675 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:11:10.676 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:10.676 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:11:10.676 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:11:10.676 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:11:10.676 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:11:10.676 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:11:10.676 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:10.676 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.675+0000 7f58b51b6700 1 Processor -- start 2023-09-30T02:11:10.676 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.675+0000 7f58b51b6700 1 -- start start 2023-09-30T02:11:10.676 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.675+0000 7f58b51b6700 10 monclient: init 2023-09-30T02:11:10.678 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.675+0000 7f58b51b6700 10 monclient: _reopen_session rank -1 2023-09-30T02:11:10.678 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.675+0000 7f58b51b6700 10 monclient: _add_conns ranks=[2,4,3,1,0] 2023-09-30T02:11:10.679 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.675+0000 7f58b51b6700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f58b00a5b70 0x7f58b00b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:10.679 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.675+0000 7f58b51b6700 10 monclient(hunting): picked mon.noname-b con 0x7f58b00a5b70 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:11:10.679 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.675+0000 7f58b51b6700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f58b00a5f40 0x7f58b0141b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:10.679 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.675+0000 7f58b51b6700 10 monclient(hunting): picked mon.noname-c con 0x7f58b00a5f40 addr v2:172.21.15.110:3300/0 2023-09-30T02:11:10.679 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.675+0000 7f58b51b6700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f58b01420c0 0x7f58b013e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:10.679 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.675+0000 7f58b51b6700 10 monclient(hunting): picked mon.noname-a con 0x7f58b01420c0 addr v2:172.21.15.59:3300/0 2023-09-30T02:11:10.679 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.675+0000 7f58b51b6700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:10.679 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.675+0000 7f58b51b6700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58b00a63b0 con 0x7f58b01420c0 2023-09-30T02:11:10.679 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.675+0000 7f58b51b6700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:10.679 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.675+0000 7f58b51b6700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58b00a75a0 con 0x7f58b00a5f40 2023-09-30T02:11:10.679 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.675+0000 7f58b51b6700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:10.679 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.675+0000 7f58b51b6700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58b00af3f0 con 0x7f58b00a5b70 2023-09-30T02:11:10.680 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.675+0000 7f58b51b6700 10 monclient(hunting): _renew_subs 2023-09-30T02:11:10.680 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.675+0000 7f58b51b6700 10 monclient(hunting): authenticate will time out at 2115.978968s 2023-09-30T02:11:10.680 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.676+0000 7f58aed9d700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f58b00a5b70 0x7f58b00b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:10.680 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.676+0000 7f58ae59c700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f58b00a5f40 0x7f58b0141b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:10.680 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.676+0000 7f58af59e700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f58b01420c0 0x7f58b013e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:10.680 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.677+0000 7f58aed9d700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f58b00a5b70 0x7f58b00b1500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:46946/0 (socket says 172.21.15.59:46946) 2023-09-30T02:11:10.680 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.677+0000 7f58aed9d700 1 -- 172.21.15.59:0/2132155628 learned_addr learned my addr 172.21.15.59:0/2132155628 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:11:10.680 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.677+0000 7f58aed9d700 10 monclient(hunting): get_auth_request con 0x7f58b00a5b70 auth_method 0 2023-09-30T02:11:10.680 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.677+0000 7f58aed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:10.680 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.677+0000 7f58aed9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:10.680 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.677+0000 7f58aed9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:10.681 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.677+0000 7f58ae59c700 10 monclient(hunting): get_auth_request con 0x7f58b00a5f40 auth_method 0 2023-09-30T02:11:10.681 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.677+0000 7f58ae59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:10.681 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.677+0000 7f58ae59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:10.681 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.677+0000 7f58ae59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:10.681 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.677+0000 7f58af59e700 10 monclient(hunting): get_auth_request con 0x7f58b01420c0 auth_method 0 2023-09-30T02:11:10.681 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.677+0000 7f58af59e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:10.681 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.677+0000 7f58af59e700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:10.681 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.677+0000 7f58af59e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:10.681 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.677+0000 7f58ae59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:10.681 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.677+0000 7f58ae59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:10.681 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.677+0000 7f58ae59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:10.681 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.677+0000 7f58aed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:10.682 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.677+0000 7f58aed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:10.682 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.677+0000 7f58aed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:10.682 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.678+0000 7f58af59e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:10.682 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.678+0000 7f58af59e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:10.682 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.678+0000 7f58af59e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:10.683 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.679+0000 7f58ae59c700 10 monclient(hunting): handle_auth_done global_id 4819 payload 274 2023-09-30T02:11:10.683 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.679+0000 7f58ae59c700 1 -- 172.21.15.59:0/2132155628 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f58b00a5b70 msgr2=0x7f58b00b1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:11:10.683 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.679+0000 7f58ae59c700 1 --2- 172.21.15.59:0/2132155628 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f58b00a5b70 0x7f58b00b1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:10.683 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.679+0000 7f58ae59c700 1 -- 172.21.15.59:0/2132155628 >> v2:172.21.15.59:3300/0 conn(0x7f58b01420c0 msgr2=0x7f58b013e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:11:10.683 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.679+0000 7f58ae59c700 1 --2- 172.21.15.59:0/2132155628 >> v2:172.21.15.59:3300/0 conn(0x7f58b01420c0 0x7f58b013e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:10.683 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.679+0000 7f58ae59c700 10 monclient: _finish_hunting 0 2023-09-30T02:11:10.683 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.679+0000 7f58ae59c700 1 monclient: found mon.noname-c 2023-09-30T02:11:10.683 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.679+0000 7f58ae59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:11:10.683 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.679+0000 7f58ae59c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3300/0 2023-09-30T02:11:10.683 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.679+0000 7f58ae59c700 1 -- 172.21.15.59:0/2132155628 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58b0142500 con 0x7f58b00a5f40 2023-09-30T02:11:10.684 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.681+0000 7f58ae59c700 1 --2- 172.21.15.59:0/2132155628 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f58b00a5f40 0x7f58b0141b80 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f58b0069af0 tx=0x7f58a40091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=32360f4e2d760d59 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:10.686 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.685+0000 7f58add9b700 1 -- 172.21.15.59:0/2132155628 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a4011630 con 0x7f58b00a5f40 2023-09-30T02:11:10.686 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.685+0000 7f58add9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:10.686 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.685+0000 7f58add9b700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:11:10.686 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.685+0000 7f58add9b700 10 monclient: dump: 2023-09-30T02:11:10.686 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:10.686 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:10.686 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:10.687 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:10.687 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:10.687 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:10.687 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:10.687 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:10.687 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:10.687 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:10.687 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.685+0000 7f58add9b700 10 monclient: _finish_auth 0 2023-09-30T02:11:10.687 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.685+0000 7f58add9b700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:11:10.687 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.685+0000 7f58add9b700 1 -- 172.21.15.59:0/2132155628 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58a40117b0 con 0x7f58b00a5f40 2023-09-30T02:11:10.688 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.685+0000 7f58add9b700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:11:10.688 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.685+0000 7f58add9b700 1 -- 172.21.15.59:0/2132155628 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a4011b70 con 0x7f58b00a5f40 2023-09-30T02:11:10.688 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.685+0000 7f58b51b6700 5 monclient: authenticate success, global_id 4819 2023-09-30T02:11:10.688 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.685+0000 7f58b51b6700 10 monclient: get_monmap_and_config success 2023-09-30T02:11:10.688 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.685+0000 7f58b51b6700 4 set_mon_vals no callback set 2023-09-30T02:11:10.688 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.685+0000 7f58add9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:10.688 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.685+0000 7f58add9b700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:11:10.688 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.685+0000 7f58add9b700 10 monclient: dump: 2023-09-30T02:11:10.688 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:10.688 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:10.688 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:10.689 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:10.689 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:10.689 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:10.689 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:10.689 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:10.689 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:10.689 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:10.692 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.686+0000 7f58b51b6700 10 monclient: shutdown 2023-09-30T02:11:10.692 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.686+0000 7f58b51b6700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:11:10.692 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.686+0000 7f58b51b6700 1 -- 172.21.15.59:0/2132155628 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f58b00a5f40 msgr2=0x7f58b0141b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:10.692 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.686+0000 7f58b51b6700 1 --2- 172.21.15.59:0/2132155628 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f58b00a5f40 0x7f58b0141b80 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f58b0069af0 tx=0x7f58a40091a0 comp rx=0 tx=0).stop 2023-09-30T02:11:10.692 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.686+0000 7f58b51b6700 1 -- 172.21.15.59:0/2132155628 shutdown_connections 2023-09-30T02:11:10.692 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.686+0000 7f58b51b6700 1 --2- 172.21.15.59:0/2132155628 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f58b00a5b70 0x7f58b00b1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:10.693 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.686+0000 7f58b51b6700 1 --2- 172.21.15.59:0/2132155628 >> v2:172.21.15.59:3300/0 conn(0x7f58b01420c0 0x7f58b013e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:10.693 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.686+0000 7f58b51b6700 1 --2- 172.21.15.59:0/2132155628 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f58b00a5f40 0x7f58b0141b80 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:10.693 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.686+0000 7f58b51b6700 1 -- 172.21.15.59:0/2132155628 >> 172.21.15.59:0/2132155628 conn(0x7f58b00631f0 msgr2=0x7f58b00635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:11:10.693 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.686+0000 7f58b51b6700 1 -- 172.21.15.59:0/2132155628 shutdown_connections 2023-09-30T02:11:10.693 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.686+0000 7f58b51b6700 1 -- 172.21.15.59:0/2132155628 wait complete. 2023-09-30T02:11:10.693 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.687+0000 7f58b51b6700 10 monclient: build_initial_monmap 2023-09-30T02:11:10.693 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.687+0000 7f58b51b6700 10 monclient: monmap: 2023-09-30T02:11:10.693 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:11:10.693 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:11:10.693 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:11:10.693 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:11:10.694 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:11:10.694 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:11:10.694 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:11:10.694 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:11:10.694 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:11:10.694 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:10.694 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.687+0000 7f58b51b6700 1 librados: starting msgr at 2023-09-30T02:11:10.694 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.687+0000 7f58b51b6700 1 librados: starting objecter 2023-09-30T02:11:10.694 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.688+0000 7f58b51b6700 1 Processor -- start 2023-09-30T02:11:10.695 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.688+0000 7f58b51b6700 1 -- start start 2023-09-30T02:11:10.695 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.688+0000 7f58b51b6700 1 librados: setting wanted keys 2023-09-30T02:11:10.695 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.688+0000 7f58b51b6700 1 librados: calling monclient init 2023-09-30T02:11:10.695 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.688+0000 7f58b51b6700 10 monclient: init 2023-09-30T02:11:10.695 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.688+0000 7f58b51b6700 10 monclient: _reopen_session rank -1 2023-09-30T02:11:10.695 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.688+0000 7f58b51b6700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T02:11:10.695 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.688+0000 7f58b51b6700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f58b00a5b70 0x7f58b013d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:10.695 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.688+0000 7f58b51b6700 10 monclient(hunting): picked mon.noname-b con 0x7f58b00a5b70 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:11:10.695 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.688+0000 7f58b51b6700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f58b00a5f40 0x7f58b00a8fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:10.695 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.688+0000 7f58b51b6700 10 monclient(hunting): picked mon.noname-c con 0x7f58b00a5f40 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:11:10.695 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.688+0000 7f58b51b6700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58b01420c0 0x7f58b00a9d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:10.695 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.688+0000 7f58b51b6700 10 monclient(hunting): picked mon.noname-a con 0x7f58b01420c0 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:11:10.695 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.688+0000 7f58b51b6700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:10.696 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.688+0000 7f58b51b6700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58b00ab3c0 con 0x7f58b01420c0 2023-09-30T02:11:10.696 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.688+0000 7f58b51b6700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:10.696 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.688+0000 7f58b51b6700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58b00a9510 con 0x7f58b00a5b70 2023-09-30T02:11:10.696 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.688+0000 7f58b51b6700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:10.697 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.688+0000 7f58b51b6700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58b013e350 con 0x7f58b00a5f40 2023-09-30T02:11:10.697 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.688+0000 7f58b51b6700 10 monclient(hunting): _renew_subs 2023-09-30T02:11:10.697 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.688+0000 7f58b51b6700 10 monclient(hunting): authenticate will time out at 2115.991697s 2023-09-30T02:11:10.697 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.689+0000 7f58aed9d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f58b00a5b70 0x7f58b013d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:10.697 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.689+0000 7f58ae59c700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f58b00a5f40 0x7f58b00a8fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:10.697 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.689+0000 7f58af59e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58b01420c0 0x7f58b00a9d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:10.697 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.689+0000 7f58ae59c700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f58b00a5f40 0x7f58b00a8fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:46948/0 (socket says 172.21.15.59:46948) 2023-09-30T02:11:10.697 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.689+0000 7f58ae59c700 1 -- 172.21.15.59:0/3766005066 learned_addr learned my addr 172.21.15.59:0/3766005066 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:11:10.697 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.689+0000 7f58aed9d700 10 monclient(hunting): get_auth_request con 0x7f58b00a5b70 auth_method 0 2023-09-30T02:11:10.697 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.689+0000 7f58aed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:10.698 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.689+0000 7f58aed9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:10.698 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.689+0000 7f58aed9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:10.698 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.689+0000 7f58ae59c700 10 monclient(hunting): get_auth_request con 0x7f58b00a5f40 auth_method 0 2023-09-30T02:11:10.698 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.689+0000 7f58ae59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:10.698 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.689+0000 7f58ae59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:10.698 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.689+0000 7f58ae59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:10.698 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.689+0000 7f58af59e700 10 monclient(hunting): get_auth_request con 0x7f58b01420c0 auth_method 0 2023-09-30T02:11:10.698 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.689+0000 7f58af59e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:10.698 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.689+0000 7f58af59e700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:10.698 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.689+0000 7f58af59e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:10.698 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.690+0000 7f58aed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:10.699 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.690+0000 7f58aed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:10.699 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.690+0000 7f58aed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:10.699 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.690+0000 7f58ae59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:10.699 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.690+0000 7f58ae59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:10.699 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.690+0000 7f58ae59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:10.699 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.690+0000 7f58af59e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:10.699 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.690+0000 7f58af59e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:10.699 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.690+0000 7f58af59e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:10.701 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.693+0000 7f58aed9d700 10 monclient(hunting): handle_auth_done global_id 4822 payload 824 2023-09-30T02:11:10.701 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.693+0000 7f58aed9d700 1 -- 172.21.15.59:0/3766005066 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f58b00a5f40 msgr2=0x7f58b00a8fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:10.701 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.693+0000 7f58aed9d700 1 --2- 172.21.15.59:0/3766005066 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f58b00a5f40 0x7f58b00a8fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:10.702 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.693+0000 7f58aed9d700 1 -- 172.21.15.59:0/3766005066 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58b01420c0 msgr2=0x7f58b00a9d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:10.702 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.693+0000 7f58aed9d700 1 --2- 172.21.15.59:0/3766005066 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58b01420c0 0x7f58b00a9d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:10.702 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.693+0000 7f58aed9d700 10 monclient: _finish_hunting 0 2023-09-30T02:11:10.702 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.693+0000 7f58aed9d700 1 monclient: found mon.noname-b 2023-09-30T02:11:10.702 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.693+0000 7f58aed9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:11:10.702 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.693+0000 7f58aed9d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:11:10.702 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.693+0000 7f58aed9d700 1 -- 172.21.15.59:0/3766005066 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58a4009420 con 0x7f58b00a5b70 2023-09-30T02:11:10.702 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.695+0000 7f58aed9d700 1 --2- 172.21.15.59:0/3766005066 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f58b00a5b70 0x7f58b013d790 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f58a0006d40 tx=0x7f58a0009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:10.702 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.695+0000 7f5896ffd700 10 client.?.objecter ms_handle_connect 0x7f58b00a5b70 2023-09-30T02:11:10.702 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.695+0000 7f5896ffd700 10 client.?.objecter resend_mon_ops 2023-09-30T02:11:10.703 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.699+0000 7f5896ffd700 1 -- 172.21.15.59:0/3766005066 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a0013680 con 0x7f58b00a5b70 2023-09-30T02:11:10.703 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.699+0000 7f5896ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:10.703 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.699+0000 7f5896ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:11:10.703 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.699+0000 7f5896ffd700 10 monclient: dump: 2023-09-30T02:11:10.703 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:10.703 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:10.703 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:10.703 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:10.703 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:10.703 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:10.703 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:10.704 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:10.704 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:10.704 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:10.704 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.699+0000 7f5896ffd700 10 monclient: _finish_auth 0 2023-09-30T02:11:10.704 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.699+0000 7f5896ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:11:10.704 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.699+0000 7f58b51b6700 5 monclient: authenticate success, global_id 4822 2023-09-30T02:11:10.704 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.699+0000 7f5896ffd700 1 -- 172.21.15.59:0/3766005066 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58a00137e0 con 0x7f58b00a5b70 2023-09-30T02:11:10.704 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.699+0000 7f5896ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:11:10.704 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.699+0000 7f58b51b6700 10 monclient: _renew_subs 2023-09-30T02:11:10.704 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.699+0000 7f5896ffd700 1 -- 172.21.15.59:0/3766005066 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a0013b60 con 0x7f58b00a5b70 2023-09-30T02:11:10.704 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.699+0000 7f58b51b6700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:11:10.704 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.699+0000 7f58b51b6700 1 -- 172.21.15.59:0/3766005066 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58b00aa240 con 0x7f58b00a5b70 2023-09-30T02:11:10.705 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.699+0000 7f5896ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:10.705 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.699+0000 7f5896ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:11:10.705 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.699+0000 7f5896ffd700 10 monclient: dump: 2023-09-30T02:11:10.705 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:10.705 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:10.705 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:10.705 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:10.705 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:10.705 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:10.705 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:10.705 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:10.705 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:10.706 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:10.706 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.699+0000 7f58ad59a700 4 set_mon_vals no callback set 2023-09-30T02:11:10.706 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.699+0000 7f58b51b6700 10 client.4822.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:11:10.706 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.699+0000 7f58b51b6700 10 monclient: _renew_subs 2023-09-30T02:11:10.706 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.699+0000 7f58b51b6700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:11:10.706 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.699+0000 7f58b51b6700 1 -- 172.21.15.59:0/3766005066 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58b00aa540 con 0x7f58b00a5b70 2023-09-30T02:11:10.706 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.699+0000 7f58b51b6700 1 librados: init done 2023-09-30T02:11:10.706 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.700+0000 7f58b51b6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:11:10.706 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.700+0000 7f58b51b6700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:11:10.706 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.700+0000 7f58b51b6700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:11:10.706 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.700+0000 7f58b51b6700 1 -- 172.21.15.59:0/3766005066 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58b0151390 con 0x7f58b00a5b70 2023-09-30T02:11:10.949 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.949+0000 7f5896ffd700 1 -- 172.21.15.59:0/3766005066 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f58a000c3e0 con 0x7f58b00a5b70 2023-09-30T02:11:10.950 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.949+0000 7f5896ffd700 1 --2- 172.21.15.59:0/3766005066 >> v2:172.21.15.110:6800/115870 conn(0x7f589c0688c0 0x7f589c06ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:10.950 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.949+0000 7f58ae59c700 1 --2- 172.21.15.59:0/3766005066 >> v2:172.21.15.110:6800/115870 conn(0x7f589c0688c0 0x7f589c06ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:10.950 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.949+0000 7f58ae59c700 10 monclient: get_auth_request con 0x7f589c0688c0 auth_method 0 2023-09-30T02:11:10.950 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.950+0000 7f58ae59c700 1 --2- 172.21.15.59:0/3766005066 >> v2:172.21.15.110:6800/115870 conn(0x7f589c0688c0 0x7f589c06ad70 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f58b013df50 tx=0x7f58a4017000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:10.950 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.950+0000 7f5896ffd700 10 client.4822.objecter ms_handle_connect 0x7f589c0688c0 2023-09-30T02:11:10.954 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.953+0000 7f5896ffd700 1 -- 172.21.15.59:0/3766005066 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f58a00563b0 con 0x7f58b00a5b70 2023-09-30T02:11:10.954 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.953+0000 7f5896ffd700 10 client.4822.objecter ms_dispatch 0x7f58b0001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:11:10.954 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.953+0000 7f5896ffd700 3 client.4822.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:11:10.954 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.953+0000 7f5896ffd700 3 client.4822.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:11:10.954 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.954+0000 7f5896ffd700 20 client.4822.objecter dump_active .. 0 homeless 2023-09-30T02:11:10.963 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.963+0000 7f5896ffd700 1 -- 172.21.15.59:0/3766005066 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f58a0051ff0 con 0x7f58b00a5b70 2023-09-30T02:11:10.963 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.963+0000 7f5896ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:11:10.963 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:10.963+0000 7f5896ffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:11:11.234 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:11.233+0000 7f58b51b6700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:11:11.234 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:11.233+0000 7f58b51b6700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:11:11.234 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:11.233+0000 7f58b51b6700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:11:11.234 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:11.233+0000 7f58b51b6700 1 -- 172.21.15.59:0/3766005066 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f58b00a9b50 con 0x7f58b00a5b70 2023-09-30T02:11:11.239 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:11.240+0000 7f5896ffd700 1 -- 172.21.15.59:0/3766005066 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f58a0017100 con 0x7f58b00a5b70 2023-09-30T02:11:11.240 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:11.240+0000 7f5896ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:11:11.240 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:11.240+0000 7f5896ffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:11:11.240 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:11:11.240 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:11:11.242 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:11.241+0000 7f58b51b6700 10 librados: watch_flush enter 2023-09-30T02:11:11.242 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:11.241+0000 7f58b51b6700 10 librados: watch_flush exit 2023-09-30T02:11:11.242 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:11.241+0000 7f58b51b6700 20 client.4822.objecter shutdown clearing up homeless session... 2023-09-30T02:11:11.242 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:11.241+0000 7f58b51b6700 10 client.4822.objecter successfully canceled tick 2023-09-30T02:11:11.243 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:11.242+0000 7f58b51b6700 1 -- 172.21.15.59:0/3766005066 >> v2:172.21.15.110:6800/115870 conn(0x7f589c0688c0 msgr2=0x7f589c06ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:11.243 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:11.242+0000 7f58b51b6700 1 --2- 172.21.15.59:0/3766005066 >> v2:172.21.15.110:6800/115870 conn(0x7f589c0688c0 0x7f589c06ad70 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f58b013df50 tx=0x7f58a4017000 comp rx=0 tx=0).stop 2023-09-30T02:11:11.243 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:11.242+0000 7f58b51b6700 10 monclient: shutdown 2023-09-30T02:11:11.243 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:11.242+0000 7f58b51b6700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:11:11.243 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:11.242+0000 7f58b51b6700 1 -- 172.21.15.59:0/3766005066 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f58b00a5b70 msgr2=0x7f58b013d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:11.243 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:11.242+0000 7f58b51b6700 1 --2- 172.21.15.59:0/3766005066 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f58b00a5b70 0x7f58b013d790 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f58a0006d40 tx=0x7f58a0009250 comp rx=0 tx=0).stop 2023-09-30T02:11:11.244 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:11.242+0000 7f58b51b6700 1 -- 172.21.15.59:0/3766005066 shutdown_connections 2023-09-30T02:11:11.244 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:11.242+0000 7f58b51b6700 1 --2- 172.21.15.59:0/3766005066 >> v2:172.21.15.110:6800/115870 conn(0x7f589c0688c0 0x7f589c06ad70 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:11.244 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:11.242+0000 7f58b51b6700 1 --2- 172.21.15.59:0/3766005066 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f58b00a5b70 0x7f58b013d790 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:11.244 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:11.242+0000 7f58b51b6700 1 --2- 172.21.15.59:0/3766005066 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58b01420c0 0x7f58b00a9d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:11.244 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:11.242+0000 7f58b51b6700 1 --2- 172.21.15.59:0/3766005066 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f58b00a5f40 0x7f58b00a8fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:11.244 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:11.242+0000 7f58b51b6700 1 -- 172.21.15.59:0/3766005066 >> 172.21.15.59:0/3766005066 conn(0x7f58b00631f0 msgr2=0x7f58b0137cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:11:11.244 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:11.242+0000 7f58b51b6700 1 -- 172.21.15.59:0/3766005066 shutdown_connections 2023-09-30T02:11:11.244 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:11.243+0000 7f58b51b6700 1 -- 172.21.15.59:0/3766005066 wait complete. 2023-09-30T02:11:11.244 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:11.243+0000 7f58b51b6700 1 librados: shutdown 2023-09-30T02:11:14.257 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:11:14.407 INFO:tasks.ceph.mon.a.smithi059.stderr:==00:00:06:24.272 116256== Warning: unimplemented fcntl command: 1036 2023-09-30T02:11:14.408 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.407+0000 7fadd6987700 10 monclient: get_monmap_and_config 2023-09-30T02:11:14.408 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.407+0000 7fadd6987700 10 monclient: build_initial_monmap 2023-09-30T02:11:14.408 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.407+0000 7fadd6987700 10 monclient: monmap: 2023-09-30T02:11:14.408 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:11:14.408 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:14.408 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:11:14.408943+0000 2023-09-30T02:11:14.408 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:11:14.408943+0000 2023-09-30T02:11:14.408 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:11:14.408 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:14.409 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:11:14.409 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:11:14.409 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:11:14.409 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:11:14.409 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:11:14.409 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:14.411 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.408+0000 7fadd6987700 1 Processor -- start 2023-09-30T02:11:14.411 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.408+0000 7fadd6987700 1 -- start start 2023-09-30T02:11:14.411 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.408+0000 7fadd6987700 10 monclient: init 2023-09-30T02:11:14.411 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.409+0000 7fadd6987700 10 monclient: _reopen_session rank -1 2023-09-30T02:11:14.411 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.409+0000 7fadd6987700 10 monclient: _add_conns ranks=[4,2,0,1,3] 2023-09-30T02:11:14.411 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.409+0000 7fadd6987700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fadd00a5b70 0x7fadd00b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:14.411 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.409+0000 7fadd6987700 10 monclient(hunting): picked mon.noname-c con 0x7fadd00a5b70 addr v2:172.21.15.110:3300/0 2023-09-30T02:11:14.411 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.409+0000 7fadd6987700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fadd00a5f40 0x7fadd0141b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:14.412 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.409+0000 7fadd6987700 10 monclient(hunting): picked mon.noname-b con 0x7fadd00a5f40 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:11:14.412 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.409+0000 7fadd6987700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fadd01420c0 addr v1:172.21.15.59:6789/0 2023-09-30T02:11:14.412 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.409+0000 7fadd6987700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadd00a63b0 con 0x7fadd01420c0 2023-09-30T02:11:14.412 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.409+0000 7fadd6987700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:14.412 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.409+0000 7fadd6987700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadd00a75a0 con 0x7fadd00a5b70 2023-09-30T02:11:14.412 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.409+0000 7fadd6987700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:14.413 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.409+0000 7fadd6987700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadd00af3f0 con 0x7fadd00a5f40 2023-09-30T02:11:14.413 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.409+0000 7fadd6987700 10 monclient(hunting): _renew_subs 2023-09-30T02:11:14.413 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.409+0000 7fadd6987700 10 monclient(hunting): authenticate will time out at 2119.712449s 2023-09-30T02:11:14.413 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.410+0000 7fadcffff700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fadd00a5b70 0x7fadd00b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:14.413 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.410+0000 7fadcffff700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fadd00a5b70 0x7fadd00b1500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:58890/0 (socket says 172.21.15.59:58890) 2023-09-30T02:11:14.413 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.410+0000 7fadcffff700 1 -- 172.21.15.59:0/1746281259 learned_addr learned my addr 172.21.15.59:0/1746281259 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:11:14.413 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.410+0000 7fadcffff700 10 monclient(hunting): get_auth_request con 0x7fadd00a5b70 auth_method 0 2023-09-30T02:11:14.413 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.410+0000 7fadcffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:14.413 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.410+0000 7fadcffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:14.413 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.410+0000 7fadcffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:14.414 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.411+0000 7fadcf7fe700 1 --2- 172.21.15.59:0/1746281259 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fadd00a5f40 0x7fadd0141b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:14.415 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.411+0000 7fadcffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:14.415 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.411+0000 7fadcffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:14.415 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.411+0000 7fadcffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:14.415 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.414+0000 7fadcffff700 10 monclient(hunting): handle_auth_done global_id 4825 payload 274 2023-09-30T02:11:14.415 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.414+0000 7fadcffff700 1 -- 172.21.15.59:0/1746281259 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fadd00a5f40 msgr2=0x7fadd0141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:11:14.415 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.414+0000 7fadcffff700 1 --2- 172.21.15.59:0/1746281259 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fadd00a5f40 0x7fadd0141b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:14.415 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.414+0000 7fadcffff700 1 -- 172.21.15.59:0/1746281259 >> v1:172.21.15.59:6789/0 conn(0x7fadd01420c0 legacy=0x7fadd013e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:11:14.415 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.414+0000 7fadcffff700 10 monclient: _finish_hunting 0 2023-09-30T02:11:14.415 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.414+0000 7fadcffff700 1 monclient: found mon.noname-c 2023-09-30T02:11:14.415 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.414+0000 7fadcffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:11:14.416 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.414+0000 7fadcffff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3300/0 2023-09-30T02:11:14.416 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.414+0000 7fadcffff700 1 -- 172.21.15.59:0/1746281259 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadd0142500 con 0x7fadd00a5b70 2023-09-30T02:11:14.416 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.416+0000 7fadcffff700 1 --2- 172.21.15.59:0/1746281259 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fadd00a5b70 0x7fadd00b1500 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fadc00099b0 tx=0x7fadc0009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=417d552f90d33801 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:14.421 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.420+0000 7fadceffd700 1 -- 172.21.15.59:0/1746281259 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadc000b820 con 0x7fadd00a5b70 2023-09-30T02:11:14.421 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.420+0000 7fadceffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:14.421 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.420+0000 7fadceffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:11:14.421 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.420+0000 7fadceffd700 10 monclient: dump: 2023-09-30T02:11:14.421 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:14.421 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:14.421 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:14.422 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:14.422 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:14.422 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:14.422 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:14.422 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:14.422 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:14.422 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:14.422 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.420+0000 7fadceffd700 10 monclient: _finish_auth 0 2023-09-30T02:11:14.422 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.420+0000 7fadceffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:11:14.422 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.420+0000 7fadceffd700 1 -- 172.21.15.59:0/1746281259 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadc000b980 con 0x7fadd00a5b70 2023-09-30T02:11:14.423 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.420+0000 7fadd6987700 5 monclient: authenticate success, global_id 4825 2023-09-30T02:11:14.423 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.420+0000 7fadd6987700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:11:14.423 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.420+0000 7fadceffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:11:14.423 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.420+0000 7fadd6987700 10 monclient: get_monmap_and_config success 2023-09-30T02:11:14.423 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.420+0000 7fadceffd700 1 -- 172.21.15.59:0/1746281259 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadc000bd20 con 0x7fadd00a5b70 2023-09-30T02:11:14.423 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.420+0000 7fadd6987700 4 set_mon_vals no callback set 2023-09-30T02:11:14.423 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.420+0000 7fadceffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:14.423 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.420+0000 7fadceffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:11:14.423 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.420+0000 7fadceffd700 10 monclient: dump: 2023-09-30T02:11:14.423 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:14.423 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:14.424 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:14.424 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:14.424 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:14.424 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:14.424 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:14.424 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:14.424 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:14.424 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:14.426 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.421+0000 7fadd6987700 10 monclient: shutdown 2023-09-30T02:11:14.426 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.421+0000 7fadd6987700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:11:14.426 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.421+0000 7fadd6987700 1 -- 172.21.15.59:0/1746281259 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fadd00a5b70 msgr2=0x7fadd00b1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:14.427 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.421+0000 7fadd6987700 1 --2- 172.21.15.59:0/1746281259 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fadd00a5b70 0x7fadd00b1500 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fadc00099b0 tx=0x7fadc0009250 comp rx=0 tx=0).stop 2023-09-30T02:11:14.427 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.421+0000 7fadd6987700 1 -- 172.21.15.59:0/1746281259 shutdown_connections 2023-09-30T02:11:14.427 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.421+0000 7fadd6987700 1 --2- 172.21.15.59:0/1746281259 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fadd00a5b70 0x7fadd00b1500 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:14.427 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.421+0000 7fadd6987700 1 --2- 172.21.15.59:0/1746281259 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fadd00a5f40 0x7fadd0141b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:14.427 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.421+0000 7fadd6987700 1 -- 172.21.15.59:0/1746281259 >> 172.21.15.59:0/1746281259 conn(0x7fadd00631f0 msgr2=0x7fadd00635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:11:14.427 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.421+0000 7fadd6987700 1 -- 172.21.15.59:0/1746281259 shutdown_connections 2023-09-30T02:11:14.427 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.421+0000 7fadd6987700 1 -- 172.21.15.59:0/1746281259 wait complete. 2023-09-30T02:11:14.427 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.422+0000 7fadd6987700 10 monclient: build_initial_monmap 2023-09-30T02:11:14.427 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.422+0000 7fadd6987700 10 monclient: monmap: 2023-09-30T02:11:14.427 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:11:14.428 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:11:14.428 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:11:14.428 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:11:14.428 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:11:14.428 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:11:14.428 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:11:14.428 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:11:14.428 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:11:14.428 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:14.428 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.422+0000 7fadd6987700 1 librados: starting msgr at 2023-09-30T02:11:14.428 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.422+0000 7fadd6987700 1 librados: starting objecter 2023-09-30T02:11:14.429 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.422+0000 7fadd6987700 1 Processor -- start 2023-09-30T02:11:14.429 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.423+0000 7fadd6987700 1 -- start start 2023-09-30T02:11:14.429 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.423+0000 7fadd6987700 1 librados: setting wanted keys 2023-09-30T02:11:14.429 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.423+0000 7fadd6987700 1 librados: calling monclient init 2023-09-30T02:11:14.429 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.423+0000 7fadd6987700 10 monclient: init 2023-09-30T02:11:14.429 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.423+0000 7fadd6987700 10 monclient: _reopen_session rank -1 2023-09-30T02:11:14.429 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.423+0000 7fadd6987700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T02:11:14.429 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.423+0000 7fadd6987700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fadd00a5b70 0x7fadd0139890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:14.429 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.423+0000 7fadd6987700 10 monclient(hunting): picked mon.noname-c con 0x7fadd00a5b70 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:11:14.429 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.423+0000 7fadd6987700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fadd00a5f40 0x7fadd0139f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:14.429 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.423+0000 7fadd6987700 10 monclient(hunting): picked mon.noname-a con 0x7fadd00a5f40 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:11:14.430 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.423+0000 7fadd6987700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fadd01420c0 0x7fadd00acdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:14.430 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.423+0000 7fadd6987700 10 monclient(hunting): picked mon.noname-b con 0x7fadd01420c0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:11:14.430 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.423+0000 7fadd6987700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:14.430 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.423+0000 7fadd6987700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadd0136a10 con 0x7fadd00a5f40 2023-09-30T02:11:14.430 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.423+0000 7fadd6987700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:14.430 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.423+0000 7fadd6987700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadd013a6f0 con 0x7fadd01420c0 2023-09-30T02:11:14.430 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.423+0000 7fadd6987700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:14.430 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.423+0000 7fadd6987700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadd013ab80 con 0x7fadd00a5b70 2023-09-30T02:11:14.430 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.423+0000 7fadd6987700 10 monclient(hunting): _renew_subs 2023-09-30T02:11:14.430 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.423+0000 7fadd6987700 10 monclient(hunting): authenticate will time out at 2119.726808s 2023-09-30T02:11:14.431 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:06:24.289 116257== Warning: unimplemented fcntl command: 1036 2023-09-30T02:11:14.432 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.425+0000 7fadd4f24700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fadd01420c0 0x7fadd00acdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:14.432 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.425+0000 7fadd4f24700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fadd01420c0 0x7fadd00acdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:58906/0 (socket says 172.21.15.59:58906) 2023-09-30T02:11:14.432 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.425+0000 7fadd4f24700 1 -- 172.21.15.59:0/3803985454 learned_addr learned my addr 172.21.15.59:0/3803985454 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:11:14.432 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.425+0000 7fadd4f24700 10 monclient(hunting): get_auth_request con 0x7fadd01420c0 auth_method 0 2023-09-30T02:11:14.432 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.425+0000 7fadd4f24700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:14.432 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.425+0000 7fadd4f24700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:14.433 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.425+0000 7fadd4f24700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:14.433 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.426+0000 7fadd4f24700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:14.433 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.426+0000 7fadd4f24700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:14.433 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.426+0000 7fadd4f24700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:14.433 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.429+0000 7fadd4f24700 10 monclient(hunting): handle_auth_done global_id 4828 payload 824 2023-09-30T02:11:14.433 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.429+0000 7fadd4f24700 1 -- 172.21.15.59:0/3803985454 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fadd00a5b70 msgr2=0x7fadd0139890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:14.433 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.429+0000 7fadd4f24700 1 --2- 172.21.15.59:0/3803985454 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fadd00a5b70 0x7fadd0139890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:14.433 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.429+0000 7fadd4f24700 1 -- 172.21.15.59:0/3803985454 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fadd00a5f40 msgr2=0x7fadd0139f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:14.433 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.429+0000 7fadd4f24700 1 --2- 172.21.15.59:0/3803985454 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fadd00a5f40 0x7fadd0139f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:14.433 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.429+0000 7fadd4f24700 10 monclient: _finish_hunting 0 2023-09-30T02:11:14.433 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.429+0000 7fadd4f24700 1 monclient: found mon.noname-b 2023-09-30T02:11:14.434 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.429+0000 7fadd4f24700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:11:14.434 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.429+0000 7fadd4f24700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:11:14.434 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.429+0000 7fadd4f24700 1 -- 172.21.15.59:0/3803985454 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadc00094f0 con 0x7fadd01420c0 2023-09-30T02:11:14.434 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.432+0000 7fadd4f24700 1 --2- 172.21.15.59:0/3803985454 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fadd01420c0 0x7fadd00acdd0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fadc800bbb0 tx=0x7fadc8009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:14.434 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.432+0000 7fadb7fff700 10 client.?.objecter ms_handle_connect 0x7fadd01420c0 2023-09-30T02:11:14.434 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.432+0000 7fadb7fff700 10 client.?.objecter resend_mon_ops 2023-09-30T02:11:14.437 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.436+0000 7fadb7fff700 1 -- 172.21.15.59:0/3803985454 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadc80092e0 con 0x7fadd01420c0 2023-09-30T02:11:14.437 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.436+0000 7fadb7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:14.437 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.436+0000 7fadb7fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:11:14.437 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.436+0000 7fadb7fff700 10 monclient: dump: 2023-09-30T02:11:14.437 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:14.437 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:14.438 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:14.438 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:14.438 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:14.438 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:14.438 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:14.438 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:14.438 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:14.438 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:14.438 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.436+0000 7fadb7fff700 10 monclient: _finish_auth 0 2023-09-30T02:11:14.438 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.436+0000 7fadb7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:11:14.438 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.436+0000 7fadd6987700 5 monclient: authenticate success, global_id 4828 2023-09-30T02:11:14.439 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.436+0000 7fadd6987700 10 monclient: _renew_subs 2023-09-30T02:11:14.439 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.436+0000 7fadb7fff700 1 -- 172.21.15.59:0/3803985454 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadc800c7a0 con 0x7fadd01420c0 2023-09-30T02:11:14.439 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.436+0000 7fadd6987700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:11:14.439 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.436+0000 7fadd6987700 1 -- 172.21.15.59:0/3803985454 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadd013bca0 con 0x7fadd01420c0 2023-09-30T02:11:14.439 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.436+0000 7fadb7fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:11:14.439 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.436+0000 7fadb7fff700 1 -- 172.21.15.59:0/3803985454 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadc800cb40 con 0x7fadd01420c0 2023-09-30T02:11:14.439 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.436+0000 7fadb7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:14.439 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.436+0000 7fadb7fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:11:14.439 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.436+0000 7fadb7fff700 10 monclient: dump: 2023-09-30T02:11:14.439 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:14.439 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:14.440 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:14.440 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:14.440 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:14.440 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:14.440 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:14.440 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:14.440 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:14.440 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:14.440 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.436+0000 7fadce7fc700 4 set_mon_vals no callback set 2023-09-30T02:11:14.440 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.436+0000 7fadd6987700 10 client.4828.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:11:14.440 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.436+0000 7fadd6987700 10 monclient: _renew_subs 2023-09-30T02:11:14.441 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.436+0000 7fadd6987700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:11:14.441 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.436+0000 7fadd6987700 1 -- 172.21.15.59:0/3803985454 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadd00ad310 con 0x7fadd01420c0 2023-09-30T02:11:14.441 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.436+0000 7fadd6987700 1 librados: init done 2023-09-30T02:11:14.441 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.437+0000 7fadd6987700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:11:14.441 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.437+0000 7fadd6987700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:11:14.442 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.437+0000 7fadd6987700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:11:14.442 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.437+0000 7fadd6987700 1 -- 172.21.15.59:0/3803985454 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadd0064a00 con 0x7fadd01420c0 2023-09-30T02:11:14.446 INFO:tasks.ceph.mon.a.smithi059.stderr:==00:00:06:24.312 116256== Warning: unimplemented fcntl command: 1036 2023-09-30T02:11:14.477 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:06:24.342 116257== Warning: unimplemented fcntl command: 1036 2023-09-30T02:11:14.482 INFO:tasks.ceph.mon.a.smithi059.stderr:==00:00:06:24.347 116256== Warning: unimplemented fcntl command: 1036 2023-09-30T02:11:14.508 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:06:24.374 116257== Warning: unimplemented fcntl command: 1036 2023-09-30T02:11:14.616 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.614+0000 7fadb7fff700 1 -- 172.21.15.59:0/3803985454 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fadc8012070 con 0x7fadd01420c0 2023-09-30T02:11:14.617 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.614+0000 7fadb7fff700 1 --2- 172.21.15.59:0/3803985454 >> v2:172.21.15.110:6800/115870 conn(0x7fadbc068960 0x7fadbc06ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:14.617 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.615+0000 7fadb7fff700 1 -- 172.21.15.59:0/3803985454 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fadc8083d60 con 0x7fadd01420c0 2023-09-30T02:11:14.617 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.615+0000 7fadb7fff700 10 client.4828.objecter ms_dispatch 0x7fadd0001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:11:14.617 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.615+0000 7fadb7fff700 3 client.4828.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:11:14.617 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.615+0000 7fadb7fff700 3 client.4828.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:11:14.617 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.615+0000 7fadb7fff700 20 client.4828.objecter dump_active .. 0 homeless 2023-09-30T02:11:14.617 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.615+0000 7fadcffff700 1 --2- 172.21.15.59:0/3803985454 >> v2:172.21.15.110:6800/115870 conn(0x7fadbc068960 0x7fadbc06ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:14.617 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.615+0000 7fadcffff700 10 monclient: get_auth_request con 0x7fadbc068960 auth_method 0 2023-09-30T02:11:14.617 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.615+0000 7fadb7fff700 10 client.4828.objecter ms_handle_connect 0x7fadbc068960 2023-09-30T02:11:14.617 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.615+0000 7fadcffff700 1 --2- 172.21.15.59:0/3803985454 >> v2:172.21.15.110:6800/115870 conn(0x7fadbc068960 0x7fadbc06ae10 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fadc00099b0 tx=0x7fadc0016000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:14.752 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.751+0000 7fadb7fff700 1 -- 172.21.15.59:0/3803985454 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fadc8051780 con 0x7fadd01420c0 2023-09-30T02:11:14.752 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.751+0000 7fadb7fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:11:14.752 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:14.751+0000 7fadb7fff700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:11:14.753 INFO:tasks.ceph.mon.b.smithi110.stderr:==00:00:06:24.600 115869== Warning: unimplemented fcntl command: 1036 2023-09-30T02:11:14.755 INFO:tasks.ceph.mon.b.smithi110.stderr:==00:00:06:24.602 115869== Warning: unimplemented fcntl command: 1036 2023-09-30T02:11:14.833 INFO:tasks.ceph.mon.b.smithi110.stderr:==00:00:06:24.680 115869== Warning: unimplemented fcntl command: 1036 2023-09-30T02:11:15.010 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:15.009+0000 7fadd6987700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:11:15.010 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:15.009+0000 7fadd6987700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:11:15.010 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:15.009+0000 7fadd6987700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:11:15.010 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:15.009+0000 7fadd6987700 1 -- 172.21.15.59:0/3803985454 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fadd013a830 con 0x7fadd01420c0 2023-09-30T02:11:15.025 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:15.025+0000 7fadb7fff700 1 -- 172.21.15.59:0/3803985454 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fadc8054da0 con 0x7fadd01420c0 2023-09-30T02:11:15.025 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:15.025+0000 7fadb7fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:11:15.025 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:15.025+0000 7fadb7fff700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:11:15.026 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:11:15.026 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:11:15.028 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:15.027+0000 7fadd6987700 10 librados: watch_flush enter 2023-09-30T02:11:15.028 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:15.027+0000 7fadd6987700 10 librados: watch_flush exit 2023-09-30T02:11:15.028 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:15.027+0000 7fadd6987700 20 client.4828.objecter shutdown clearing up homeless session... 2023-09-30T02:11:15.028 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:15.027+0000 7fadd6987700 10 client.4828.objecter successfully canceled tick 2023-09-30T02:11:15.028 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:15.027+0000 7fadd6987700 1 -- 172.21.15.59:0/3803985454 >> v2:172.21.15.110:6800/115870 conn(0x7fadbc068960 msgr2=0x7fadbc06ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:15.029 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:15.027+0000 7fadd6987700 1 --2- 172.21.15.59:0/3803985454 >> v2:172.21.15.110:6800/115870 conn(0x7fadbc068960 0x7fadbc06ae10 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fadc00099b0 tx=0x7fadc0016000 comp rx=0 tx=0).stop 2023-09-30T02:11:15.029 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:15.027+0000 7fadd6987700 10 monclient: shutdown 2023-09-30T02:11:15.029 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:15.027+0000 7fadd6987700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:11:15.029 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:15.027+0000 7fadd6987700 1 -- 172.21.15.59:0/3803985454 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fadd01420c0 msgr2=0x7fadd00acdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:15.029 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:15.027+0000 7fadd6987700 1 --2- 172.21.15.59:0/3803985454 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fadd01420c0 0x7fadd00acdd0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fadc800bbb0 tx=0x7fadc8009840 comp rx=0 tx=0).stop 2023-09-30T02:11:15.029 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:15.027+0000 7fadd6987700 1 -- 172.21.15.59:0/3803985454 shutdown_connections 2023-09-30T02:11:15.029 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:15.027+0000 7fadd6987700 1 --2- 172.21.15.59:0/3803985454 >> v2:172.21.15.110:6800/115870 conn(0x7fadbc068960 0x7fadbc06ae10 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:15.029 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:15.027+0000 7fadd6987700 1 --2- 172.21.15.59:0/3803985454 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fadd01420c0 0x7fadd00acdd0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:15.030 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:15.027+0000 7fadd6987700 1 --2- 172.21.15.59:0/3803985454 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fadd00a5b70 0x7fadd0139890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:15.030 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:15.027+0000 7fadd6987700 1 --2- 172.21.15.59:0/3803985454 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fadd00a5f40 0x7fadd0139f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:15.030 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:15.027+0000 7fadd6987700 1 -- 172.21.15.59:0/3803985454 >> 172.21.15.59:0/3803985454 conn(0x7fadd00631f0 msgr2=0x7fadd0144e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:11:15.030 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:15.027+0000 7fadd6987700 1 -- 172.21.15.59:0/3803985454 shutdown_connections 2023-09-30T02:11:15.030 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:15.027+0000 7fadd6987700 1 -- 172.21.15.59:0/3803985454 wait complete. 2023-09-30T02:11:15.030 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:15.027+0000 7fadd6987700 1 librados: shutdown 2023-09-30T02:11:18.043 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:11:18.196 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.196+0000 7f084d4eb700 10 monclient: get_monmap_and_config 2023-09-30T02:11:18.197 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.196+0000 7f084d4eb700 10 monclient: build_initial_monmap 2023-09-30T02:11:18.197 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.196+0000 7f084d4eb700 10 monclient: monmap: 2023-09-30T02:11:18.197 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:11:18.197 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:18.197 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:11:18.197484+0000 2023-09-30T02:11:18.197 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:11:18.197484+0000 2023-09-30T02:11:18.197 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:11:18.197 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:18.198 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:11:18.198 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:11:18.198 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:11:18.198 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:11:18.198 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:11:18.198 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:18.198 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.196+0000 7f084d4eb700 1 Processor -- start 2023-09-30T02:11:18.198 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.196+0000 7f084d4eb700 1 -- start start 2023-09-30T02:11:18.198 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.196+0000 7f084d4eb700 10 monclient: init 2023-09-30T02:11:18.199 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.197+0000 7f084d4eb700 10 monclient: _reopen_session rank -1 2023-09-30T02:11:18.199 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.197+0000 7f084d4eb700 10 monclient: _add_conns ranks=[4,0,1,3,2] 2023-09-30T02:11:18.200 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.197+0000 7f084d4eb700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f08480a5b70 0x7f08480b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:18.200 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.197+0000 7f084d4eb700 10 monclient(hunting): picked mon.noname-c con 0x7f08480a5b70 addr v2:172.21.15.110:3300/0 2023-09-30T02:11:18.200 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.197+0000 7f084d4eb700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f08480a5f40 addr v1:172.21.15.59:6789/0 2023-09-30T02:11:18.200 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.197+0000 7f084d4eb700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0848142290 addr v1:172.21.15.110:6789/0 2023-09-30T02:11:18.200 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.197+0000 7f084d4eb700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08480a75a0 con 0x7f08480a5f40 2023-09-30T02:11:18.200 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.197+0000 7f084d4eb700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08480af3f0 con 0x7f0848142290 2023-09-30T02:11:18.200 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.197+0000 7f084d4eb700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:18.200 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.197+0000 7f084d4eb700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08480a63e0 con 0x7f08480a5b70 2023-09-30T02:11:18.200 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.197+0000 7f084d4eb700 10 monclient(hunting): _renew_subs 2023-09-30T02:11:18.200 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.197+0000 7f084d4eb700 10 monclient(hunting): authenticate will time out at 2123.500517s 2023-09-30T02:11:18.200 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.198+0000 7f08467fc700 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f08480a5f40 0x7f0848141b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:51936/0 (socket says 172.21.15.59:51936) 2023-09-30T02:11:18.201 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.198+0000 7f08467fc700 1 -- 172.21.15.59:0/528021832 learned_addr learned my addr 172.21.15.59:0/528021832 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:11:18.202 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.202+0000 7f0845ffb700 1 -- 172.21.15.59:0/528021832 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2208716478 0 0) 0x7f08480a75a0 con 0x7f08480a5f40 2023-09-30T02:11:18.202 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.202+0000 7f0845ffb700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:18.203 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.202+0000 7f0845ffb700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:18.203 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.202+0000 7f0845ffb700 10 monclient(hunting): my global_id is 4794 2023-09-30T02:11:18.203 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.202+0000 7f0845ffb700 1 -- 172.21.15.59:0/528021832 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0834003570 con 0x7f08480a5f40 2023-09-30T02:11:18.209 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.208+0000 7f0845ffb700 1 -- 172.21.15.59:0/528021832 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 909937473 0 0) 0x7f0834003570 con 0x7f08480a5f40 2023-09-30T02:11:18.209 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.208+0000 7f0845ffb700 1 -- 172.21.15.59:0/528021832 >> v2:172.21.15.110:3300/0 conn(0x7f08480a5b70 msgr2=0x7f08480b1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:11:18.209 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.208+0000 7f0845ffb700 1 --2- 172.21.15.59:0/528021832 >> v2:172.21.15.110:3300/0 conn(0x7f08480a5b70 0x7f08480b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:18.210 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.208+0000 7f0845ffb700 1 -- 172.21.15.59:0/528021832 >> v1:172.21.15.110:6789/0 conn(0x7f0848142290 legacy=0x7f084813af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:11:18.210 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.208+0000 7f0845ffb700 10 monclient: _finish_hunting 0 2023-09-30T02:11:18.210 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.208+0000 7f0845ffb700 1 monclient: found mon.noname-a-legacy 2023-09-30T02:11:18.210 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.208+0000 7f0845ffb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:11:18.210 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.208+0000 7f0845ffb700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.59:6789/0 2023-09-30T02:11:18.210 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.208+0000 7f0845ffb700 1 -- 172.21.15.59:0/528021832 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f084813b640 con 0x7f08480a5f40 2023-09-30T02:11:18.210 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.208+0000 7f0845ffb700 10 monclient: _finish_auth 0 2023-09-30T02:11:18.210 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.208+0000 7f0845ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:11:18.210 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.208+0000 7f0845ffb700 1 -- 172.21.15.59:0/528021832 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2174283827 0 0) 0x7f0830001470 con 0x7f08480a5f40 2023-09-30T02:11:18.210 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.208+0000 7f084d4eb700 5 monclient: authenticate success, global_id 4794 2023-09-30T02:11:18.210 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.208+0000 7f084d4eb700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:11:18.211 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.208+0000 7f0845ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:18.211 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.208+0000 7f0845ffb700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-09-30T02:11:18.211 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.208+0000 7f0845ffb700 10 monclient: dump: 2023-09-30T02:11:18.211 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:18.211 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:18.211 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:18.211 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:18.211 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:18.211 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:18.211 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:18.211 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:18.211 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:18.212 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:18.212 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.208+0000 7f0845ffb700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] but i'm connected to v1:172.21.15.59:6789/0, reconnecting 2023-09-30T02:11:18.212 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.208+0000 7f0845ffb700 10 monclient: _reopen_session rank -1 2023-09-30T02:11:18.212 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.208+0000 7f0845ffb700 1 -- 172.21.15.59:0/528021832 >> v1:172.21.15.59:6789/0 conn(0x7f08480a5f40 legacy=0x7f0848141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:18.212 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.208+0000 7f0845ffb700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T02:11:18.212 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.208+0000 7f0845ffb700 1 --2- 172.21.15.59:0/528021832 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0834003b30 0x7f0834003f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:18.212 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.208+0000 7f0845ffb700 10 monclient(hunting): picked mon.c con 0x7f0834003b30 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:11:18.212 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.208+0000 7f0845ffb700 1 --2- 172.21.15.59:0/528021832 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0834004810 0x7f0834004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:18.212 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.208+0000 7f0845ffb700 10 monclient(hunting): picked mon.a con 0x7f0834004810 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:11:18.212 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.208+0000 7f0845ffb700 1 --2- 172.21.15.59:0/528021832 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0834005410 0x7f083400bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:18.212 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.208+0000 7f0845ffb700 10 monclient(hunting): picked mon.b con 0x7f0834005410 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:11:18.213 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.208+0000 7f0845ffb700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:18.213 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.208+0000 7f0845ffb700 1 -- 172.21.15.59:0/528021832 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08480a63e0 con 0x7f0834004810 2023-09-30T02:11:18.213 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.208+0000 7f0845ffb700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:18.213 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.208+0000 7f0845ffb700 1 -- 172.21.15.59:0/528021832 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f083400c3a0 con 0x7f0834005410 2023-09-30T02:11:18.213 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.208+0000 7f0845ffb700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:18.213 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.208+0000 7f0845ffb700 1 -- 172.21.15.59:0/528021832 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f083400c560 con 0x7f0834003b30 2023-09-30T02:11:18.213 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.208+0000 7f0845ffb700 10 monclient(hunting): _renew_subs 2023-09-30T02:11:18.213 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.208+0000 7f0845ffb700 10 monclient(hunting): _finish_auth 0 2023-09-30T02:11:18.213 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.208+0000 7f084d4eb700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-09-30T02:11:18.214 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.211+0000 7f08467fc700 1 --2- 172.21.15.59:0/528021832 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0834004810 0x7f0834004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:18.214 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.212+0000 7f08467fc700 10 monclient(hunting): get_auth_request con 0x7f0834004810 auth_method 0 2023-09-30T02:11:18.214 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.212+0000 7f08467fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:18.214 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.212+0000 7f08467fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:18.214 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.212+0000 7f08467fc700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T02:11:18.215 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.214+0000 7f08467fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:18.215 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.214+0000 7f08467fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:18.215 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.214+0000 7f08467fc700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T02:11:18.218 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.217+0000 7f0846ffd700 1 --2- 172.21.15.59:0/528021832 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0834003b30 0x7f0834003f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:18.219 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.218+0000 7f0846ffd700 10 monclient(hunting): get_auth_request con 0x7f0834003b30 auth_method 0 2023-09-30T02:11:18.219 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.218+0000 7f0846ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:18.219 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.218+0000 7f0846ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:18.219 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.218+0000 7f0846ffd700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T02:11:18.219 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.219+0000 7f0846ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:18.219 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.219+0000 7f0846ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:18.219 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.219+0000 7f0846ffd700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T02:11:18.220 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.219+0000 7f08467fc700 10 monclient(hunting): handle_auth_done global_id 4794 payload 293 2023-09-30T02:11:18.220 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.219+0000 7f08467fc700 1 -- 172.21.15.59:0/528021832 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0834003b30 msgr2=0x7f0834003f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:11:18.220 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.219+0000 7f08467fc700 1 --2- 172.21.15.59:0/528021832 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0834003b30 0x7f0834003f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:18.220 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.219+0000 7f08467fc700 1 -- 172.21.15.59:0/528021832 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0834005410 msgr2=0x7f083400bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:11:18.220 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.219+0000 7f08467fc700 1 --2- 172.21.15.59:0/528021832 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f0834005410 0x7f083400bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:18.220 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.219+0000 7f08467fc700 10 monclient: _finish_hunting 0 2023-09-30T02:11:18.220 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.219+0000 7f08467fc700 1 monclient: found mon.a 2023-09-30T02:11:18.220 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.219+0000 7f08467fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:11:18.221 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.219+0000 7f08467fc700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:11:18.221 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.219+0000 7f08467fc700 1 -- 172.21.15.59:0/528021832 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0834003570 con 0x7f0834004810 2023-09-30T02:11:18.221 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.219+0000 7f08467fc700 10 monclient: _finish_auth 0 2023-09-30T02:11:18.221 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.219+0000 7f08467fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:11:18.221 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.219+0000 7f08477fe700 1 -- 172.21.15.59:0/528021832 reap_dead start 2023-09-30T02:11:18.222 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.222+0000 7f08467fc700 1 --2- 172.21.15.59:0/528021832 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0834004810 0x7f0834004c80 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f083000c2d0 tx=0x7f0830001da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f19b114a30b50a0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:18.228 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.227+0000 7f0845ffb700 1 -- 172.21.15.59:0/528021832 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f083000cba0 con 0x7f0834004810 2023-09-30T02:11:18.228 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.227+0000 7f0845ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:18.228 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.227+0000 7f0845ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T02:11:18.228 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.227+0000 7f0845ffb700 10 monclient: dump: 2023-09-30T02:11:18.228 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:18.228 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:18.228 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:18.229 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:18.229 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:18.229 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:18.229 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:18.229 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:18.229 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:18.229 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:18.229 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.227+0000 7f0845ffb700 1 -- 172.21.15.59:0/528021832 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f083000cd40 con 0x7f0834004810 2023-09-30T02:11:18.229 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.227+0000 7f084d4eb700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:11:18.229 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.227+0000 7f0845ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:11:18.229 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.227+0000 7f084d4eb700 10 monclient: get_monmap_and_config success 2023-09-30T02:11:18.230 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.227+0000 7f084d4eb700 4 set_mon_vals no callback set 2023-09-30T02:11:18.230 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.227+0000 7f0845ffb700 1 -- 172.21.15.59:0/528021832 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0830005b70 con 0x7f0834004810 2023-09-30T02:11:18.230 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.227+0000 7f0845ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:18.230 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.227+0000 7f0845ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T02:11:18.230 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.227+0000 7f0845ffb700 10 monclient: dump: 2023-09-30T02:11:18.230 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:18.230 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:18.230 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:18.230 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:18.230 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:18.230 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:18.231 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:18.231 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:18.231 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:18.231 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:18.233 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.228+0000 7f084d4eb700 10 monclient: shutdown 2023-09-30T02:11:18.233 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.228+0000 7f084d4eb700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:11:18.233 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.228+0000 7f084d4eb700 1 -- 172.21.15.59:0/528021832 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0834004810 msgr2=0x7f0834004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:18.234 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.228+0000 7f084d4eb700 1 --2- 172.21.15.59:0/528021832 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0834004810 0x7f0834004c80 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f083000c2d0 tx=0x7f0830001da0 comp rx=0 tx=0).stop 2023-09-30T02:11:18.234 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.228+0000 7f084d4eb700 1 -- 172.21.15.59:0/528021832 shutdown_connections 2023-09-30T02:11:18.234 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.228+0000 7f084d4eb700 1 --2- 172.21.15.59:0/528021832 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0834004810 0x7f0834004c80 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:18.234 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.228+0000 7f084d4eb700 1 -- 172.21.15.59:0/528021832 >> 172.21.15.59:0/528021832 conn(0x7f08480631f0 msgr2=0x7f08480635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:11:18.234 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.228+0000 7f084d4eb700 1 -- 172.21.15.59:0/528021832 shutdown_connections 2023-09-30T02:11:18.234 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.228+0000 7f084d4eb700 1 -- 172.21.15.59:0/528021832 wait complete. 2023-09-30T02:11:18.234 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.229+0000 7f084d4eb700 10 monclient: build_initial_monmap 2023-09-30T02:11:18.234 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.229+0000 7f084d4eb700 10 monclient: monmap: 2023-09-30T02:11:18.234 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:11:18.234 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:11:18.234 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:11:18.235 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:11:18.235 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:11:18.235 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:11:18.235 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:11:18.235 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:11:18.235 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:11:18.235 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:18.235 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.230+0000 7f084d4eb700 1 librados: starting msgr at 2023-09-30T02:11:18.235 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.230+0000 7f084d4eb700 1 librados: starting objecter 2023-09-30T02:11:18.235 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.230+0000 7f084d4eb700 1 Processor -- start 2023-09-30T02:11:18.235 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.230+0000 7f084d4eb700 1 -- start start 2023-09-30T02:11:18.235 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.230+0000 7f084d4eb700 1 librados: setting wanted keys 2023-09-30T02:11:18.235 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.230+0000 7f084d4eb700 1 librados: calling monclient init 2023-09-30T02:11:18.236 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.230+0000 7f084d4eb700 10 monclient: init 2023-09-30T02:11:18.236 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.231+0000 7f084d4eb700 10 monclient: _reopen_session rank -1 2023-09-30T02:11:18.236 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.231+0000 7f084d4eb700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T02:11:18.236 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.231+0000 7f084d4eb700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0834004810 0x7f0848139fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:18.236 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.231+0000 7f084d4eb700 10 monclient(hunting): picked mon.noname-c con 0x7f0834004810 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:11:18.236 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.231+0000 7f084d4eb700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f084813dca0 0x7f084813e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:18.236 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.231+0000 7f084d4eb700 10 monclient(hunting): picked mon.noname-b con 0x7f084813dca0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:11:18.236 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.231+0000 7f084d4eb700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0848136c40 0x7f08481370b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:18.236 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.231+0000 7f084d4eb700 10 monclient(hunting): picked mon.noname-a con 0x7f0848136c40 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:11:18.236 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.231+0000 7f084d4eb700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:18.236 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.231+0000 7f084d4eb700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f084813e9f0 con 0x7f0848136c40 2023-09-30T02:11:18.237 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.231+0000 7f084d4eb700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:18.237 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.231+0000 7f084d4eb700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f084813e6b0 con 0x7f084813dca0 2023-09-30T02:11:18.237 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.231+0000 7f084d4eb700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:18.237 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.231+0000 7f084d4eb700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08481375f0 con 0x7f0834004810 2023-09-30T02:11:18.237 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.231+0000 7f084d4eb700 10 monclient(hunting): _renew_subs 2023-09-30T02:11:18.237 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.231+0000 7f084d4eb700 10 monclient(hunting): authenticate will time out at 2123.534504s 2023-09-30T02:11:18.237 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.232+0000 7f0846ffd700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0834004810 0x7f0848139fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:18.237 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.233+0000 7f0846ffd700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0834004810 0x7f0848139fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:41690/0 (socket says 172.21.15.59:41690) 2023-09-30T02:11:18.237 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.233+0000 7f0846ffd700 1 -- 172.21.15.59:0/3589524325 learned_addr learned my addr 172.21.15.59:0/3589524325 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:11:18.237 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.233+0000 7f0846ffd700 10 monclient(hunting): get_auth_request con 0x7f0834004810 auth_method 0 2023-09-30T02:11:18.237 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.233+0000 7f0846ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:18.237 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.233+0000 7f0846ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:18.238 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.233+0000 7f0846ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:18.239 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.233+0000 7f0846ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:18.239 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.233+0000 7f0846ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:18.239 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.233+0000 7f0846ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:18.239 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.236+0000 7f08467fc700 1 --2- 172.21.15.59:0/3589524325 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f084813dca0 0x7f084813e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:18.239 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.237+0000 7f08467fc700 10 monclient(hunting): get_auth_request con 0x7f084813dca0 auth_method 0 2023-09-30T02:11:18.239 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.237+0000 7f08467fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:18.239 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.237+0000 7f08467fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:18.239 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.237+0000 7f08467fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:18.239 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.237+0000 7f0846ffd700 10 monclient(hunting): handle_auth_done global_id 4715 payload 824 2023-09-30T02:11:18.240 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.237+0000 7f0846ffd700 1 -- 172.21.15.59:0/3589524325 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f084813dca0 msgr2=0x7f084813e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:18.240 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.237+0000 7f0846ffd700 1 --2- 172.21.15.59:0/3589524325 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f084813dca0 0x7f084813e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:18.240 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.238+0000 7f0846ffd700 1 -- 172.21.15.59:0/3589524325 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0848136c40 msgr2=0x7f08481370b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:18.240 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.238+0000 7f0846ffd700 1 --2- 172.21.15.59:0/3589524325 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0848136c40 0x7f08481370b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:18.240 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.238+0000 7f0846ffd700 10 monclient: _finish_hunting 0 2023-09-30T02:11:18.240 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.238+0000 7f0846ffd700 1 monclient: found mon.noname-c 2023-09-30T02:11:18.240 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.238+0000 7f0846ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:11:18.240 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.238+0000 7f0846ffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.59:3301/0 2023-09-30T02:11:18.240 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.238+0000 7f0846ffd700 1 -- 172.21.15.59:0/3589524325 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f084814d930 con 0x7f0834004810 2023-09-30T02:11:18.241 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.240+0000 7f0846ffd700 1 --2- 172.21.15.59:0/3589524325 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0834004810 0x7f0848139fa0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f083800e960 tx=0x7f0838009060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:18.241 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.240+0000 7f083effd700 10 client.?.objecter ms_handle_connect 0x7f0834004810 2023-09-30T02:11:18.241 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.240+0000 7f083effd700 10 client.?.objecter resend_mon_ops 2023-09-30T02:11:18.247 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.246+0000 7f083effd700 1 -- 172.21.15.59:0/3589524325 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f083800c7f0 con 0x7f0834004810 2023-09-30T02:11:18.248 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.246+0000 7f083effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:18.248 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.246+0000 7f083effd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:11:18.248 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.246+0000 7f083effd700 10 monclient: dump: 2023-09-30T02:11:18.248 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:18.248 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:18.248 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:18.248 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:18.248 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:18.248 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:18.248 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:18.248 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:18.249 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:18.249 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:18.249 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.247+0000 7f083effd700 10 monclient: _finish_auth 0 2023-09-30T02:11:18.249 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.247+0000 7f083effd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:11:18.249 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.247+0000 7f084d4eb700 5 monclient: authenticate success, global_id 4715 2023-09-30T02:11:18.249 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.247+0000 7f083effd700 1 -- 172.21.15.59:0/3589524325 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f083800c950 con 0x7f0834004810 2023-09-30T02:11:18.249 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.247+0000 7f083effd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:11:18.249 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.247+0000 7f084d4eb700 10 monclient: _renew_subs 2023-09-30T02:11:18.249 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.247+0000 7f083effd700 1 -- 172.21.15.59:0/3589524325 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f083800cd10 con 0x7f0834004810 2023-09-30T02:11:18.249 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.247+0000 7f084d4eb700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:11:18.249 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.247+0000 7f084d4eb700 1 -- 172.21.15.59:0/3589524325 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f084814db80 con 0x7f0834004810 2023-09-30T02:11:18.249 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.247+0000 7f083effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:18.250 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.247+0000 7f083effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:11:18.250 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.247+0000 7f083effd700 10 monclient: dump: 2023-09-30T02:11:18.250 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:18.250 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:18.250 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:18.250 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:18.250 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:18.250 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:18.250 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:18.250 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:18.250 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:18.251 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:18.251 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.247+0000 7f08457fa700 4 set_mon_vals no callback set 2023-09-30T02:11:18.251 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.247+0000 7f084d4eb700 10 client.4715.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:11:18.251 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.247+0000 7f084d4eb700 10 monclient: _renew_subs 2023-09-30T02:11:18.251 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.247+0000 7f084d4eb700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:11:18.251 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.247+0000 7f084d4eb700 1 -- 172.21.15.59:0/3589524325 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f084814e040 con 0x7f0834004810 2023-09-30T02:11:18.251 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.247+0000 7f084d4eb700 1 librados: init done 2023-09-30T02:11:18.252 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.248+0000 7f084d4eb700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:11:18.252 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.248+0000 7f084d4eb700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:11:18.252 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.248+0000 7f084d4eb700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:11:18.252 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.248+0000 7f084d4eb700 1 -- 172.21.15.59:0/3589524325 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f084813eb30 con 0x7f0834004810 2023-09-30T02:11:18.280 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.279+0000 7f083effd700 1 -- 172.21.15.59:0/3589524325 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f0838012600 con 0x7f0834004810 2023-09-30T02:11:18.280 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.279+0000 7f083effd700 1 --2- 172.21.15.59:0/3589524325 >> v2:172.21.15.110:6800/115870 conn(0x7f0828068890 0x7f082806ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:18.280 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.279+0000 7f083effd700 1 -- 172.21.15.59:0/3589524325 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f0838084290 con 0x7f0834004810 2023-09-30T02:11:18.280 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.279+0000 7f083effd700 10 client.4715.objecter ms_dispatch 0x7f0848001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:11:18.280 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.279+0000 7f083effd700 3 client.4715.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:11:18.280 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.279+0000 7f083effd700 3 client.4715.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:11:18.280 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.279+0000 7f083effd700 20 client.4715.objecter dump_active .. 0 homeless 2023-09-30T02:11:18.280 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.279+0000 7f08467fc700 1 --2- 172.21.15.59:0/3589524325 >> v2:172.21.15.110:6800/115870 conn(0x7f0828068890 0x7f082806ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:18.280 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.279+0000 7f08467fc700 10 monclient: get_auth_request con 0x7f0828068890 auth_method 0 2023-09-30T02:11:18.281 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.280+0000 7f083effd700 10 client.4715.objecter ms_handle_connect 0x7f0828068890 2023-09-30T02:11:18.281 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.280+0000 7f08467fc700 1 --2- 172.21.15.59:0/3589524325 >> v2:172.21.15.110:6800/115870 conn(0x7f0828068890 0x7f082806ad40 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f083000c2d0 tx=0x7f0830015000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:18.553 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.553+0000 7f083effd700 1 -- 172.21.15.59:0/3589524325 <== mon.2 v2:172.21.15.59:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0838051b30 con 0x7f0834004810 2023-09-30T02:11:18.553 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.553+0000 7f083effd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:11:18.554 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.553+0000 7f083effd700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:11:18.810 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.809+0000 7f084d4eb700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:11:18.810 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.809+0000 7f084d4eb700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:11:18.810 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.809+0000 7f084d4eb700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:11:18.810 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.809+0000 7f084d4eb700 1 -- 172.21.15.59:0/3589524325 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f084813eb30 con 0x7f0834004810 2023-09-30T02:11:18.816 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.816+0000 7f083effd700 1 -- 172.21.15.59:0/3589524325 <== mon.2 v2:172.21.15.59:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f0838016100 con 0x7f0834004810 2023-09-30T02:11:18.816 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.816+0000 7f083effd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:11:18.816 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.816+0000 7f083effd700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:11:18.816 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:11:18.816 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:11:18.820 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.818+0000 7f084d4eb700 10 librados: watch_flush enter 2023-09-30T02:11:18.820 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.818+0000 7f084d4eb700 10 librados: watch_flush exit 2023-09-30T02:11:18.820 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.818+0000 7f084d4eb700 20 client.4715.objecter shutdown clearing up homeless session... 2023-09-30T02:11:18.820 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.818+0000 7f084d4eb700 10 client.4715.objecter successfully canceled tick 2023-09-30T02:11:18.820 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.818+0000 7f084d4eb700 1 -- 172.21.15.59:0/3589524325 >> v2:172.21.15.110:6800/115870 conn(0x7f0828068890 msgr2=0x7f082806ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:18.821 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.818+0000 7f084d4eb700 1 --2- 172.21.15.59:0/3589524325 >> v2:172.21.15.110:6800/115870 conn(0x7f0828068890 0x7f082806ad40 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f083000c2d0 tx=0x7f0830015000 comp rx=0 tx=0).stop 2023-09-30T02:11:18.821 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.818+0000 7f084d4eb700 10 monclient: shutdown 2023-09-30T02:11:18.821 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.818+0000 7f084d4eb700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:11:18.821 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.818+0000 7f084d4eb700 1 -- 172.21.15.59:0/3589524325 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0834004810 msgr2=0x7f0848139fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:18.821 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.818+0000 7f084d4eb700 1 --2- 172.21.15.59:0/3589524325 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0834004810 0x7f0848139fa0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f083800e960 tx=0x7f0838009060 comp rx=0 tx=0).stop 2023-09-30T02:11:18.821 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.818+0000 7f084d4eb700 1 -- 172.21.15.59:0/3589524325 shutdown_connections 2023-09-30T02:11:18.822 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.818+0000 7f084d4eb700 1 --2- 172.21.15.59:0/3589524325 >> v2:172.21.15.110:6800/115870 conn(0x7f0828068890 0x7f082806ad40 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:18.822 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.818+0000 7f084d4eb700 1 --2- 172.21.15.59:0/3589524325 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0848136c40 0x7f08481370b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:18.822 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.818+0000 7f084d4eb700 1 --2- 172.21.15.59:0/3589524325 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f0834004810 0x7f0848139fa0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:18.822 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.818+0000 7f084d4eb700 1 --2- 172.21.15.59:0/3589524325 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f084813dca0 0x7f084813e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:18.822 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.818+0000 7f084d4eb700 1 -- 172.21.15.59:0/3589524325 >> 172.21.15.59:0/3589524325 conn(0x7f08480631f0 msgr2=0x7f08480635c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:11:18.822 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.818+0000 7f084d4eb700 1 -- 172.21.15.59:0/3589524325 shutdown_connections 2023-09-30T02:11:18.823 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.818+0000 7f084d4eb700 1 -- 172.21.15.59:0/3589524325 wait complete. 2023-09-30T02:11:18.823 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:18.818+0000 7f084d4eb700 1 librados: shutdown 2023-09-30T02:11:21.832 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:11:21.962 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.961+0000 7fe81cb07700 10 monclient: get_monmap_and_config 2023-09-30T02:11:21.962 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.961+0000 7fe81cb07700 10 monclient: build_initial_monmap 2023-09-30T02:11:21.962 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.961+0000 7fe81cb07700 10 monclient: monmap: 2023-09-30T02:11:21.962 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:11:21.962 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:21.963 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:11:21.962993+0000 2023-09-30T02:11:21.963 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:11:21.962993+0000 2023-09-30T02:11:21.963 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:11:21.963 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:21.963 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:11:21.963 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:11:21.963 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:11:21.963 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:11:21.963 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:11:21.963 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:21.963 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.962+0000 7fe81cb07700 1 Processor -- start 2023-09-30T02:11:21.964 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.962+0000 7fe81cb07700 1 -- start start 2023-09-30T02:11:21.964 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.962+0000 7fe81cb07700 10 monclient: init 2023-09-30T02:11:21.965 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.963+0000 7fe81cb07700 10 monclient: _reopen_session rank -1 2023-09-30T02:11:21.965 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.963+0000 7fe81cb07700 10 monclient: _add_conns ranks=[0,3,1,2,4] 2023-09-30T02:11:21.965 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.963+0000 7fe81cb07700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe8180a5b70 addr v1:172.21.15.59:6789/0 2023-09-30T02:11:21.965 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.963+0000 7fe81cb07700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fe818142290 0x7fe8180b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:21.965 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.963+0000 7fe81cb07700 10 monclient(hunting): picked mon.noname-a con 0x7fe818142290 addr v2:172.21.15.59:3300/0 2023-09-30T02:11:21.965 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.963+0000 7fe81cb07700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe8180a5f40 addr v1:172.21.15.110:6789/0 2023-09-30T02:11:21.966 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.963+0000 7fe81cb07700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8180a75a0 con 0x7fe8180a5b70 2023-09-30T02:11:21.966 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.963+0000 7fe81cb07700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8180a6360 con 0x7fe8180a5f40 2023-09-30T02:11:21.966 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.963+0000 7fe81cb07700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:21.966 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.963+0000 7fe81cb07700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8180af3f0 con 0x7fe818142290 2023-09-30T02:11:21.966 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.963+0000 7fe81cb07700 10 monclient(hunting): _renew_subs 2023-09-30T02:11:21.966 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.963+0000 7fe81cb07700 10 monclient(hunting): authenticate will time out at 2127.266204s 2023-09-30T02:11:21.966 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.963+0000 7fe815d9b700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fe818142290 0x7fe8180b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:21.966 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.964+0000 7fe816d9d700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fe8180a5f40 0x7fe81813af30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.59:42960/0 (socket says 172.21.15.59:42960) 2023-09-30T02:11:21.966 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.964+0000 7fe816d9d700 1 -- 172.21.15.59:0/1158986137 learned_addr learned my addr 172.21.15.59:0/1158986137 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:11:21.966 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.964+0000 7fe815d9b700 10 monclient(hunting): get_auth_request con 0x7fe818142290 auth_method 0 2023-09-30T02:11:21.966 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.964+0000 7fe815d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:21.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.964+0000 7fe815d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:21.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.964+0000 7fe815d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:21.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.965+0000 7fe815d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:21.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.965+0000 7fe815d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:21.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.965+0000 7fe815d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:21.968 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.967+0000 7fe81559a700 1 -- 172.21.15.59:0/1158986137 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 164661935 0 0) 0x7fe8180a6360 con 0x7fe8180a5f40 2023-09-30T02:11:21.968 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.967+0000 7fe81559a700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:21.968 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.967+0000 7fe81559a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:21.968 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.967+0000 7fe81559a700 10 monclient(hunting): my global_id is 4834 2023-09-30T02:11:21.968 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.967+0000 7fe81559a700 1 -- 172.21.15.59:0/1158986137 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe800003570 con 0x7fe8180a5f40 2023-09-30T02:11:21.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.968+0000 7fe815d9b700 10 monclient(hunting): handle_auth_done global_id 4797 payload 274 2023-09-30T02:11:21.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.968+0000 7fe815d9b700 1 -- 172.21.15.59:0/1158986137 >> v1:172.21.15.110:6789/0 conn(0x7fe8180a5f40 legacy=0x7fe81813af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:21.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.968+0000 7fe815d9b700 1 -- 172.21.15.59:0/1158986137 >> v1:172.21.15.59:6789/0 conn(0x7fe8180a5b70 legacy=0x7fe818141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:21.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.968+0000 7fe815d9b700 10 monclient: _finish_hunting 0 2023-09-30T02:11:21.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.968+0000 7fe815d9b700 1 monclient: found mon.noname-a 2023-09-30T02:11:21.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.968+0000 7fe815d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:11:21.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.968+0000 7fe815d9b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.59:3300/0 2023-09-30T02:11:21.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.968+0000 7fe815d9b700 1 -- 172.21.15.59:0/1158986137 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe81813b640 con 0x7fe818142290 2023-09-30T02:11:21.971 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.971+0000 7fe815d9b700 1 --2- 172.21.15.59:0/1158986137 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe818142290 0x7fe8180b1500 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fe80c0098e0 tx=0x7fe80c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b9cc45833cb6fa7 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:21.981 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.978+0000 7fe81559a700 1 -- 172.21.15.59:0/1158986137 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe80c00b640 con 0x7fe818142290 2023-09-30T02:11:21.981 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.978+0000 7fe81559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:21.981 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.978+0000 7fe81559a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T02:11:21.981 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.978+0000 7fe81559a700 10 monclient: dump: 2023-09-30T02:11:21.981 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:21.981 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:21.981 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:21.981 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:21.981 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:21.981 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:21.982 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:21.982 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:21.982 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:21.982 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:21.982 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.978+0000 7fe81559a700 10 monclient: _finish_auth 0 2023-09-30T02:11:21.982 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.978+0000 7fe81559a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:11:21.982 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.978+0000 7fe81559a700 1 -- 172.21.15.59:0/1158986137 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe80c00b7a0 con 0x7fe818142290 2023-09-30T02:11:21.982 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.978+0000 7fe81559a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:11:21.982 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.978+0000 7fe81559a700 1 -- 172.21.15.59:0/1158986137 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe80c00bb40 con 0x7fe818142290 2023-09-30T02:11:21.982 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.978+0000 7fe81559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:21.982 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.978+0000 7fe81559a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T02:11:21.982 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.978+0000 7fe81559a700 10 monclient: dump: 2023-09-30T02:11:21.983 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:21.983 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:21.983 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:21.983 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:21.983 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:21.983 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:21.983 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:21.983 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:21.983 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:21.983 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:21.983 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.978+0000 7fe81cb07700 5 monclient: authenticate success, global_id 4797 2023-09-30T02:11:21.983 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.978+0000 7fe81cb07700 10 monclient: get_monmap_and_config success 2023-09-30T02:11:21.983 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.978+0000 7fe81cb07700 4 set_mon_vals no callback set 2023-09-30T02:11:21.984 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.979+0000 7fe81cb07700 10 monclient: shutdown 2023-09-30T02:11:21.984 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.979+0000 7fe81cb07700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:11:21.984 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.979+0000 7fe81cb07700 1 -- 172.21.15.59:0/1158986137 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe818142290 msgr2=0x7fe8180b1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:21.984 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.979+0000 7fe81cb07700 1 --2- 172.21.15.59:0/1158986137 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe818142290 0x7fe8180b1500 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fe80c0098e0 tx=0x7fe80c0091a0 comp rx=0 tx=0).stop 2023-09-30T02:11:21.984 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.979+0000 7fe81cb07700 1 -- 172.21.15.59:0/1158986137 shutdown_connections 2023-09-30T02:11:21.984 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.979+0000 7fe81cb07700 1 --2- 172.21.15.59:0/1158986137 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe818142290 0x7fe8180b1500 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:21.984 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.979+0000 7fe81cb07700 1 -- 172.21.15.59:0/1158986137 >> 172.21.15.59:0/1158986137 conn(0x7fe8180631f0 msgr2=0x7fe8180635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:11:21.984 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.979+0000 7fe81cb07700 1 -- 172.21.15.59:0/1158986137 shutdown_connections 2023-09-30T02:11:21.984 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.979+0000 7fe81cb07700 1 -- 172.21.15.59:0/1158986137 wait complete. 2023-09-30T02:11:21.984 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.980+0000 7fe81cb07700 10 monclient: build_initial_monmap 2023-09-30T02:11:21.984 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.980+0000 7fe81cb07700 10 monclient: monmap: 2023-09-30T02:11:21.984 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:11:21.985 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:11:21.985 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:11:21.985 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:11:21.985 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:11:21.985 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:11:21.985 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:11:21.985 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:11:21.985 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:11:21.985 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:21.985 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.980+0000 7fe81cb07700 1 librados: starting msgr at 2023-09-30T02:11:21.985 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.980+0000 7fe81cb07700 1 librados: starting objecter 2023-09-30T02:11:21.985 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.980+0000 7fe81cb07700 1 Processor -- start 2023-09-30T02:11:21.985 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.980+0000 7fe81cb07700 1 -- start start 2023-09-30T02:11:21.985 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.980+0000 7fe81cb07700 1 librados: setting wanted keys 2023-09-30T02:11:21.986 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.980+0000 7fe81cb07700 1 librados: calling monclient init 2023-09-30T02:11:21.986 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.980+0000 7fe81cb07700 10 monclient: init 2023-09-30T02:11:21.987 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.981+0000 7fe81cb07700 10 monclient: _reopen_session rank -1 2023-09-30T02:11:21.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.981+0000 7fe81cb07700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T02:11:21.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.981+0000 7fe81cb07700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe8180a5b70 0x7fe818140ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:21.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.981+0000 7fe81cb07700 10 monclient(hunting): picked mon.noname-c con 0x7fe8180a5b70 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:11:21.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.981+0000 7fe81cb07700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8180a5f40 0x7fe81813e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:21.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.981+0000 7fe81cb07700 10 monclient(hunting): picked mon.noname-a con 0x7fe8180a5f40 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:11:21.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.981+0000 7fe81cb07700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe818142290 0x7fe81814de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:21.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.981+0000 7fe81cb07700 10 monclient(hunting): picked mon.noname-b con 0x7fe818142290 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:11:21.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.981+0000 7fe81cb07700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:21.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.981+0000 7fe81cb07700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe81813f100 con 0x7fe8180a5f40 2023-09-30T02:11:21.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.981+0000 7fe81cb07700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:21.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.981+0000 7fe81cb07700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe81813eca0 con 0x7fe818142290 2023-09-30T02:11:21.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.981+0000 7fe81cb07700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:21.989 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.981+0000 7fe81cb07700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe81813ee10 con 0x7fe8180a5b70 2023-09-30T02:11:21.989 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.981+0000 7fe81cb07700 10 monclient(hunting): _renew_subs 2023-09-30T02:11:21.989 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.981+0000 7fe81cb07700 10 monclient(hunting): authenticate will time out at 2127.284564s 2023-09-30T02:11:21.989 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.982+0000 7fe816d9d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe818142290 0x7fe81814de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:21.989 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.982+0000 7fe81659c700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe8180a5b70 0x7fe818140ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:21.989 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.982+0000 7fe815d9b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8180a5f40 0x7fe81813e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:21.989 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.982+0000 7fe816d9d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe818142290 0x7fe81814de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:48362/0 (socket says 172.21.15.59:48362) 2023-09-30T02:11:21.989 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.982+0000 7fe816d9d700 1 -- 172.21.15.59:0/1480551062 learned_addr learned my addr 172.21.15.59:0/1480551062 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:11:21.990 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.982+0000 7fe816d9d700 10 monclient(hunting): get_auth_request con 0x7fe818142290 auth_method 0 2023-09-30T02:11:21.990 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.982+0000 7fe816d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:21.990 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.982+0000 7fe816d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:21.990 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.982+0000 7fe816d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:21.990 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.982+0000 7fe815d9b700 10 monclient(hunting): get_auth_request con 0x7fe8180a5f40 auth_method 0 2023-09-30T02:11:21.990 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.982+0000 7fe815d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:21.990 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.982+0000 7fe815d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:21.990 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.982+0000 7fe815d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:21.990 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.982+0000 7fe81659c700 10 monclient(hunting): get_auth_request con 0x7fe8180a5b70 auth_method 0 2023-09-30T02:11:21.990 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.982+0000 7fe81659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:21.990 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.982+0000 7fe81659c700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:21.990 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.982+0000 7fe81659c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:21.991 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.983+0000 7fe816d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:21.991 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.983+0000 7fe816d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:21.991 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.983+0000 7fe816d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:21.991 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.983+0000 7fe815d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:21.991 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.983+0000 7fe815d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:21.991 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.983+0000 7fe815d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:21.991 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.983+0000 7fe81659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:21.991 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.983+0000 7fe81659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:21.991 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.983+0000 7fe81659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:21.992 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.988+0000 7fe816d9d700 10 monclient(hunting): handle_auth_done global_id 4837 payload 824 2023-09-30T02:11:21.992 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.988+0000 7fe816d9d700 1 -- 172.21.15.59:0/1480551062 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe8180a5b70 msgr2=0x7fe818140ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:21.992 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.988+0000 7fe816d9d700 1 --2- 172.21.15.59:0/1480551062 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe8180a5b70 0x7fe818140ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:21.992 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.988+0000 7fe816d9d700 1 -- 172.21.15.59:0/1480551062 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8180a5f40 msgr2=0x7fe81813e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:21.992 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.988+0000 7fe816d9d700 1 --2- 172.21.15.59:0/1480551062 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8180a5f40 0x7fe81813e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:21.992 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.988+0000 7fe816d9d700 10 monclient: _finish_hunting 0 2023-09-30T02:11:21.992 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.988+0000 7fe816d9d700 1 monclient: found mon.noname-b 2023-09-30T02:11:21.992 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.988+0000 7fe816d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:11:21.993 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.988+0000 7fe816d9d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:11:21.993 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.988+0000 7fe816d9d700 1 -- 172.21.15.59:0/1480551062 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe80c009420 con 0x7fe818142290 2023-09-30T02:11:21.993 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.993+0000 7fe816d9d700 1 --2- 172.21.15.59:0/1480551062 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe818142290 0x7fe81814de00 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fe810001460 tx=0x7fe810001ae0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:21.993 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.993+0000 7fe8067fc700 10 client.?.objecter ms_handle_connect 0x7fe818142290 2023-09-30T02:11:21.993 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.993+0000 7fe8067fc700 10 client.?.objecter resend_mon_ops 2023-09-30T02:11:21.998 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.997+0000 7fe8067fc700 1 -- 172.21.15.59:0/1480551062 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe81000a890 con 0x7fe818142290 2023-09-30T02:11:21.998 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.997+0000 7fe8067fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:21.998 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.997+0000 7fe8067fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:11:21.998 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.997+0000 7fe8067fc700 10 monclient: dump: 2023-09-30T02:11:21.998 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:21.998 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:21.998 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:21.998 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:21.998 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:21.998 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:21.998 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:21.999 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:21.999 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:21.999 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:21.999 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.997+0000 7fe8067fc700 10 monclient: _finish_auth 0 2023-09-30T02:11:21.999 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.997+0000 7fe8067fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:11:21.999 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.997+0000 7fe81cb07700 5 monclient: authenticate success, global_id 4837 2023-09-30T02:11:21.999 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.997+0000 7fe81cb07700 10 monclient: _renew_subs 2023-09-30T02:11:21.999 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.997+0000 7fe81cb07700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:11:21.999 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.997+0000 7fe81cb07700 1 -- 172.21.15.59:0/1480551062 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe81813e350 con 0x7fe818142290 2023-09-30T02:11:21.999 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.997+0000 7fe81cb07700 10 client.4837.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:11:21.999 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.997+0000 7fe81cb07700 10 monclient: _renew_subs 2023-09-30T02:11:21.999 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.997+0000 7fe81cb07700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:11:21.999 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.997+0000 7fe81cb07700 1 -- 172.21.15.59:0/1480551062 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe81814e5b0 con 0x7fe818142290 2023-09-30T02:11:22.000 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.997+0000 7fe81cb07700 1 librados: init done 2023-09-30T02:11:22.000 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.998+0000 7fe81cb07700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:11:22.000 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.998+0000 7fe81cb07700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:11:22.000 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.998+0000 7fe81cb07700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:11:22.000 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:21.998+0000 7fe81cb07700 1 -- 172.21.15.59:0/1480551062 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe818064a00 con 0x7fe818142290 2023-09-30T02:11:22.211 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:22.210+0000 7fe8067fc700 1 -- 172.21.15.59:0/1480551062 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe81000ac00 con 0x7fe818142290 2023-09-30T02:11:22.211 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:22.210+0000 7fe8067fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:11:22.211 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:22.210+0000 7fe814d99700 4 set_mon_vals no callback set 2023-09-30T02:11:22.248 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:22.248+0000 7fe8067fc700 1 -- 172.21.15.59:0/1480551062 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe81000ad70 con 0x7fe818142290 2023-09-30T02:11:22.248 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:22.248+0000 7fe8067fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:22.248 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:22.248+0000 7fe8067fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:11:22.248 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:22.248+0000 7fe8067fc700 10 monclient: dump: 2023-09-30T02:11:22.248 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:22.248 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:22.248 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:22.249 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:22.249 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:22.249 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:22.249 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:22.249 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:22.249 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:22.249 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:22.250 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:22.248+0000 7fe8067fc700 1 -- 172.21.15.59:0/1480551062 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fe810025430 con 0x7fe818142290 2023-09-30T02:11:22.250 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:22.248+0000 7fe8067fc700 1 --2- 172.21.15.59:0/1480551062 >> v2:172.21.15.110:6800/115870 conn(0x7fe7f8068890 0x7fe7f806ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:22.250 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:22.249+0000 7fe81659c700 1 --2- 172.21.15.59:0/1480551062 >> v2:172.21.15.110:6800/115870 conn(0x7fe7f8068890 0x7fe7f806ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:22.250 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:22.249+0000 7fe81659c700 10 monclient: get_auth_request con 0x7fe7f8068890 auth_method 0 2023-09-30T02:11:22.250 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:22.249+0000 7fe81659c700 1 --2- 172.21.15.59:0/1480551062 >> v2:172.21.15.110:6800/115870 conn(0x7fe7f8068890 0x7fe7f806ad40 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe808003aa0 tx=0x7fe808000bc0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:22.250 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:22.249+0000 7fe8067fc700 10 client.4837.objecter ms_handle_connect 0x7fe7f8068890 2023-09-30T02:11:22.253 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:22.252+0000 7fe8067fc700 1 -- 172.21.15.59:0/1480551062 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fe810056800 con 0x7fe818142290 2023-09-30T02:11:22.253 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:22.252+0000 7fe8067fc700 10 client.4837.objecter ms_dispatch 0x7fe818001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:11:22.253 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:22.253+0000 7fe8067fc700 3 client.4837.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:11:22.253 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:22.253+0000 7fe8067fc700 3 client.4837.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:11:22.253 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:22.253+0000 7fe8067fc700 20 client.4837.objecter dump_active .. 0 homeless 2023-09-30T02:11:22.263 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:22.263+0000 7fe8067fc700 1 -- 172.21.15.59:0/1480551062 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe810052440 con 0x7fe818142290 2023-09-30T02:11:22.263 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:22.263+0000 7fe8067fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:11:22.263 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:22.263+0000 7fe8067fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:11:22.512 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:22.511+0000 7fe81cb07700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:11:22.512 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:22.511+0000 7fe81cb07700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:11:22.512 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:22.511+0000 7fe81cb07700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:11:22.512 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:22.511+0000 7fe81cb07700 1 -- 172.21.15.59:0/1480551062 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe81806a720 con 0x7fe818142290 2023-09-30T02:11:22.517 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:22.518+0000 7fe8067fc700 1 -- 172.21.15.59:0/1480551062 <== mon.1 v2:172.21.15.110:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fe810055a60 con 0x7fe818142290 2023-09-30T02:11:22.517 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:22.518+0000 7fe8067fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:11:22.518 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:22.518+0000 7fe8067fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:11:22.518 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:11:22.518 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:11:22.520 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:22.519+0000 7fe81cb07700 10 librados: watch_flush enter 2023-09-30T02:11:22.520 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:22.519+0000 7fe81cb07700 10 librados: watch_flush exit 2023-09-30T02:11:22.520 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:22.519+0000 7fe81cb07700 20 client.4837.objecter shutdown clearing up homeless session... 2023-09-30T02:11:22.520 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:22.519+0000 7fe81cb07700 10 client.4837.objecter successfully canceled tick 2023-09-30T02:11:22.520 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:22.519+0000 7fe81cb07700 1 -- 172.21.15.59:0/1480551062 >> v2:172.21.15.110:6800/115870 conn(0x7fe7f8068890 msgr2=0x7fe7f806ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:22.521 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:22.519+0000 7fe81cb07700 1 --2- 172.21.15.59:0/1480551062 >> v2:172.21.15.110:6800/115870 conn(0x7fe7f8068890 0x7fe7f806ad40 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe808003aa0 tx=0x7fe808000bc0 comp rx=0 tx=0).stop 2023-09-30T02:11:22.521 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:22.519+0000 7fe81cb07700 10 monclient: shutdown 2023-09-30T02:11:22.521 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:22.519+0000 7fe81cb07700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:11:22.521 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:22.519+0000 7fe81cb07700 1 -- 172.21.15.59:0/1480551062 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe818142290 msgr2=0x7fe81814de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:22.521 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:22.519+0000 7fe81cb07700 1 --2- 172.21.15.59:0/1480551062 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe818142290 0x7fe81814de00 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fe810001460 tx=0x7fe810001ae0 comp rx=0 tx=0).stop 2023-09-30T02:11:22.521 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:22.519+0000 7fe81cb07700 1 -- 172.21.15.59:0/1480551062 shutdown_connections 2023-09-30T02:11:22.521 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:22.519+0000 7fe81cb07700 1 --2- 172.21.15.59:0/1480551062 >> v2:172.21.15.110:6800/115870 conn(0x7fe7f8068890 0x7fe7f806ad40 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:22.521 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:22.519+0000 7fe81cb07700 1 --2- 172.21.15.59:0/1480551062 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe818142290 0x7fe81814de00 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:22.521 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:22.519+0000 7fe81cb07700 1 --2- 172.21.15.59:0/1480551062 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe8180a5b70 0x7fe818140ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:22.521 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:22.519+0000 7fe81cb07700 1 --2- 172.21.15.59:0/1480551062 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8180a5f40 0x7fe81813e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:22.521 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:22.519+0000 7fe81cb07700 1 -- 172.21.15.59:0/1480551062 >> 172.21.15.59:0/1480551062 conn(0x7fe8180631f0 msgr2=0x7fe818070f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:11:22.521 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:22.519+0000 7fe81cb07700 1 -- 172.21.15.59:0/1480551062 shutdown_connections 2023-09-30T02:11:22.522 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:22.519+0000 7fe81cb07700 1 -- 172.21.15.59:0/1480551062 wait complete. 2023-09-30T02:11:22.522 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:22.519+0000 7fe81cb07700 1 librados: shutdown 2023-09-30T02:11:25.535 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:11:25.674 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.673+0000 7f364714c700 10 monclient: get_monmap_and_config 2023-09-30T02:11:25.674 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.673+0000 7f364714c700 10 monclient: build_initial_monmap 2023-09-30T02:11:25.674 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.673+0000 7f364714c700 10 monclient: monmap: 2023-09-30T02:11:25.674 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:11:25.674 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:25.674 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:11:25.675185+0000 2023-09-30T02:11:25.674 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:11:25.675185+0000 2023-09-30T02:11:25.675 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:11:25.675 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:25.675 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:11:25.675 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:11:25.675 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:11:25.675 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:11:25.675 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:11:25.675 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:25.676 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.674+0000 7f364714c700 1 Processor -- start 2023-09-30T02:11:25.677 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.674+0000 7f364714c700 1 -- start start 2023-09-30T02:11:25.677 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.674+0000 7f364714c700 10 monclient: init 2023-09-30T02:11:25.677 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.675+0000 7f364714c700 10 monclient: _reopen_session rank -1 2023-09-30T02:11:25.677 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.675+0000 7f364714c700 10 monclient: _add_conns ranks=[2,1,0,3,4] 2023-09-30T02:11:25.677 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.675+0000 7f364714c700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f36400a5b70 0x7f36400b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:25.677 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.675+0000 7f364714c700 10 monclient(hunting): picked mon.noname-b con 0x7f36400a5b70 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:11:25.677 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.675+0000 7f364714c700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f36400a5f40 addr v1:172.21.15.110:6789/0 2023-09-30T02:11:25.677 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.675+0000 7f364714c700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3640142290 addr v1:172.21.15.59:6789/0 2023-09-30T02:11:25.677 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.675+0000 7f364714c700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36400a63b0 con 0x7f3640142290 2023-09-30T02:11:25.677 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.675+0000 7f364714c700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36400a75a0 con 0x7f36400a5f40 2023-09-30T02:11:25.677 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.675+0000 7f364714c700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:25.678 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.675+0000 7f364714c700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36401426d0 con 0x7f36400a5b70 2023-09-30T02:11:25.678 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.675+0000 7f364714c700 10 monclient(hunting): _renew_subs 2023-09-30T02:11:25.678 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.675+0000 7f364714c700 10 monclient(hunting): authenticate will time out at 2130.978399s 2023-09-30T02:11:25.678 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.676+0000 7f3644ee8700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f36400a5b70 0x7f36400b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:25.678 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.676+0000 7f36456e9700 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f3640142290 0x7f364013af30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:51946/0 (socket says 172.21.15.59:51946) 2023-09-30T02:11:25.678 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.676+0000 7f36456e9700 1 -- 172.21.15.59:0/4234394407 learned_addr learned my addr 172.21.15.59:0/4234394407 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:11:25.678 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.676+0000 7f3644ee8700 10 monclient(hunting): get_auth_request con 0x7f36400a5b70 auth_method 0 2023-09-30T02:11:25.678 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.676+0000 7f3644ee8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:25.678 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.676+0000 7f3644ee8700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:25.678 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.676+0000 7f3644ee8700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:25.679 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.677+0000 7f3644ee8700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:25.681 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.677+0000 7f3644ee8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:25.681 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.677+0000 7f3644ee8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:25.682 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.679+0000 7f363f7fe700 1 -- 172.21.15.59:0/4234394407 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 919618261 0 0) 0x7f36400a75a0 con 0x7f36400a5f40 2023-09-30T02:11:25.682 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.679+0000 7f363f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:25.682 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.679+0000 7f363f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:25.682 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.679+0000 7f363f7fe700 10 monclient(hunting): my global_id is 4840 2023-09-30T02:11:25.682 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.679+0000 7f363f7fe700 1 -- 172.21.15.59:0/4234394407 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f362c003570 con 0x7f36400a5f40 2023-09-30T02:11:25.682 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.679+0000 7f363f7fe700 1 -- 172.21.15.59:0/4234394407 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2696037532 0 0) 0x7f36400a63b0 con 0x7f3640142290 2023-09-30T02:11:25.683 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.679+0000 7f363f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:25.683 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.679+0000 7f363f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:25.683 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.679+0000 7f363f7fe700 10 monclient(hunting): my global_id is 4806 2023-09-30T02:11:25.683 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.679+0000 7f363f7fe700 1 -- 172.21.15.59:0/4234394407 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36400a75a0 con 0x7f3640142290 2023-09-30T02:11:25.683 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.680+0000 7f3644ee8700 10 monclient(hunting): handle_auth_done global_id 4721 payload 274 2023-09-30T02:11:25.683 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.680+0000 7f3644ee8700 1 -- 172.21.15.59:0/4234394407 >> v1:172.21.15.110:6789/0 conn(0x7f36400a5f40 legacy=0x7f3640141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:25.683 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.680+0000 7f3644ee8700 1 -- 172.21.15.59:0/4234394407 >> v1:172.21.15.59:6789/0 conn(0x7f3640142290 legacy=0x7f364013af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:25.683 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.680+0000 7f3644ee8700 10 monclient: _finish_hunting 0 2023-09-30T02:11:25.683 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.680+0000 7f3644ee8700 1 monclient: found mon.noname-b 2023-09-30T02:11:25.683 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.680+0000 7f3644ee8700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:11:25.683 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.680+0000 7f3644ee8700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.59:3301/0 2023-09-30T02:11:25.683 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.680+0000 7f3644ee8700 1 -- 172.21.15.59:0/4234394407 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36400af3f0 con 0x7f36400a5b70 2023-09-30T02:11:25.684 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.682+0000 7f3644ee8700 1 --2- 172.21.15.59:0/4234394407 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f36400a5b70 0x7f36400b1500 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f3630009910 tx=0x7f36300091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b0383cbbb75eabe server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:25.688 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.687+0000 7f363f7fe700 1 -- 172.21.15.59:0/4234394407 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3630011630 con 0x7f36400a5b70 2023-09-30T02:11:25.688 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.687+0000 7f363f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:25.688 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.687+0000 7f363f7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:11:25.688 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.687+0000 7f363f7fe700 10 monclient: dump: 2023-09-30T02:11:25.688 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:25.688 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:25.689 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:25.689 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:25.689 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:25.689 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:25.689 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:25.689 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:25.689 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:25.689 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:25.689 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.687+0000 7f363f7fe700 10 monclient: _finish_auth 0 2023-09-30T02:11:25.689 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.687+0000 7f363f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:11:25.689 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.687+0000 7f363f7fe700 1 -- 172.21.15.59:0/4234394407 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3630011790 con 0x7f36400a5b70 2023-09-30T02:11:25.689 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.687+0000 7f364714c700 5 monclient: authenticate success, global_id 4721 2023-09-30T02:11:25.690 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.687+0000 7f364714c700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:11:25.690 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.687+0000 7f363f7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:11:25.690 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.687+0000 7f364714c700 10 monclient: get_monmap_and_config success 2023-09-30T02:11:25.690 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.687+0000 7f364714c700 4 set_mon_vals no callback set 2023-09-30T02:11:25.690 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.687+0000 7f363f7fe700 1 -- 172.21.15.59:0/4234394407 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3630011b30 con 0x7f36400a5b70 2023-09-30T02:11:25.690 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.687+0000 7f363f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:25.690 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.687+0000 7f363f7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:11:25.690 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.687+0000 7f363f7fe700 10 monclient: dump: 2023-09-30T02:11:25.690 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:25.690 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:25.690 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:25.691 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:25.691 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:25.691 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:25.691 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:25.691 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:25.691 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:25.691 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:25.691 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.687+0000 7f364714c700 10 monclient: shutdown 2023-09-30T02:11:25.691 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.687+0000 7f364714c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:11:25.691 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.687+0000 7f364714c700 1 -- 172.21.15.59:0/4234394407 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f36400a5b70 msgr2=0x7f36400b1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:25.691 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.687+0000 7f364714c700 1 --2- 172.21.15.59:0/4234394407 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f36400a5b70 0x7f36400b1500 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f3630009910 tx=0x7f36300091a0 comp rx=0 tx=0).stop 2023-09-30T02:11:25.691 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.688+0000 7f364714c700 1 -- 172.21.15.59:0/4234394407 shutdown_connections 2023-09-30T02:11:25.691 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.688+0000 7f364714c700 1 --2- 172.21.15.59:0/4234394407 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f36400a5b70 0x7f36400b1500 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:25.692 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.688+0000 7f364714c700 1 -- 172.21.15.59:0/4234394407 >> 172.21.15.59:0/4234394407 conn(0x7f36400631f0 msgr2=0x7f36400635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:11:25.692 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.688+0000 7f364714c700 1 -- 172.21.15.59:0/4234394407 shutdown_connections 2023-09-30T02:11:25.692 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.688+0000 7f364714c700 1 -- 172.21.15.59:0/4234394407 wait complete. 2023-09-30T02:11:25.694 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.689+0000 7f364714c700 10 monclient: build_initial_monmap 2023-09-30T02:11:25.695 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.689+0000 7f364714c700 10 monclient: monmap: 2023-09-30T02:11:25.695 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:11:25.695 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:11:25.695 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:11:25.695 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:11:25.695 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:11:25.695 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:11:25.695 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:11:25.695 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:11:25.695 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:11:25.696 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:25.696 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.689+0000 7f364714c700 1 librados: starting msgr at 2023-09-30T02:11:25.696 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.689+0000 7f364714c700 1 librados: starting objecter 2023-09-30T02:11:25.696 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.689+0000 7f364714c700 1 Processor -- start 2023-09-30T02:11:25.696 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.689+0000 7f364714c700 1 -- start start 2023-09-30T02:11:25.696 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.689+0000 7f364714c700 1 librados: setting wanted keys 2023-09-30T02:11:25.696 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.689+0000 7f364714c700 1 librados: calling monclient init 2023-09-30T02:11:25.696 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.689+0000 7f364714c700 10 monclient: init 2023-09-30T02:11:25.696 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.690+0000 7f364714c700 10 monclient: _reopen_session rank -1 2023-09-30T02:11:25.696 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.690+0000 7f364714c700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T02:11:25.696 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.690+0000 7f364714c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36400a5b70 0x7f3640140a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:25.696 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.690+0000 7f364714c700 10 monclient(hunting): picked mon.noname-a con 0x7f36400a5b70 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:11:25.697 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.690+0000 7f364714c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f36400a5f40 0x7f364013eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:25.697 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.690+0000 7f364714c700 10 monclient(hunting): picked mon.noname-b con 0x7f36400a5f40 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:11:25.697 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.690+0000 7f364714c700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3640142290 0x7f364014d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:25.697 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.690+0000 7f364714c700 10 monclient(hunting): picked mon.noname-c con 0x7f3640142290 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:11:25.697 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.690+0000 7f364714c700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:25.697 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.690+0000 7f364714c700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f364013e080 con 0x7f36400a5b70 2023-09-30T02:11:25.697 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.690+0000 7f364714c700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:25.697 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.690+0000 7f364714c700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36400a8930 con 0x7f36400a5f40 2023-09-30T02:11:25.697 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.690+0000 7f364714c700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:25.697 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.690+0000 7f364714c700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f364013db60 con 0x7f3640142290 2023-09-30T02:11:25.697 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.690+0000 7f364714c700 10 monclient(hunting): _renew_subs 2023-09-30T02:11:25.697 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.690+0000 7f364714c700 10 monclient(hunting): authenticate will time out at 2130.993525s 2023-09-30T02:11:25.698 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.691+0000 7f36456e9700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3640142290 0x7f364014d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:25.698 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.691+0000 7f3644ee8700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36400a5b70 0x7f3640140a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:25.698 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.691+0000 7f363ffff700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f36400a5f40 0x7f364013eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:25.698 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.691+0000 7f36456e9700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3640142290 0x7f364014d8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:41712/0 (socket says 172.21.15.59:41712) 2023-09-30T02:11:25.698 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.691+0000 7f36456e9700 1 -- 172.21.15.59:0/2533301364 learned_addr learned my addr 172.21.15.59:0/2533301364 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:11:25.698 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.691+0000 7f36456e9700 10 monclient(hunting): get_auth_request con 0x7f3640142290 auth_method 0 2023-09-30T02:11:25.698 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.691+0000 7f36456e9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:25.698 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.691+0000 7f36456e9700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:25.698 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.691+0000 7f36456e9700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:25.698 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.691+0000 7f3644ee8700 10 monclient(hunting): get_auth_request con 0x7f36400a5b70 auth_method 0 2023-09-30T02:11:25.698 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.691+0000 7f3644ee8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:25.698 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.691+0000 7f3644ee8700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:25.699 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.691+0000 7f3644ee8700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:25.699 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.691+0000 7f363ffff700 10 monclient(hunting): get_auth_request con 0x7f36400a5f40 auth_method 0 2023-09-30T02:11:25.699 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.691+0000 7f363ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:25.699 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.691+0000 7f363ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:25.699 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.691+0000 7f363ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:25.699 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.692+0000 7f36456e9700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:25.699 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.692+0000 7f36456e9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:25.699 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.692+0000 7f36456e9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:25.699 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.692+0000 7f3644ee8700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:25.699 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.692+0000 7f3644ee8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:25.699 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.692+0000 7f3644ee8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:25.699 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.692+0000 7f363ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:25.700 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.692+0000 7f363ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:25.700 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.692+0000 7f363ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:25.701 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.695+0000 7f36456e9700 10 monclient(hunting): handle_auth_done global_id 4724 payload 824 2023-09-30T02:11:25.701 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.695+0000 7f36456e9700 1 -- 172.21.15.59:0/2533301364 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f36400a5f40 msgr2=0x7f364013eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:25.701 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.695+0000 7f36456e9700 1 --2- 172.21.15.59:0/2533301364 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f36400a5f40 0x7f364013eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:25.701 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.695+0000 7f36456e9700 1 -- 172.21.15.59:0/2533301364 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36400a5b70 msgr2=0x7f3640140a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:25.701 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.695+0000 7f36456e9700 1 --2- 172.21.15.59:0/2533301364 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36400a5b70 0x7f3640140a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:25.701 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.695+0000 7f36456e9700 10 monclient: _finish_hunting 0 2023-09-30T02:11:25.701 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.695+0000 7f36456e9700 1 monclient: found mon.noname-c 2023-09-30T02:11:25.701 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.695+0000 7f36456e9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:11:25.701 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.695+0000 7f36456e9700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.59:3301/0 2023-09-30T02:11:25.701 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.695+0000 7f36456e9700 1 -- 172.21.15.59:0/2533301364 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3630009420 con 0x7f3640142290 2023-09-30T02:11:25.701 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.698+0000 7f36456e9700 1 --2- 172.21.15.59:0/2533301364 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3640142290 0x7f364014d8a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3638001460 tx=0x7f3638001a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:25.702 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.698+0000 7f363cff9700 10 client.?.objecter ms_handle_connect 0x7f3640142290 2023-09-30T02:11:25.702 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.698+0000 7f363cff9700 10 client.?.objecter resend_mon_ops 2023-09-30T02:11:25.703 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.702+0000 7f363cff9700 1 -- 172.21.15.59:0/2533301364 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f363800a870 con 0x7f3640142290 2023-09-30T02:11:25.703 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.702+0000 7f363cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:25.703 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.702+0000 7f363cff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:11:25.703 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.702+0000 7f363cff9700 10 monclient: dump: 2023-09-30T02:11:25.703 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:25.703 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:25.704 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:25.704 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:25.704 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:25.704 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:25.704 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:25.704 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:25.704 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:25.704 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:25.704 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.702+0000 7f363cff9700 10 monclient: _finish_auth 0 2023-09-30T02:11:25.704 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.702+0000 7f363cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:11:25.704 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.702+0000 7f363cff9700 1 -- 172.21.15.59:0/2533301364 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f363800a9d0 con 0x7f3640142290 2023-09-30T02:11:25.704 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.702+0000 7f364714c700 5 monclient: authenticate success, global_id 4724 2023-09-30T02:11:25.705 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.702+0000 7f364714c700 10 monclient: _renew_subs 2023-09-30T02:11:25.705 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.702+0000 7f364714c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:11:25.705 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.702+0000 7f364714c700 1 -- 172.21.15.59:0/2533301364 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f364014dde0 con 0x7f3640142290 2023-09-30T02:11:25.705 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.702+0000 7f363cff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:11:25.705 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.702+0000 7f363cff9700 1 -- 172.21.15.59:0/2533301364 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f363800ad50 con 0x7f3640142290 2023-09-30T02:11:25.705 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.702+0000 7f363cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:25.705 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.702+0000 7f363cff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:11:25.705 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.702+0000 7f363cff9700 10 monclient: dump: 2023-09-30T02:11:25.705 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:25.705 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:25.705 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:25.705 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:25.706 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:25.706 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:25.706 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:25.706 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:25.706 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:25.706 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:25.706 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.702+0000 7f363effd700 4 set_mon_vals no callback set 2023-09-30T02:11:25.706 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.702+0000 7f364714c700 10 client.4724.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:11:25.706 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.702+0000 7f364714c700 10 monclient: _renew_subs 2023-09-30T02:11:25.706 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.702+0000 7f364714c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:11:25.706 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.702+0000 7f364714c700 1 -- 172.21.15.59:0/2533301364 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f364014e2d0 con 0x7f3640142290 2023-09-30T02:11:25.706 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.702+0000 7f364714c700 1 librados: init done 2023-09-30T02:11:25.707 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.703+0000 7f364714c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:11:25.707 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.703+0000 7f364714c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:11:25.707 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.703+0000 7f364714c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:11:25.707 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.703+0000 7f364714c700 1 -- 172.21.15.59:0/2533301364 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f364013e230 con 0x7f3640142290 2023-09-30T02:11:25.929 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.929+0000 7f363cff9700 1 -- 172.21.15.59:0/2533301364 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f3638007300 con 0x7f3640142290 2023-09-30T02:11:25.929 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.929+0000 7f363cff9700 1 --2- 172.21.15.59:0/2533301364 >> v2:172.21.15.110:6800/115870 conn(0x7f3620068960 0x7f362006ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:25.930 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.930+0000 7f3644ee8700 1 --2- 172.21.15.59:0/2533301364 >> v2:172.21.15.110:6800/115870 conn(0x7f3620068960 0x7f362006ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:25.930 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.930+0000 7f3644ee8700 10 monclient: get_auth_request con 0x7f3620068960 auth_method 0 2023-09-30T02:11:25.930 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.930+0000 7f363cff9700 10 client.4724.objecter ms_handle_connect 0x7f3620068960 2023-09-30T02:11:25.930 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:25.930+0000 7f3644ee8700 1 --2- 172.21.15.59:0/2533301364 >> v2:172.21.15.110:6800/115870 conn(0x7f3620068960 0x7f362006ae10 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f36300093f0 tx=0x7f3630017000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:26.000 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:26.000+0000 7f363cff9700 1 -- 172.21.15.59:0/2533301364 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f3638056180 con 0x7f3640142290 2023-09-30T02:11:26.001 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:26.000+0000 7f363cff9700 10 client.4724.objecter ms_dispatch 0x7f3640001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:11:26.001 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:26.000+0000 7f363cff9700 3 client.4724.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:11:26.001 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:26.000+0000 7f363cff9700 3 client.4724.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:11:26.001 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:26.000+0000 7f363cff9700 20 client.4724.objecter dump_active .. 0 homeless 2023-09-30T02:11:26.001 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:26.000+0000 7f363cff9700 1 -- 172.21.15.59:0/2533301364 <== mon.2 v2:172.21.15.59:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f36380524c0 con 0x7f3640142290 2023-09-30T02:11:26.001 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:26.000+0000 7f363cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:11:26.001 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:26.000+0000 7f363cff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:11:26.285 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:26.285+0000 7f364714c700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:11:26.286 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:26.285+0000 7f364714c700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:11:26.286 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:26.285+0000 7f364714c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:11:26.286 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:26.285+0000 7f364714c700 1 -- 172.21.15.59:0/2533301364 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f364013e230 con 0x7f3640142290 2023-09-30T02:11:26.290 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:26.290+0000 7f363cff9700 1 -- 172.21.15.59:0/2533301364 <== mon.2 v2:172.21.15.59:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f364013e230 con 0x7f3640142290 2023-09-30T02:11:26.290 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:26.290+0000 7f363cff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:11:26.290 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:26.290+0000 7f363cff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:11:26.291 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:11:26.291 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:11:26.294 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:26.293+0000 7f364714c700 10 librados: watch_flush enter 2023-09-30T02:11:26.294 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:26.293+0000 7f364714c700 10 librados: watch_flush exit 2023-09-30T02:11:26.294 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:26.293+0000 7f364714c700 20 client.4724.objecter shutdown clearing up homeless session... 2023-09-30T02:11:26.294 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:26.293+0000 7f364714c700 10 client.4724.objecter successfully canceled tick 2023-09-30T02:11:26.294 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:26.293+0000 7f364714c700 1 -- 172.21.15.59:0/2533301364 >> v2:172.21.15.110:6800/115870 conn(0x7f3620068960 msgr2=0x7f362006ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:26.295 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:26.293+0000 7f364714c700 1 --2- 172.21.15.59:0/2533301364 >> v2:172.21.15.110:6800/115870 conn(0x7f3620068960 0x7f362006ae10 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f36300093f0 tx=0x7f3630017000 comp rx=0 tx=0).stop 2023-09-30T02:11:26.295 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:26.293+0000 7f364714c700 10 monclient: shutdown 2023-09-30T02:11:26.295 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:26.293+0000 7f364714c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:11:26.295 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:26.293+0000 7f364714c700 1 -- 172.21.15.59:0/2533301364 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3640142290 msgr2=0x7f364014d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:26.297 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:26.293+0000 7f364714c700 1 --2- 172.21.15.59:0/2533301364 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3640142290 0x7f364014d8a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3638001460 tx=0x7f3638001a40 comp rx=0 tx=0).stop 2023-09-30T02:11:26.297 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:26.293+0000 7f364714c700 1 -- 172.21.15.59:0/2533301364 shutdown_connections 2023-09-30T02:11:26.297 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:26.293+0000 7f364714c700 1 --2- 172.21.15.59:0/2533301364 >> v2:172.21.15.110:6800/115870 conn(0x7f3620068960 0x7f362006ae10 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:26.297 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:26.293+0000 7f364714c700 1 --2- 172.21.15.59:0/2533301364 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3640142290 0x7f364014d8a0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:26.297 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:26.293+0000 7f364714c700 1 --2- 172.21.15.59:0/2533301364 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36400a5b70 0x7f3640140a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:26.297 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:26.293+0000 7f364714c700 1 --2- 172.21.15.59:0/2533301364 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f36400a5f40 0x7f364013eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:26.297 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:26.293+0000 7f364714c700 1 -- 172.21.15.59:0/2533301364 >> 172.21.15.59:0/2533301364 conn(0x7f36400631f0 msgr2=0x7f36400635c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:11:26.298 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:26.294+0000 7f364714c700 1 -- 172.21.15.59:0/2533301364 shutdown_connections 2023-09-30T02:11:26.298 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:26.294+0000 7f364714c700 1 -- 172.21.15.59:0/2533301364 wait complete. 2023-09-30T02:11:26.298 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:26.294+0000 7f364714c700 1 librados: shutdown 2023-09-30T02:11:29.309 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T02:11:29.456 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.456+0000 7feceacae700 10 monclient: get_monmap_and_config 2023-09-30T02:11:29.456 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.456+0000 7feceacae700 10 monclient: build_initial_monmap 2023-09-30T02:11:29.456 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.456+0000 7feceacae700 10 monclient: monmap: 2023-09-30T02:11:29.456 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:11:29.456 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:29.456 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:11:29.456995+0000 2023-09-30T02:11:29.456 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:11:29.456995+0000 2023-09-30T02:11:29.456 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:11:29.456 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:29.456 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:11:29.457 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:11:29.457 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:11:29.457 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:11:29.457 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:11:29.457 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:29.459 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.456+0000 7feceacae700 1 Processor -- start 2023-09-30T02:11:29.459 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.456+0000 7feceacae700 1 -- start start 2023-09-30T02:11:29.459 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.456+0000 7feceacae700 10 monclient: init 2023-09-30T02:11:29.459 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.458+0000 7feceacae700 10 monclient: _reopen_session rank -1 2023-09-30T02:11:29.459 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.458+0000 7feceacae700 10 monclient: _add_conns ranks=[3,4,2,0,1] 2023-09-30T02:11:29.459 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.458+0000 7feceacae700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fece40b16e0 0x7fece40a54e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:29.460 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.458+0000 7feceacae700 10 monclient(hunting): picked mon.noname-a con 0x7fece40b16e0 addr v2:172.21.15.59:3300/0 2023-09-30T02:11:29.460 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.458+0000 7feceacae700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fece40a7650 0x7fece40a7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:29.460 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.458+0000 7feceacae700 10 monclient(hunting): picked mon.noname-c con 0x7fece40a7650 addr v2:172.21.15.110:3300/0 2023-09-30T02:11:29.460 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.458+0000 7feceacae700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fece40a7ff0 0x7fece40af440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:29.460 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.458+0000 7feceacae700 10 monclient(hunting): picked mon.noname-b con 0x7fece40a7ff0 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:11:29.460 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.458+0000 7feceacae700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:29.460 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.458+0000 7feceacae700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fece40a5d40 con 0x7fece40b16e0 2023-09-30T02:11:29.460 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.458+0000 7feceacae700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:29.460 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.458+0000 7feceacae700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fece40b1ab0 con 0x7fece40a7650 2023-09-30T02:11:29.460 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.458+0000 7feceacae700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:29.460 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.458+0000 7feceacae700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fece40a5a20 con 0x7fece40a7ff0 2023-09-30T02:11:29.460 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.458+0000 7feceacae700 10 monclient(hunting): _renew_subs 2023-09-30T02:11:29.460 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.458+0000 7feceacae700 10 monclient(hunting): authenticate will time out at 2134.760325s 2023-09-30T02:11:29.461 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.458+0000 7fece924b700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fece40a7ff0 0x7fece40af440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:29.461 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.458+0000 7fece3fff700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fece40a7650 0x7fece40a7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:29.461 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.458+0000 7fece924b700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fece40a7ff0 0x7fece40af440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:33656/0 (socket says 172.21.15.59:33656) 2023-09-30T02:11:29.461 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.458+0000 7fece924b700 1 -- 172.21.15.59:0/3013155631 learned_addr learned my addr 172.21.15.59:0/3013155631 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:11:29.461 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.458+0000 7fece924b700 10 monclient(hunting): get_auth_request con 0x7fece40a7ff0 auth_method 0 2023-09-30T02:11:29.461 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.458+0000 7fece924b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:29.461 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.458+0000 7fece924b700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:29.461 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.458+0000 7fece924b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:29.461 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.458+0000 7fece3fff700 10 monclient(hunting): get_auth_request con 0x7fece40a7650 auth_method 0 2023-09-30T02:11:29.461 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.458+0000 7fece3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:29.461 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.458+0000 7fece3fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:29.461 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.458+0000 7fece3fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:29.462 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.459+0000 7fece924b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:29.462 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.459+0000 7fece924b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:29.462 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.459+0000 7fece924b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:29.462 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.459+0000 7fece3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:29.462 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.459+0000 7fece3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:29.462 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.459+0000 7fece3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:29.462 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.461+0000 7fece3fff700 10 monclient(hunting): handle_auth_done global_id 4846 payload 274 2023-09-30T02:11:29.463 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.461+0000 7fece3fff700 1 -- 172.21.15.59:0/3013155631 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fece40a7ff0 msgr2=0x7fece40af440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:11:29.463 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.461+0000 7fece3fff700 1 --2- 172.21.15.59:0/3013155631 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fece40a7ff0 0x7fece40af440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:29.463 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.461+0000 7fece3fff700 1 -- 172.21.15.59:0/3013155631 >> v2:172.21.15.59:3300/0 conn(0x7fece40b16e0 msgr2=0x7fece40a54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:11:29.463 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.461+0000 7fece3fff700 1 --2- 172.21.15.59:0/3013155631 >> v2:172.21.15.59:3300/0 conn(0x7fece40b16e0 0x7fece40a54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:29.463 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.461+0000 7fece3fff700 10 monclient: _finish_hunting 0 2023-09-30T02:11:29.463 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.461+0000 7fece3fff700 1 monclient: found mon.noname-c 2023-09-30T02:11:29.463 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.461+0000 7fece3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:11:29.463 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.461+0000 7fece3fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3300/0 2023-09-30T02:11:29.463 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.461+0000 7fece3fff700 1 -- 172.21.15.59:0/3013155631 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fece40a6c70 con 0x7fece40a7650 2023-09-30T02:11:29.464 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.464+0000 7fece3fff700 1 --2- 172.21.15.59:0/3013155631 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fece40a7650 0x7fece40a7ab0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fece40abb10 tx=0x7fecd00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=39b5f2b149c9ba4e server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:29.468 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.469+0000 7fece37fe700 1 -- 172.21.15.59:0/3013155631 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecd0011630 con 0x7fece40a7650 2023-09-30T02:11:29.468 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.469+0000 7fece37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:29.468 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.469+0000 7fece37fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:11:29.468 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.469+0000 7fece37fe700 10 monclient: dump: 2023-09-30T02:11:29.468 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:29.469 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:29.469 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:29.469 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:29.469 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:29.469 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:29.469 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:29.469 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:29.469 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:29.469 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:29.469 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.469+0000 7fece37fe700 10 monclient: _finish_auth 0 2023-09-30T02:11:29.469 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.469+0000 7fece37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:11:29.469 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.469+0000 7fece37fe700 1 -- 172.21.15.59:0/3013155631 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecd0011790 con 0x7fece40a7650 2023-09-30T02:11:29.469 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.469+0000 7fece37fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:11:29.470 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.469+0000 7fece37fe700 1 -- 172.21.15.59:0/3013155631 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecd0011b30 con 0x7fece40a7650 2023-09-30T02:11:29.470 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.469+0000 7feceacae700 5 monclient: authenticate success, global_id 4846 2023-09-30T02:11:29.470 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.469+0000 7feceacae700 10 monclient: get_monmap_and_config success 2023-09-30T02:11:29.470 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.469+0000 7fece37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:29.470 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.469+0000 7fece37fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:11:29.470 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.469+0000 7feceacae700 4 set_mon_vals no callback set 2023-09-30T02:11:29.470 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.469+0000 7fece37fe700 10 monclient: dump: 2023-09-30T02:11:29.470 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:29.470 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:29.470 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:29.470 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:29.470 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:29.471 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:29.471 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:29.471 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:29.471 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:29.471 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:29.473 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.469+0000 7feceacae700 10 monclient: shutdown 2023-09-30T02:11:29.473 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.469+0000 7feceacae700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:11:29.473 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.469+0000 7feceacae700 1 -- 172.21.15.59:0/3013155631 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fece40a7650 msgr2=0x7fece40a7ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:29.473 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.469+0000 7feceacae700 1 --2- 172.21.15.59:0/3013155631 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fece40a7650 0x7fece40a7ab0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fece40abb10 tx=0x7fecd00091a0 comp rx=0 tx=0).stop 2023-09-30T02:11:29.473 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.469+0000 7feceacae700 1 -- 172.21.15.59:0/3013155631 shutdown_connections 2023-09-30T02:11:29.473 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.469+0000 7feceacae700 1 --2- 172.21.15.59:0/3013155631 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fece40a7ff0 0x7fece40af440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:29.473 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.469+0000 7feceacae700 1 --2- 172.21.15.59:0/3013155631 >> v2:172.21.15.59:3300/0 conn(0x7fece40b16e0 0x7fece40a54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:29.473 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.469+0000 7feceacae700 1 --2- 172.21.15.59:0/3013155631 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fece40a7650 0x7fece40a7ab0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:29.474 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.469+0000 7feceacae700 1 -- 172.21.15.59:0/3013155631 >> 172.21.15.59:0/3013155631 conn(0x7fece4068fe0 msgr2=0x7fece40631f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:11:29.474 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.469+0000 7feceacae700 1 -- 172.21.15.59:0/3013155631 shutdown_connections 2023-09-30T02:11:29.474 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.469+0000 7feceacae700 1 -- 172.21.15.59:0/3013155631 wait complete. 2023-09-30T02:11:29.474 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.469+0000 7feceacae700 10 monclient: build_initial_monmap 2023-09-30T02:11:29.474 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.470+0000 7feceacae700 10 monclient: monmap: 2023-09-30T02:11:29.474 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:11:29.474 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:11:29.474 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:11:29.474 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:11:29.474 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:11:29.474 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:11:29.474 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:11:29.475 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:11:29.475 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:11:29.475 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:29.475 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.470+0000 7feceacae700 1 librados: starting msgr at 2023-09-30T02:11:29.475 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.470+0000 7feceacae700 1 librados: starting objecter 2023-09-30T02:11:29.475 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.470+0000 7feceacae700 1 Processor -- start 2023-09-30T02:11:29.475 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.470+0000 7feceacae700 1 -- start start 2023-09-30T02:11:29.475 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.470+0000 7feceacae700 1 librados: setting wanted keys 2023-09-30T02:11:29.475 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.470+0000 7feceacae700 1 librados: calling monclient init 2023-09-30T02:11:29.475 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.470+0000 7feceacae700 10 monclient: init 2023-09-30T02:11:29.475 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.471+0000 7feceacae700 10 monclient: _reopen_session rank -1 2023-09-30T02:11:29.475 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.471+0000 7feceacae700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T02:11:29.476 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.471+0000 7feceacae700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fece40a7650 0x7fece40aa510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:29.476 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.471+0000 7feceacae700 10 monclient(hunting): picked mon.noname-b con 0x7fece40a7650 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:11:29.476 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.471+0000 7feceacae700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fece40a7ff0 0x7fece413e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:29.476 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.471+0000 7feceacae700 10 monclient(hunting): picked mon.noname-c con 0x7fece40a7ff0 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:11:29.476 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.471+0000 7feceacae700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fece40b16e0 0x7fece4141ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:29.476 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.471+0000 7feceacae700 10 monclient(hunting): picked mon.noname-a con 0x7fece40b16e0 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:11:29.476 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.471+0000 7feceacae700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:29.476 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.471+0000 7feceacae700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fece41415c0 con 0x7fece40b16e0 2023-09-30T02:11:29.476 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.471+0000 7feceacae700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:29.477 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.471+0000 7feceacae700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fece41413b0 con 0x7fece40a7650 2023-09-30T02:11:29.477 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.471+0000 7feceacae700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:29.477 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.471+0000 7feceacae700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fece40a93a0 con 0x7fece40a7ff0 2023-09-30T02:11:29.477 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.471+0000 7feceacae700 10 monclient(hunting): _renew_subs 2023-09-30T02:11:29.477 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.471+0000 7feceacae700 10 monclient(hunting): authenticate will time out at 2134.774666s 2023-09-30T02:11:29.477 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.472+0000 7fece3fff700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fece40a7ff0 0x7fece413e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:29.477 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.472+0000 7fece3fff700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fece40a7ff0 0x7fece413e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:33658/0 (socket says 172.21.15.59:33658) 2023-09-30T02:11:29.477 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.472+0000 7fece3fff700 1 -- 172.21.15.59:0/3928779313 learned_addr learned my addr 172.21.15.59:0/3928779313 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:11:29.477 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.472+0000 7fece3fff700 10 monclient(hunting): get_auth_request con 0x7fece40a7ff0 auth_method 0 2023-09-30T02:11:29.477 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.472+0000 7fece3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:29.478 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.472+0000 7fece3fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:29.478 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.472+0000 7fece3fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:29.478 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.472+0000 7fece8a4a700 1 --2- 172.21.15.59:0/3928779313 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fece40a7650 0x7fece40aa510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:29.478 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.474+0000 7fece8a4a700 10 monclient(hunting): get_auth_request con 0x7fece40a7650 auth_method 0 2023-09-30T02:11:29.478 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.474+0000 7fece8a4a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:29.478 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.474+0000 7fece8a4a700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:29.478 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.474+0000 7fece8a4a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:29.478 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.474+0000 7fece3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:29.478 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.474+0000 7fece3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:29.478 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.474+0000 7fece3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:29.479 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.474+0000 7fece8a4a700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:29.479 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.474+0000 7fece8a4a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:29.479 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.474+0000 7fece8a4a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:29.479 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.478+0000 7fece3fff700 10 monclient(hunting): handle_auth_done global_id 4730 payload 824 2023-09-30T02:11:29.479 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.478+0000 7fece3fff700 1 -- 172.21.15.59:0/3928779313 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fece40a7650 msgr2=0x7fece40aa510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:29.479 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.478+0000 7fece3fff700 1 --2- 172.21.15.59:0/3928779313 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fece40a7650 0x7fece40aa510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:29.480 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.478+0000 7fece3fff700 1 -- 172.21.15.59:0/3928779313 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fece40b16e0 msgr2=0x7fece4141ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:29.480 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.478+0000 7fece3fff700 1 --2- 172.21.15.59:0/3928779313 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fece40b16e0 0x7fece4141ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:29.480 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.478+0000 7fece3fff700 10 monclient: _finish_hunting 0 2023-09-30T02:11:29.480 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.478+0000 7fece3fff700 1 monclient: found mon.noname-c 2023-09-30T02:11:29.480 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.478+0000 7fece3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:11:29.480 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.478+0000 7fece3fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.59:3301/0 2023-09-30T02:11:29.480 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.478+0000 7fece3fff700 1 -- 172.21.15.59:0/3928779313 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecd0009420 con 0x7fece40a7ff0 2023-09-30T02:11:29.480 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.478+0000 7fece8a4a700 1 --2- 172.21.15.59:0/3928779313 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fece40a7650 0x7fece40aa510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T02:11:29.481 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.480+0000 7fece3fff700 1 --2- 172.21.15.59:0/3928779313 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fece40a7ff0 0x7fece413e750 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fece40b1bf0 tx=0x7fecd0017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:29.481 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.480+0000 7fece0ff9700 10 client.?.objecter ms_handle_connect 0x7fece40a7ff0 2023-09-30T02:11:29.481 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.480+0000 7fece0ff9700 10 client.?.objecter resend_mon_ops 2023-09-30T02:11:29.485 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.486+0000 7fece0ff9700 1 -- 172.21.15.59:0/3928779313 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecd0000d10 con 0x7fece40a7ff0 2023-09-30T02:11:29.486 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.486+0000 7fece0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:29.486 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.486+0000 7fece0ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:11:29.486 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.486+0000 7fece0ff9700 10 monclient: dump: 2023-09-30T02:11:29.486 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:29.486 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:29.486 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:29.486 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:29.486 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:29.486 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:29.486 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:29.486 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:29.487 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:29.487 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:29.487 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.486+0000 7fece0ff9700 10 monclient: _finish_auth 0 2023-09-30T02:11:29.487 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.486+0000 7fece0ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:11:29.487 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.486+0000 7fece0ff9700 1 -- 172.21.15.59:0/3928779313 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecd0011b10 con 0x7fece40a7ff0 2023-09-30T02:11:29.487 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.486+0000 7feceacae700 5 monclient: authenticate success, global_id 4730 2023-09-30T02:11:29.487 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.486+0000 7feceacae700 10 monclient: _renew_subs 2023-09-30T02:11:29.487 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.486+0000 7feceacae700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:11:29.487 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.486+0000 7feceacae700 1 -- 172.21.15.59:0/3928779313 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fece40a9680 con 0x7fece40a7ff0 2023-09-30T02:11:29.487 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.486+0000 7fece0ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:11:29.487 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.486+0000 7fece0ff9700 1 -- 172.21.15.59:0/3928779313 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecd0023660 con 0x7fece40a7ff0 2023-09-30T02:11:29.487 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.486+0000 7fece0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:29.488 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.486+0000 7fece0ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:11:29.488 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.486+0000 7fece0ff9700 10 monclient: dump: 2023-09-30T02:11:29.488 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:29.488 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:29.488 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:29.488 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:29.488 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:29.488 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:29.488 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:29.488 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:29.488 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:29.488 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:29.489 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.486+0000 7fece2ffd700 4 set_mon_vals no callback set 2023-09-30T02:11:29.489 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.486+0000 7feceacae700 10 client.4730.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:11:29.489 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.486+0000 7feceacae700 10 monclient: _renew_subs 2023-09-30T02:11:29.489 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.486+0000 7feceacae700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:11:29.489 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.486+0000 7feceacae700 1 -- 172.21.15.59:0/3928779313 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fece4140810 con 0x7fece40a7ff0 2023-09-30T02:11:29.489 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.486+0000 7feceacae700 1 librados: init done 2023-09-30T02:11:29.491 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.487+0000 7feceacae700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:11:29.491 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.487+0000 7feceacae700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:11:29.491 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.487+0000 7feceacae700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:11:29.491 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.487+0000 7feceacae700 1 -- 172.21.15.59:0/3928779313 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fece40ab090 con 0x7fece40a7ff0 2023-09-30T02:11:29.793 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.792+0000 7fece0ff9700 1 -- 172.21.15.59:0/3928779313 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fecd00193b0 con 0x7fece40a7ff0 2023-09-30T02:11:29.793 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.793+0000 7fece0ff9700 1 --2- 172.21.15.59:0/3928779313 >> v2:172.21.15.110:6800/115870 conn(0x7fecc8068910 0x7fecc806adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:29.793 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.793+0000 7fece0ff9700 1 -- 172.21.15.59:0/3928779313 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fecd0085550 con 0x7fece40a7ff0 2023-09-30T02:11:29.793 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.793+0000 7fece0ff9700 10 client.4730.objecter ms_dispatch 0x7fece4001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:11:29.793 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.793+0000 7fece0ff9700 3 client.4730.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:11:29.793 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.793+0000 7fece0ff9700 3 client.4730.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:11:29.794 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.793+0000 7fece0ff9700 20 client.4730.objecter dump_active .. 0 homeless 2023-09-30T02:11:29.794 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.793+0000 7fece0ff9700 1 -- 172.21.15.59:0/3928779313 <== mon.2 v2:172.21.15.59:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fecd00859e0 con 0x7fece40a7ff0 2023-09-30T02:11:29.794 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.793+0000 7fece0ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:11:29.794 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.793+0000 7fece0ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:11:29.794 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.793+0000 7fece8a4a700 1 --2- 172.21.15.59:0/3928779313 >> v2:172.21.15.110:6800/115870 conn(0x7fecc8068910 0x7fecc806adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:29.794 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.793+0000 7fece8a4a700 10 monclient: get_auth_request con 0x7fecc8068910 auth_method 0 2023-09-30T02:11:29.794 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.793+0000 7fece0ff9700 10 client.4730.objecter ms_handle_connect 0x7fecc8068910 2023-09-30T02:11:29.794 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:29.793+0000 7fece8a4a700 1 --2- 172.21.15.59:0/3928779313 >> v2:172.21.15.110:6800/115870 conn(0x7fecc8068910 0x7fecc806adc0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fecd8009fd0 tx=0x7fecd80091a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:30.054 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.054+0000 7feceacae700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T02:11:30.054 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.054+0000 7feceacae700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:11:30.054 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.054+0000 7feceacae700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:11:30.054 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.054+0000 7feceacae700 1 -- 172.21.15.59:0/3928779313 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fece4072b30 con 0x7fece40a7ff0 2023-09-30T02:11:30.059 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.060+0000 7fece0ff9700 1 -- 172.21.15.59:0/3928779313 <== mon.2 v2:172.21.15.59:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fecd0015100 con 0x7fece40a7ff0 2023-09-30T02:11:30.059 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.060+0000 7fece0ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T02:11:30.059 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.060+0000 7fece0ff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:11:30.059 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:11:30.059 INFO:teuthology.orchestra.run.smithi059.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T02:11:30.062 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.062+0000 7feceacae700 10 librados: watch_flush enter 2023-09-30T02:11:30.062 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.062+0000 7feceacae700 10 librados: watch_flush exit 2023-09-30T02:11:30.062 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.062+0000 7feceacae700 20 client.4730.objecter shutdown clearing up homeless session... 2023-09-30T02:11:30.062 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.062+0000 7feceacae700 10 client.4730.objecter successfully canceled tick 2023-09-30T02:11:30.063 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.062+0000 7feceacae700 1 -- 172.21.15.59:0/3928779313 >> v2:172.21.15.110:6800/115870 conn(0x7fecc8068910 msgr2=0x7fecc806adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:30.063 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.062+0000 7feceacae700 1 --2- 172.21.15.59:0/3928779313 >> v2:172.21.15.110:6800/115870 conn(0x7fecc8068910 0x7fecc806adc0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fecd8009fd0 tx=0x7fecd80091a0 comp rx=0 tx=0).stop 2023-09-30T02:11:30.063 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.062+0000 7feceacae700 10 monclient: shutdown 2023-09-30T02:11:30.063 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.062+0000 7feceacae700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:11:30.063 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.062+0000 7feceacae700 1 -- 172.21.15.59:0/3928779313 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fece40a7ff0 msgr2=0x7fece413e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:30.063 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.062+0000 7feceacae700 1 --2- 172.21.15.59:0/3928779313 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fece40a7ff0 0x7fece413e750 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fece40b1bf0 tx=0x7fecd0017000 comp rx=0 tx=0).stop 2023-09-30T02:11:30.063 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.062+0000 7feceacae700 1 -- 172.21.15.59:0/3928779313 shutdown_connections 2023-09-30T02:11:30.063 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.062+0000 7feceacae700 1 --2- 172.21.15.59:0/3928779313 >> v2:172.21.15.110:6800/115870 conn(0x7fecc8068910 0x7fecc806adc0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:30.063 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.062+0000 7feceacae700 1 --2- 172.21.15.59:0/3928779313 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fece40a7650 0x7fece40aa510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:30.063 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.062+0000 7feceacae700 1 --2- 172.21.15.59:0/3928779313 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fece40b16e0 0x7fece4141ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:30.063 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.062+0000 7feceacae700 1 --2- 172.21.15.59:0/3928779313 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fece40a7ff0 0x7fece413e750 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:30.064 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.062+0000 7feceacae700 1 -- 172.21.15.59:0/3928779313 >> 172.21.15.59:0/3928779313 conn(0x7fece4068fe0 msgr2=0x7fece40631f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:11:30.064 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.062+0000 7feceacae700 1 -- 172.21.15.59:0/3928779313 shutdown_connections 2023-09-30T02:11:30.064 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.062+0000 7feceacae700 1 -- 172.21.15.59:0/3928779313 wait complete. 2023-09-30T02:11:30.064 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.062+0000 7feceacae700 1 librados: shutdown 2023-09-30T02:11:30.076 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2023-09-30T02:11:30.076 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-30T02:11:30.208 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.208+0000 7f6bf3b54700 10 monclient: get_monmap_and_config 2023-09-30T02:11:30.208 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.208+0000 7f6bf3b54700 10 monclient: build_initial_monmap 2023-09-30T02:11:30.208 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.208+0000 7f6bf3b54700 10 monclient: monmap: 2023-09-30T02:11:30.209 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:11:30.209 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:30.209 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:11:30.209200+0000 2023-09-30T02:11:30.209 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:11:30.209200+0000 2023-09-30T02:11:30.209 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:11:30.209 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:30.209 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:11:30.209 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:11:30.209 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:11:30.209 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:11:30.210 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:11:30.210 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:30.210 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.209+0000 7f6bf3b54700 1 Processor -- start 2023-09-30T02:11:30.211 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.209+0000 7f6bf3b54700 1 -- start start 2023-09-30T02:11:30.212 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.209+0000 7f6bf3b54700 10 monclient: init 2023-09-30T02:11:30.212 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.210+0000 7f6bf3b54700 10 monclient: _reopen_session rank -1 2023-09-30T02:11:30.212 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.210+0000 7f6bf3b54700 10 monclient: _add_conns ranks=[4,2,0,3,1] 2023-09-30T02:11:30.212 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.210+0000 7f6bf3b54700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f6bec0a5b70 0x7f6bec0b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:30.213 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.210+0000 7f6bf3b54700 10 monclient(hunting): picked mon.noname-c con 0x7f6bec0a5b70 addr v2:172.21.15.110:3300/0 2023-09-30T02:11:30.213 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.210+0000 7f6bf3b54700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6bec0a5f40 0x7f6bec141b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:30.213 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.210+0000 7f6bf3b54700 10 monclient(hunting): picked mon.noname-b con 0x7f6bec0a5f40 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:11:30.213 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.210+0000 7f6bf3b54700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6bec1420c0 addr v1:172.21.15.59:6789/0 2023-09-30T02:11:30.213 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.210+0000 7f6bf3b54700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bec0a63b0 con 0x7f6bec1420c0 2023-09-30T02:11:30.213 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.210+0000 7f6bf3b54700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:30.213 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.210+0000 7f6bf3b54700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bec0a75a0 con 0x7f6bec0a5b70 2023-09-30T02:11:30.213 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.210+0000 7f6bf3b54700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:30.213 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.210+0000 7f6bf3b54700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bec0af3f0 con 0x7f6bec0a5f40 2023-09-30T02:11:30.213 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.210+0000 7f6bf3b54700 10 monclient(hunting): _renew_subs 2023-09-30T02:11:30.213 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.210+0000 7f6bf3b54700 10 monclient(hunting): authenticate will time out at 2135.512501s 2023-09-30T02:11:30.214 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.211+0000 7f6bf18f0700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f6bec0a5b70 0x7f6bec0b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:30.214 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.211+0000 7f6bf20f1700 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f6bec1420c0 0x7f6bec13e230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:58742/0 (socket says 172.21.15.59:58742) 2023-09-30T02:11:30.214 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.211+0000 7f6bf20f1700 1 -- 172.21.15.59:0/1347454605 learned_addr learned my addr 172.21.15.59:0/1347454605 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:11:30.215 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.211+0000 7f6bf18f0700 10 monclient(hunting): get_auth_request con 0x7f6bec0a5b70 auth_method 0 2023-09-30T02:11:30.215 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.211+0000 7f6bf18f0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:30.215 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.211+0000 7f6bf18f0700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:30.215 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.211+0000 7f6bf18f0700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:30.215 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.212+0000 7f6bf18f0700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:30.215 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.212+0000 7f6bf18f0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:30.215 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.212+0000 7f6bf18f0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:30.218 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.214+0000 7f6bf18f0700 10 monclient(hunting): handle_auth_done global_id 4852 payload 274 2023-09-30T02:11:30.218 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.214+0000 7f6bf18f0700 1 -- 172.21.15.59:0/1347454605 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6bec0a5f40 msgr2=0x7f6bec141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:11:30.218 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.214+0000 7f6bf18f0700 1 --2- 172.21.15.59:0/1347454605 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6bec0a5f40 0x7f6bec141b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:30.218 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.214+0000 7f6bf18f0700 1 -- 172.21.15.59:0/1347454605 >> v1:172.21.15.59:6789/0 conn(0x7f6bec1420c0 legacy=0x7f6bec13e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:30.218 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.214+0000 7f6bf18f0700 10 monclient: _finish_hunting 0 2023-09-30T02:11:30.218 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.214+0000 7f6bf18f0700 1 monclient: found mon.noname-c 2023-09-30T02:11:30.218 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.214+0000 7f6bf18f0700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:11:30.218 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.214+0000 7f6bf18f0700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3300/0 2023-09-30T02:11:30.218 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.214+0000 7f6bf18f0700 1 -- 172.21.15.59:0/1347454605 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bec142500 con 0x7f6bec0a5b70 2023-09-30T02:11:30.218 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.216+0000 7f6bf18f0700 1 --2- 172.21.15.59:0/1347454605 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6bec0a5b70 0x7f6bec0b1500 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f6be80098e0 tx=0x7f6be80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=366310e9ea4b5cba server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:30.221 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.220+0000 7f6bf08ee700 1 -- 172.21.15.59:0/1347454605 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6be8009e80 con 0x7f6bec0a5b70 2023-09-30T02:11:30.221 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.220+0000 7f6bf08ee700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:30.221 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.220+0000 7f6bf08ee700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:11:30.221 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.220+0000 7f6bf08ee700 10 monclient: dump: 2023-09-30T02:11:30.221 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:30.221 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:30.221 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:30.221 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:30.221 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:30.221 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:30.221 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:30.222 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:30.222 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:30.222 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:30.222 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.220+0000 7f6bf08ee700 10 monclient: _finish_auth 0 2023-09-30T02:11:30.222 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.220+0000 7f6bf08ee700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:11:30.222 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.220+0000 7f6bf3b54700 5 monclient: authenticate success, global_id 4852 2023-09-30T02:11:30.222 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.220+0000 7f6bf3b54700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:11:30.222 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.220+0000 7f6bf08ee700 1 -- 172.21.15.59:0/1347454605 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6be800b860 con 0x7f6bec0a5b70 2023-09-30T02:11:30.222 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.220+0000 7f6bf08ee700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:11:30.222 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.220+0000 7f6bf3b54700 10 monclient: get_monmap_and_config success 2023-09-30T02:11:30.222 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.220+0000 7f6bf3b54700 4 set_mon_vals no callback set 2023-09-30T02:11:30.222 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.220+0000 7f6bf08ee700 1 -- 172.21.15.59:0/1347454605 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6be800bbe0 con 0x7f6bec0a5b70 2023-09-30T02:11:30.223 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.220+0000 7f6bf08ee700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:30.223 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.220+0000 7f6bf08ee700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:11:30.223 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.220+0000 7f6bf08ee700 10 monclient: dump: 2023-09-30T02:11:30.223 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:30.223 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:30.223 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:30.223 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:30.223 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:30.223 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:30.223 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:30.223 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:30.223 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:30.224 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:30.224 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.221+0000 7f6bf3b54700 10 monclient: shutdown 2023-09-30T02:11:30.224 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.221+0000 7f6bf3b54700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:11:30.224 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.221+0000 7f6bf3b54700 1 -- 172.21.15.59:0/1347454605 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6bec0a5b70 msgr2=0x7f6bec0b1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:30.224 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.221+0000 7f6bf3b54700 1 --2- 172.21.15.59:0/1347454605 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6bec0a5b70 0x7f6bec0b1500 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f6be80098e0 tx=0x7f6be80091a0 comp rx=0 tx=0).stop 2023-09-30T02:11:30.224 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.221+0000 7f6bf3b54700 1 -- 172.21.15.59:0/1347454605 shutdown_connections 2023-09-30T02:11:30.224 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.221+0000 7f6bf3b54700 1 --2- 172.21.15.59:0/1347454605 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6bec0a5b70 0x7f6bec0b1500 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:30.224 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.221+0000 7f6bf3b54700 1 --2- 172.21.15.59:0/1347454605 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6bec0a5f40 0x7f6bec141b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:30.224 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.221+0000 7f6bf3b54700 1 -- 172.21.15.59:0/1347454605 >> 172.21.15.59:0/1347454605 conn(0x7f6bec0631f0 msgr2=0x7f6bec0635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:11:30.224 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.221+0000 7f6bf3b54700 1 -- 172.21.15.59:0/1347454605 shutdown_connections 2023-09-30T02:11:30.224 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.221+0000 7f6bf3b54700 1 -- 172.21.15.59:0/1347454605 wait complete. 2023-09-30T02:11:30.227 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.222+0000 7f6bf3b54700 10 monclient: build_initial_monmap 2023-09-30T02:11:30.227 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.222+0000 7f6bf3b54700 10 monclient: monmap: 2023-09-30T02:11:30.227 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:11:30.227 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:11:30.227 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:11:30.227 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:11:30.227 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:11:30.227 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:11:30.227 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:11:30.227 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:11:30.227 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:11:30.228 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:30.228 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.223+0000 7f6bf3b54700 1 librados: starting msgr at 2023-09-30T02:11:30.228 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.223+0000 7f6bf3b54700 1 librados: starting objecter 2023-09-30T02:11:30.228 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.223+0000 7f6bf3b54700 1 Processor -- start 2023-09-30T02:11:30.228 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.223+0000 7f6bf3b54700 1 -- start start 2023-09-30T02:11:30.228 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.223+0000 7f6bf3b54700 1 librados: setting wanted keys 2023-09-30T02:11:30.228 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.223+0000 7f6bf3b54700 1 librados: calling monclient init 2023-09-30T02:11:30.228 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.223+0000 7f6bf3b54700 10 monclient: init 2023-09-30T02:11:30.228 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.224+0000 7f6bf3b54700 10 monclient: _reopen_session rank -1 2023-09-30T02:11:30.228 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.224+0000 7f6bf3b54700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T02:11:30.228 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.224+0000 7f6bf3b54700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6bec0a5b70 0x7f6bec1517c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:30.228 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.224+0000 7f6bf3b54700 10 monclient(hunting): picked mon.noname-b con 0x7f6bec0a5b70 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:11:30.229 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.224+0000 7f6bf3b54700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6bec0a5f40 0x7f6bec137be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:30.229 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.224+0000 7f6bf3b54700 10 monclient(hunting): picked mon.noname-c con 0x7f6bec0a5f40 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:11:30.229 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.224+0000 7f6bf3b54700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6bec1420c0 0x7f6bec138120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:30.229 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.224+0000 7f6bf3b54700 10 monclient(hunting): picked mon.noname-a con 0x7f6bec1420c0 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:11:30.229 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.224+0000 7f6bf3b54700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:30.229 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.224+0000 7f6bf3b54700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bec0b2d80 con 0x7f6bec1420c0 2023-09-30T02:11:30.229 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.224+0000 7f6bf3b54700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:30.229 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.224+0000 7f6bf3b54700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bec0a99a0 con 0x7f6bec0a5b70 2023-09-30T02:11:30.229 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.224+0000 7f6bf3b54700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:30.229 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.224+0000 7f6bf3b54700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bec138660 con 0x7f6bec0a5f40 2023-09-30T02:11:30.229 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.224+0000 7f6bf3b54700 10 monclient(hunting): _renew_subs 2023-09-30T02:11:30.229 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.224+0000 7f6bf3b54700 10 monclient(hunting): authenticate will time out at 2135.526196s 2023-09-30T02:11:30.229 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.224+0000 7f6bf18f0700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6bec0a5b70 0x7f6bec1517c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:30.230 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.225+0000 7f6bf10ef700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6bec0a5f40 0x7f6bec137be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:30.230 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.225+0000 7f6bf10ef700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6bec0a5f40 0x7f6bec137be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:33680/0 (socket says 172.21.15.59:33680) 2023-09-30T02:11:30.230 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.225+0000 7f6bf10ef700 1 -- 172.21.15.59:0/3098809426 learned_addr learned my addr 172.21.15.59:0/3098809426 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:11:30.230 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.225+0000 7f6bf10ef700 10 monclient(hunting): get_auth_request con 0x7f6bec0a5f40 auth_method 0 2023-09-30T02:11:30.230 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.225+0000 7f6bf10ef700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:30.230 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.225+0000 7f6bf10ef700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:30.230 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.225+0000 7f6bf10ef700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:30.230 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.225+0000 7f6bf18f0700 10 monclient(hunting): get_auth_request con 0x7f6bec0a5b70 auth_method 0 2023-09-30T02:11:30.230 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.225+0000 7f6bf18f0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:30.230 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.225+0000 7f6bf18f0700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:30.230 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.225+0000 7f6bf18f0700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:30.230 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.226+0000 7f6bf10ef700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:30.231 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.226+0000 7f6bf10ef700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:30.231 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.226+0000 7f6bf10ef700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:30.231 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.226+0000 7f6bf18f0700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:30.231 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.226+0000 7f6bf18f0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:30.231 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.226+0000 7f6bf18f0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:30.232 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.230+0000 7f6bf10ef700 10 monclient(hunting): handle_auth_done global_id 4733 payload 824 2023-09-30T02:11:30.232 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.230+0000 7f6bf10ef700 1 -- 172.21.15.59:0/3098809426 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6bec0a5b70 msgr2=0x7f6bec1517c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:30.232 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.230+0000 7f6bf10ef700 1 --2- 172.21.15.59:0/3098809426 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6bec0a5b70 0x7f6bec1517c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:30.232 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.230+0000 7f6bf10ef700 1 -- 172.21.15.59:0/3098809426 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6bec1420c0 msgr2=0x7f6bec138120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:30.232 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.230+0000 7f6bf10ef700 1 --2- 172.21.15.59:0/3098809426 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6bec1420c0 0x7f6bec138120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:30.232 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.230+0000 7f6bf10ef700 10 monclient: _finish_hunting 0 2023-09-30T02:11:30.232 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.230+0000 7f6bf10ef700 1 monclient: found mon.noname-c 2023-09-30T02:11:30.232 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.230+0000 7f6bf10ef700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:11:30.233 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.230+0000 7f6bf10ef700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.59:3301/0 2023-09-30T02:11:30.233 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.230+0000 7f6bf10ef700 1 -- 172.21.15.59:0/3098809426 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6be8009420 con 0x7f6bec0a5f40 2023-09-30T02:11:30.233 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.232+0000 7f6bf10ef700 1 --2- 172.21.15.59:0/3098809426 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6bec0a5f40 0x7f6bec137be0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f6bdc00b840 tx=0x7f6bdc009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:30.233 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.232+0000 7f6be1ffb700 10 client.?.objecter ms_handle_connect 0x7f6bec0a5f40 2023-09-30T02:11:30.233 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.232+0000 7f6be1ffb700 10 client.?.objecter resend_mon_ops 2023-09-30T02:11:30.242 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.242+0000 7f6be1ffb700 1 -- 172.21.15.59:0/3098809426 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bdc00f510 con 0x7f6bec0a5f40 2023-09-30T02:11:30.242 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.242+0000 7f6be1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:30.242 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.242+0000 7f6be1ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:11:30.242 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.242+0000 7f6be1ffb700 10 monclient: dump: 2023-09-30T02:11:30.242 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:30.242 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:30.242 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:30.243 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:30.243 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:30.243 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:30.243 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:30.243 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:30.243 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:30.243 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:30.243 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.242+0000 7f6be1ffb700 10 monclient: _finish_auth 0 2023-09-30T02:11:30.243 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.242+0000 7f6be1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:11:30.243 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.242+0000 7f6be1ffb700 1 -- 172.21.15.59:0/3098809426 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bdc00f670 con 0x7f6bec0a5f40 2023-09-30T02:11:30.243 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.242+0000 7f6be1ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:11:30.243 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.242+0000 7f6be1ffb700 1 -- 172.21.15.59:0/3098809426 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bdc00fa30 con 0x7f6bec0a5f40 2023-09-30T02:11:30.244 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.242+0000 7f6bf3b54700 5 monclient: authenticate success, global_id 4733 2023-09-30T02:11:30.244 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.242+0000 7f6be1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:30.244 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.242+0000 7f6be1ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:11:30.244 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.242+0000 7f6be1ffb700 10 monclient: dump: 2023-09-30T02:11:30.244 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:30.244 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:30.244 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:30.244 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:30.244 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:30.244 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:30.244 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:30.244 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:30.244 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:30.245 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:30.245 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.242+0000 7f6be3fff700 4 set_mon_vals no callback set 2023-09-30T02:11:30.245 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.242+0000 7f6bf3b54700 10 monclient: _renew_subs 2023-09-30T02:11:30.245 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.242+0000 7f6bf3b54700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:11:30.245 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.242+0000 7f6bf3b54700 1 -- 172.21.15.59:0/3098809426 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bec0b57c0 con 0x7f6bec0a5f40 2023-09-30T02:11:30.245 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.242+0000 7f6bf3b54700 10 client.4733.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:11:30.245 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.242+0000 7f6bf3b54700 10 monclient: _renew_subs 2023-09-30T02:11:30.245 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.242+0000 7f6bf3b54700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:11:30.245 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.242+0000 7f6bf3b54700 1 -- 172.21.15.59:0/3098809426 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bec1523d0 con 0x7f6bec0a5f40 2023-09-30T02:11:30.245 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.242+0000 7f6bf3b54700 1 librados: init done 2023-09-30T02:11:30.246 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.243+0000 7f6bf3b54700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:11:30.246 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.243+0000 7f6bf3b54700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:11:30.246 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.243+0000 7f6bf3b54700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:11:30.246 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.243+0000 7f6bf3b54700 1 -- 172.21.15.59:0/3098809426 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bec0b2f30 con 0x7f6bec0a5f40 2023-09-30T02:11:30.269 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.269+0000 7f6be1ffb700 1 -- 172.21.15.59:0/3098809426 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f6bdc024430 con 0x7f6bec0a5f40 2023-09-30T02:11:30.269 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.269+0000 7f6be1ffb700 1 --2- 172.21.15.59:0/3098809426 >> v2:172.21.15.110:6800/115870 conn(0x7f6bd0068890 0x7f6bd006ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:30.269 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.269+0000 7f6be1ffb700 1 -- 172.21.15.59:0/3098809426 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f6bdc0843e0 con 0x7f6bec0a5f40 2023-09-30T02:11:30.269 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.269+0000 7f6be1ffb700 10 client.4733.objecter ms_dispatch 0x7f6bec001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:11:30.269 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.269+0000 7f6be1ffb700 3 client.4733.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:11:30.269 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.269+0000 7f6be1ffb700 3 client.4733.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:11:30.269 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.269+0000 7f6be1ffb700 20 client.4733.objecter dump_active .. 0 homeless 2023-09-30T02:11:30.269 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.269+0000 7f6bf18f0700 1 --2- 172.21.15.59:0/3098809426 >> v2:172.21.15.110:6800/115870 conn(0x7f6bd0068890 0x7f6bd006ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:30.269 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.269+0000 7f6bf18f0700 10 monclient: get_auth_request con 0x7f6bd0068890 auth_method 0 2023-09-30T02:11:30.270 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.270+0000 7f6be1ffb700 10 client.4733.objecter ms_handle_connect 0x7f6bd0068890 2023-09-30T02:11:30.270 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.270+0000 7f6bf18f0700 1 --2- 172.21.15.59:0/3098809426 >> v2:172.21.15.110:6800/115870 conn(0x7f6bd0068890 0x7f6bd006ad40 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f6be80091e0 tx=0x7f6be8017000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:30.540 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.541+0000 7f6be1ffb700 1 -- 172.21.15.59:0/3098809426 <== mon.2 v2:172.21.15.59:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6bdc051d00 con 0x7f6bec0a5f40 2023-09-30T02:11:30.540 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.541+0000 7f6be1ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:11:30.541 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.541+0000 7f6be1ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:11:30.773 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.774+0000 7f6bf3b54700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-09-30T02:11:30.773 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.774+0000 7f6bf3b54700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T02:11:30.773 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.774+0000 7f6bf3b54700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:11:30.773 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.774+0000 7f6bf3b54700 1 -- 172.21.15.59:0/3098809426 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6bec0a9ae0 con 0x7f6bec0a5f40 2023-09-30T02:11:30.803 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.804+0000 7f6be1ffb700 1 -- 172.21.15.59:0/3098809426 <== mon.2 v2:172.21.15.59:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+11935 (secure 0 0 0) 0x7f6bec0a9ae0 con 0x7f6bec0a5f40 2023-09-30T02:11:30.804 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.804+0000 7f6be1ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T02:11:30.804 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.804+0000 7f6be1ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:11:30.804 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:11:30.804 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":25,"fsid":"623052cb-4f82-4b2b-b72b-3ed5be614613","created":"2023-09-30T02:05:11.679291+0000","modified":"2023-09-30T02:06:19.209689+0000","last_up_change":"2023-09-30T02:06:10.164005+0000","last_in_change":"2023-09-30T02:05:18.270595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-30T02:06:06.187725+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-09-30T02:06:15.695024+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"efdab44b-be57-4b81-8567-ff2deec1d3e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":116664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6801","nonce":116664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6803","nonce":116664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":116664}]},"public_addr":"172.21.15.59:6800/116664","cluster_addr":"172.21.15.59:6801/116664","heartbeat_back_addr":"172.21.15.59:6803/116664","heartbeat_front_addr":"172.21.15.59:6802/116664","state":["exists","up"]},{"osd":1,"uuid":"eff9786d-8c39-4044-91d2-a31f33e3b16e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":116665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6809","nonce":116665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6811","nonce":116665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":116665}]},"public_addr":"172.21.15.59:6808/116665","cluster_addr":"172.21.15.59:6809/116665","heartbeat_back_addr":"172.21.15.59:6811/116665","heartbeat_front_addr":"172.21.15.59:6810/116665","state":["exists","up"]},{"osd":2,"uuid":"b276ed4b-317b-4ddf-96ab-7ad71b8e3ff7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":116666}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6805","nonce":116666}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6807","nonce":116666}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":116666}]},"public_addr":"172.21.15.59:6804/116666","cluster_addr":"172.21.15.59:6805/116666","heartbeat_back_addr":"172.21.15.59:6807/116666","heartbeat_front_addr":"172.21.15.59:6806/116666","state":["exists","up"]},{"osd":3,"uuid":"48bfaf43-5846-4385-b9c6-6bc424cd5acd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":116667}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6813","nonce":116667}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6815","nonce":116667}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":116667}]},"public_addr":"172.21.15.59:6812/116667","cluster_addr":"172.21.15.59:6813/116667","heartbeat_back_addr":"172.21.15.59:6815/116667","heartbeat_front_addr":"172.21.15.59:6814/116667","state":["exists","up"]},{"osd":4,"uuid":"76fa5443-1571-4e72-9d22-6ab19e59839a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6801","nonce":116560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":116560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":116560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6803","nonce":116560}]},"public_addr":"172.21.15.110:6801/116560","cluster_addr":"172.21.15.110:6802/116560","heartbeat_back_addr":"172.21.15.110:6804/116560","heartbeat_front_addr":"172.21.15.110:6803/116560","state":["exists","up"]},{"osd":5,"uuid":"e9436478-4140-4fd6-ba2e-04fa6f6cf367","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6809","nonce":116562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":116562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":116562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6811","nonce":116562}]},"public_addr":"172.21.15.110:6809/116562","cluster_addr":"172.21.15.110:6810/116562","heartbeat_back_addr":"172.21.15.110:6812/116562","heartbeat_front_addr":"172.21.15.110:6811/116562","state":["exists","up"]},{"osd":6,"uuid":"76676042-2074-4cf8-9ef9-5c87bf604944","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6813","nonce":116563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":116563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":116563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6815","nonce":116563}]},"public_addr":"172.21.15.110:6813/116563","cluster_addr":"172.21.15.110:6814/116563","heartbeat_back_addr":"172.21.15.110:6816/116563","heartbeat_front_addr":"172.21.15.110:6815/116563","state":["exists","up"]},{"osd":7,"uuid":"045479f4-f7f7-4b95-bc38-a2c52ac31b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6805","nonce":116561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":116561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":116561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6807","nonce":116561}]},"public_addr":"172.21.15.110:6805/116561","cluster_addr":"172.21.15.110:6806/116561","heartbeat_back_addr":"172.21.15.110:6808/116561","heartbeat_front_addr":"172.21.15.110:6807/116561","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:06:05.267167+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:06:06.183162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:06:05.646500+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:06:06.546668+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:06:02.158984+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:06:01.902487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:06:01.310184+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:06:01.669641+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T02:11:30.806 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.806+0000 7f6bf3b54700 10 librados: watch_flush enter 2023-09-30T02:11:30.807 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.806+0000 7f6bf3b54700 10 librados: watch_flush exit 2023-09-30T02:11:30.807 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.806+0000 7f6bf3b54700 20 client.4733.objecter shutdown clearing up homeless session... 2023-09-30T02:11:30.807 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.806+0000 7f6bf3b54700 10 client.4733.objecter successfully canceled tick 2023-09-30T02:11:30.807 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.807+0000 7f6bf3b54700 1 -- 172.21.15.59:0/3098809426 >> v2:172.21.15.110:6800/115870 conn(0x7f6bd0068890 msgr2=0x7f6bd006ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:30.807 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.807+0000 7f6bf3b54700 1 --2- 172.21.15.59:0/3098809426 >> v2:172.21.15.110:6800/115870 conn(0x7f6bd0068890 0x7f6bd006ad40 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f6be80091e0 tx=0x7f6be8017000 comp rx=0 tx=0).stop 2023-09-30T02:11:30.807 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.807+0000 7f6bf3b54700 10 monclient: shutdown 2023-09-30T02:11:30.807 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.807+0000 7f6bf3b54700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:11:30.807 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.807+0000 7f6bf3b54700 1 -- 172.21.15.59:0/3098809426 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6bec0a5f40 msgr2=0x7f6bec137be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:30.807 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.807+0000 7f6bf3b54700 1 --2- 172.21.15.59:0/3098809426 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6bec0a5f40 0x7f6bec137be0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f6bdc00b840 tx=0x7f6bdc009250 comp rx=0 tx=0).stop 2023-09-30T02:11:30.807 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.807+0000 7f6bf3b54700 1 -- 172.21.15.59:0/3098809426 shutdown_connections 2023-09-30T02:11:30.808 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.807+0000 7f6bf3b54700 1 --2- 172.21.15.59:0/3098809426 >> v2:172.21.15.110:6800/115870 conn(0x7f6bd0068890 0x7f6bd006ad40 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:30.808 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.807+0000 7f6bf3b54700 1 --2- 172.21.15.59:0/3098809426 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f6bec0a5b70 0x7f6bec1517c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:30.808 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.807+0000 7f6bf3b54700 1 --2- 172.21.15.59:0/3098809426 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6bec1420c0 0x7f6bec138120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:30.808 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.807+0000 7f6bf3b54700 1 --2- 172.21.15.59:0/3098809426 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f6bec0a5f40 0x7f6bec137be0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:30.808 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.807+0000 7f6bf3b54700 1 -- 172.21.15.59:0/3098809426 >> 172.21.15.59:0/3098809426 conn(0x7f6bec0631f0 msgr2=0x7f6bec140b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:11:30.808 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.807+0000 7f6bf3b54700 1 -- 172.21.15.59:0/3098809426 shutdown_connections 2023-09-30T02:11:30.808 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.807+0000 7f6bf3b54700 1 -- 172.21.15.59:0/3098809426 wait complete. 2023-09-30T02:11:30.808 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.807+0000 7f6bf3b54700 1 librados: shutdown 2023-09-30T02:11:30.824 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-09-30T02:11:30.824 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T02:11:30.958 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.958+0000 7f75e3b1b700 10 monclient: get_monmap_and_config 2023-09-30T02:11:30.959 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.958+0000 7f75e3b1b700 10 monclient: build_initial_monmap 2023-09-30T02:11:30.959 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.959+0000 7f75e3b1b700 10 monclient: monmap: 2023-09-30T02:11:30.959 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:11:30.959 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:30.959 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:11:30.959399+0000 2023-09-30T02:11:30.959 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:11:30.959399+0000 2023-09-30T02:11:30.959 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:11:30.959 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:30.959 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:11:30.959 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:11:30.959 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:11:30.959 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:11:30.960 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:11:30.960 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:30.960 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.959+0000 7f75e3b1b700 1 Processor -- start 2023-09-30T02:11:30.961 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.959+0000 7f75e3b1b700 1 -- start start 2023-09-30T02:11:30.962 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.959+0000 7f75e3b1b700 10 monclient: init 2023-09-30T02:11:30.962 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.960+0000 7f75e3b1b700 10 monclient: _reopen_session rank -1 2023-09-30T02:11:30.962 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.960+0000 7f75e3b1b700 10 monclient: _add_conns ranks=[2,4,0,3,1] 2023-09-30T02:11:30.962 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.960+0000 7f75e3b1b700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f75dc0a5b70 0x7f75dc0b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:30.962 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.960+0000 7f75e3b1b700 10 monclient(hunting): picked mon.noname-b con 0x7f75dc0a5b70 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:11:30.962 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.960+0000 7f75e3b1b700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f75dc0a5f40 0x7f75dc141b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:30.962 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.960+0000 7f75e3b1b700 10 monclient(hunting): picked mon.noname-c con 0x7f75dc0a5f40 addr v2:172.21.15.110:3300/0 2023-09-30T02:11:30.962 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.960+0000 7f75e3b1b700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f75dc1420c0 addr v1:172.21.15.59:6789/0 2023-09-30T02:11:30.963 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.960+0000 7f75e3b1b700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75dc0a63b0 con 0x7f75dc1420c0 2023-09-30T02:11:30.963 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.960+0000 7f75e3b1b700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:30.963 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.960+0000 7f75e3b1b700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75dc0a75a0 con 0x7f75dc0a5f40 2023-09-30T02:11:30.963 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.960+0000 7f75e3b1b700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:30.963 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.960+0000 7f75e3b1b700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75dc0af3f0 con 0x7f75dc0a5b70 2023-09-30T02:11:30.963 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.960+0000 7f75e3b1b700 10 monclient(hunting): _renew_subs 2023-09-30T02:11:30.963 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.960+0000 7f75e3b1b700 10 monclient(hunting): authenticate will time out at 2136.262595s 2023-09-30T02:11:30.963 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.961+0000 7f75e18b7700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f75dc0a5b70 0x7f75dc0b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:30.963 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.961+0000 7f75e20b8700 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f75dc1420c0 0x7f75dc13e230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:58754/0 (socket says 172.21.15.59:58754) 2023-09-30T02:11:30.963 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.961+0000 7f75e20b8700 1 -- 172.21.15.59:0/3971334075 learned_addr learned my addr 172.21.15.59:0/3971334075 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:11:30.963 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.961+0000 7f75e10b6700 1 --2- 172.21.15.59:0/3971334075 >> v2:172.21.15.110:3300/0 conn(0x7f75dc0a5f40 0x7f75dc141b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:30.963 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.961+0000 7f75e10b6700 10 monclient(hunting): get_auth_request con 0x7f75dc0a5f40 auth_method 0 2023-09-30T02:11:30.964 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.961+0000 7f75e10b6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:30.964 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.961+0000 7f75e10b6700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:30.964 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.961+0000 7f75e10b6700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:30.964 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.961+0000 7f75e18b7700 10 monclient(hunting): get_auth_request con 0x7f75dc0a5b70 auth_method 0 2023-09-30T02:11:30.964 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.961+0000 7f75e18b7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:30.964 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.961+0000 7f75e18b7700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:30.964 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.961+0000 7f75e18b7700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:30.964 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.962+0000 7f75e10b6700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:30.964 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.962+0000 7f75e10b6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:30.964 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.962+0000 7f75e10b6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:30.964 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.962+0000 7f75e18b7700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:30.964 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.962+0000 7f75e18b7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:30.965 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.962+0000 7f75e18b7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:30.965 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.964+0000 7f75e10b6700 10 monclient(hunting): handle_auth_done global_id 4858 payload 274 2023-09-30T02:11:30.965 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.964+0000 7f75e10b6700 1 -- 172.21.15.59:0/3971334075 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f75dc0a5b70 msgr2=0x7f75dc0b1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:11:30.965 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.964+0000 7f75e10b6700 1 --2- 172.21.15.59:0/3971334075 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f75dc0a5b70 0x7f75dc0b1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:30.965 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.964+0000 7f75e10b6700 1 -- 172.21.15.59:0/3971334075 >> v1:172.21.15.59:6789/0 conn(0x7f75dc1420c0 legacy=0x7f75dc13e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:30.966 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.964+0000 7f75e10b6700 10 monclient: _finish_hunting 0 2023-09-30T02:11:30.966 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.964+0000 7f75e10b6700 1 monclient: found mon.noname-c 2023-09-30T02:11:30.966 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.964+0000 7f75e10b6700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:11:30.966 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.964+0000 7f75e10b6700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3300/0 2023-09-30T02:11:30.966 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.964+0000 7f75e10b6700 1 -- 172.21.15.59:0/3971334075 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75dc142500 con 0x7f75dc0a5f40 2023-09-30T02:11:30.966 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.966+0000 7f75e10b6700 1 --2- 172.21.15.59:0/3971334075 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f75dc0a5f40 0x7f75dc141b80 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f75cc0098e0 tx=0x7f75cc0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf9437c639658495 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:30.970 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.970+0000 7f75e08b5700 1 -- 172.21.15.59:0/3971334075 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75cc00b640 con 0x7f75dc0a5f40 2023-09-30T02:11:30.970 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.970+0000 7f75e08b5700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:30.970 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.970+0000 7f75e08b5700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:11:30.970 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.970+0000 7f75e08b5700 10 monclient: dump: 2023-09-30T02:11:30.970 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:30.970 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:30.970 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:30.970 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:30.971 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:30.971 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:30.971 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:30.971 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:30.971 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:30.971 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:30.971 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.970+0000 7f75e08b5700 10 monclient: _finish_auth 0 2023-09-30T02:11:30.971 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.970+0000 7f75e08b5700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:11:30.971 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.970+0000 7f75e08b5700 1 -- 172.21.15.59:0/3971334075 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75cc00b7a0 con 0x7f75dc0a5f40 2023-09-30T02:11:30.971 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.970+0000 7f75e08b5700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:11:30.971 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.970+0000 7f75e08b5700 1 -- 172.21.15.59:0/3971334075 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75cc00bb40 con 0x7f75dc0a5f40 2023-09-30T02:11:30.971 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.970+0000 7f75e08b5700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:30.972 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.970+0000 7f75e08b5700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:11:30.972 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.970+0000 7f75e08b5700 10 monclient: dump: 2023-09-30T02:11:30.972 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:30.972 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:30.972 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:30.972 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:30.972 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:30.972 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:30.972 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:30.972 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:30.972 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:30.972 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:30.973 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.970+0000 7f75e3b1b700 5 monclient: authenticate success, global_id 4858 2023-09-30T02:11:30.973 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.970+0000 7f75e3b1b700 10 monclient: get_monmap_and_config success 2023-09-30T02:11:30.973 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.970+0000 7f75e3b1b700 4 set_mon_vals no callback set 2023-09-30T02:11:30.976 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.971+0000 7f75e3b1b700 10 monclient: shutdown 2023-09-30T02:11:30.976 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.971+0000 7f75e3b1b700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:11:30.976 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.971+0000 7f75e3b1b700 1 -- 172.21.15.59:0/3971334075 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f75dc0a5f40 msgr2=0x7f75dc141b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:30.976 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.971+0000 7f75e3b1b700 1 --2- 172.21.15.59:0/3971334075 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f75dc0a5f40 0x7f75dc141b80 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f75cc0098e0 tx=0x7f75cc0091a0 comp rx=0 tx=0).stop 2023-09-30T02:11:30.976 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.971+0000 7f75e3b1b700 1 -- 172.21.15.59:0/3971334075 shutdown_connections 2023-09-30T02:11:30.976 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.971+0000 7f75e3b1b700 1 --2- 172.21.15.59:0/3971334075 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f75dc0a5b70 0x7f75dc0b1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:30.976 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.971+0000 7f75e3b1b700 1 --2- 172.21.15.59:0/3971334075 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f75dc0a5f40 0x7f75dc141b80 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:30.976 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.971+0000 7f75e3b1b700 1 -- 172.21.15.59:0/3971334075 >> 172.21.15.59:0/3971334075 conn(0x7f75dc0631f0 msgr2=0x7f75dc0635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:11:30.976 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.971+0000 7f75e3b1b700 1 -- 172.21.15.59:0/3971334075 shutdown_connections 2023-09-30T02:11:30.977 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.971+0000 7f75e3b1b700 1 -- 172.21.15.59:0/3971334075 wait complete. 2023-09-30T02:11:30.977 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.972+0000 7f75e3b1b700 10 monclient: build_initial_monmap 2023-09-30T02:11:30.977 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.972+0000 7f75e3b1b700 10 monclient: monmap: 2023-09-30T02:11:30.977 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:11:30.977 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:11:30.977 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:11:30.977 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:11:30.977 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:11:30.977 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:11:30.977 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:11:30.977 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:11:30.977 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:11:30.977 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:30.978 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.972+0000 7f75e3b1b700 1 librados: starting msgr at 2023-09-30T02:11:30.978 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.972+0000 7f75e3b1b700 1 librados: starting objecter 2023-09-30T02:11:30.978 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.972+0000 7f75e3b1b700 1 Processor -- start 2023-09-30T02:11:30.978 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.972+0000 7f75e3b1b700 1 -- start start 2023-09-30T02:11:30.978 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.972+0000 7f75e3b1b700 1 librados: setting wanted keys 2023-09-30T02:11:30.978 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.972+0000 7f75e3b1b700 1 librados: calling monclient init 2023-09-30T02:11:30.978 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.972+0000 7f75e3b1b700 10 monclient: init 2023-09-30T02:11:30.978 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.973+0000 7f75e3b1b700 10 monclient: _reopen_session rank -1 2023-09-30T02:11:30.978 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.973+0000 7f75e3b1b700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T02:11:30.978 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.973+0000 7f75e3b1b700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f75dc0a5b70 0x7f75dc141100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:30.978 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.973+0000 7f75e3b1b700 10 monclient(hunting): picked mon.noname-c con 0x7f75dc0a5b70 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:11:30.978 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.973+0000 7f75e3b1b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75dc0a5f40 0x7f75dc151870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:30.979 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.973+0000 7f75e3b1b700 10 monclient(hunting): picked mon.noname-a con 0x7f75dc0a5f40 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:11:30.979 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.973+0000 7f75e3b1b700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f75dc1420c0 0x7f75dc0a78a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:30.979 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.973+0000 7f75e3b1b700 10 monclient(hunting): picked mon.noname-b con 0x7f75dc1420c0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:11:30.979 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.973+0000 7f75e3b1b700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:30.979 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.973+0000 7f75e3b1b700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75dc0aa710 con 0x7f75dc0a5f40 2023-09-30T02:11:30.979 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.973+0000 7f75e3b1b700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:30.979 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.973+0000 7f75e3b1b700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75dc14be80 con 0x7f75dc1420c0 2023-09-30T02:11:30.979 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.973+0000 7f75e3b1b700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:30.979 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.973+0000 7f75e3b1b700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75dc0a8ca0 con 0x7f75dc0a5b70 2023-09-30T02:11:30.979 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.973+0000 7f75e3b1b700 10 monclient(hunting): _renew_subs 2023-09-30T02:11:30.979 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.973+0000 7f75e3b1b700 10 monclient(hunting): authenticate will time out at 2136.275492s 2023-09-30T02:11:30.979 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.974+0000 7f75e20b8700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f75dc1420c0 0x7f75dc0a78a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:30.980 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.974+0000 7f75e18b7700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f75dc0a5b70 0x7f75dc141100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:30.980 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.974+0000 7f75e10b6700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75dc0a5f40 0x7f75dc151870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:30.980 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.974+0000 7f75e20b8700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f75dc1420c0 0x7f75dc0a78a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:60450/0 (socket says 172.21.15.59:60450) 2023-09-30T02:11:30.980 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.974+0000 7f75e20b8700 1 -- 172.21.15.59:0/514637314 learned_addr learned my addr 172.21.15.59:0/514637314 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:11:30.980 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.974+0000 7f75e20b8700 10 monclient(hunting): get_auth_request con 0x7f75dc1420c0 auth_method 0 2023-09-30T02:11:30.980 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.974+0000 7f75e20b8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:30.980 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.974+0000 7f75e20b8700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:30.980 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.974+0000 7f75e20b8700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:30.980 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.974+0000 7f75e18b7700 10 monclient(hunting): get_auth_request con 0x7f75dc0a5b70 auth_method 0 2023-09-30T02:11:30.980 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.974+0000 7f75e18b7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:30.980 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.974+0000 7f75e18b7700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:30.980 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.974+0000 7f75e18b7700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:30.981 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.974+0000 7f75e10b6700 10 monclient(hunting): get_auth_request con 0x7f75dc0a5f40 auth_method 0 2023-09-30T02:11:30.981 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.974+0000 7f75e10b6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:30.981 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.974+0000 7f75e10b6700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:30.981 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.974+0000 7f75e10b6700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:30.981 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.974+0000 7f75e20b8700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:30.981 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.974+0000 7f75e20b8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:30.981 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.974+0000 7f75e20b8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:30.981 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.975+0000 7f75e18b7700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:30.981 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.975+0000 7f75e18b7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:30.981 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.975+0000 7f75e18b7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:30.981 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.975+0000 7f75e10b6700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:30.981 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.975+0000 7f75e10b6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:30.982 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.975+0000 7f75e10b6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:30.982 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.978+0000 7f75e20b8700 10 monclient(hunting): handle_auth_done global_id 4861 payload 824 2023-09-30T02:11:30.983 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.978+0000 7f75e20b8700 1 -- 172.21.15.59:0/514637314 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f75dc0a5b70 msgr2=0x7f75dc141100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:30.983 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.978+0000 7f75e20b8700 1 --2- 172.21.15.59:0/514637314 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f75dc0a5b70 0x7f75dc141100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:30.983 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.978+0000 7f75e20b8700 1 -- 172.21.15.59:0/514637314 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75dc0a5f40 msgr2=0x7f75dc151870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:30.983 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.978+0000 7f75e20b8700 1 --2- 172.21.15.59:0/514637314 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75dc0a5f40 0x7f75dc151870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:30.983 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.978+0000 7f75e20b8700 10 monclient: _finish_hunting 0 2023-09-30T02:11:30.983 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.978+0000 7f75e20b8700 1 monclient: found mon.noname-b 2023-09-30T02:11:30.983 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.978+0000 7f75e20b8700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:11:30.983 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.978+0000 7f75e20b8700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:11:30.983 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.978+0000 7f75e20b8700 1 -- 172.21.15.59:0/514637314 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75cc009420 con 0x7f75dc1420c0 2023-09-30T02:11:30.983 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.980+0000 7f75e20b8700 1 --2- 172.21.15.59:0/514637314 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f75dc1420c0 0x7f75dc0a78a0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f75d400cbc0 tx=0x7f75d40019c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:30.983 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.980+0000 7f75d1ffb700 10 client.?.objecter ms_handle_connect 0x7f75dc1420c0 2023-09-30T02:11:30.984 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.980+0000 7f75d1ffb700 10 client.?.objecter resend_mon_ops 2023-09-30T02:11:30.985 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.984+0000 7f75d1ffb700 1 -- 172.21.15.59:0/514637314 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75d400a7e0 con 0x7f75dc1420c0 2023-09-30T02:11:30.985 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.984+0000 7f75d1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:30.985 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.984+0000 7f75d1ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:11:30.985 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.984+0000 7f75d1ffb700 10 monclient: dump: 2023-09-30T02:11:30.985 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:30.985 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:30.985 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:30.985 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:30.986 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:30.986 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:30.986 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:30.986 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:30.986 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:30.986 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:30.986 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.984+0000 7f75d1ffb700 10 monclient: _finish_auth 0 2023-09-30T02:11:30.986 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.984+0000 7f75d1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:11:30.986 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.984+0000 7f75e3b1b700 5 monclient: authenticate success, global_id 4861 2023-09-30T02:11:30.986 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.984+0000 7f75d1ffb700 1 -- 172.21.15.59:0/514637314 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75d400a940 con 0x7f75dc1420c0 2023-09-30T02:11:30.986 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.984+0000 7f75d1ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:11:30.986 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.984+0000 7f75e3b1b700 10 monclient: _renew_subs 2023-09-30T02:11:30.987 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.984+0000 7f75d1ffb700 1 -- 172.21.15.59:0/514637314 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75d400acc0 con 0x7f75dc1420c0 2023-09-30T02:11:30.987 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.984+0000 7f75e3b1b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:11:30.987 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.984+0000 7f75e3b1b700 1 -- 172.21.15.59:0/514637314 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75dc0aa230 con 0x7f75dc1420c0 2023-09-30T02:11:30.987 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.984+0000 7f75d1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:30.987 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.984+0000 7f75d1ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:11:30.987 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.984+0000 7f75d3fff700 4 set_mon_vals no callback set 2023-09-30T02:11:30.987 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.984+0000 7f75d1ffb700 10 monclient: dump: 2023-09-30T02:11:30.987 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:30.987 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:30.987 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:30.987 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:30.987 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:30.988 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:30.988 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:30.988 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:30.988 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:30.988 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:30.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.984+0000 7f75e3b1b700 10 client.4861.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:11:30.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.984+0000 7f75e3b1b700 10 monclient: _renew_subs 2023-09-30T02:11:30.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.984+0000 7f75e3b1b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:11:30.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.984+0000 7f75e3b1b700 1 -- 172.21.15.59:0/514637314 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75dc151f70 con 0x7f75dc1420c0 2023-09-30T02:11:30.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.984+0000 7f75e3b1b700 1 librados: init done 2023-09-30T02:11:30.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.985+0000 7f75e3b1b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:11:30.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.985+0000 7f75e3b1b700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:11:30.989 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.985+0000 7f75e3b1b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:11:30.989 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:30.985+0000 7f75e3b1b700 1 -- 172.21.15.59:0/514637314 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75dc0aa880 con 0x7f75dc1420c0 2023-09-30T02:11:31.009 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.009+0000 7f75d1ffb700 1 -- 172.21.15.59:0/514637314 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f75d4007300 con 0x7f75dc1420c0 2023-09-30T02:11:31.009 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.010+0000 7f75d1ffb700 1 --2- 172.21.15.59:0/514637314 >> v2:172.21.15.110:6800/115870 conn(0x7f75c0068910 0x7f75c006adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:31.010 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.010+0000 7f75d1ffb700 1 -- 172.21.15.59:0/514637314 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f75d4084c10 con 0x7f75dc1420c0 2023-09-30T02:11:31.010 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.010+0000 7f75d1ffb700 10 client.4861.objecter ms_dispatch 0x7f75dc001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:11:31.010 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.010+0000 7f75d1ffb700 3 client.4861.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:11:31.010 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.010+0000 7f75d1ffb700 3 client.4861.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:11:31.010 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.010+0000 7f75e18b7700 1 --2- 172.21.15.59:0/514637314 >> v2:172.21.15.110:6800/115870 conn(0x7f75c0068910 0x7f75c006adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:31.010 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.010+0000 7f75d1ffb700 20 client.4861.objecter dump_active .. 0 homeless 2023-09-30T02:11:31.010 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.010+0000 7f75e18b7700 10 monclient: get_auth_request con 0x7f75c0068910 auth_method 0 2023-09-30T02:11:31.010 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.010+0000 7f75d1ffb700 10 client.4861.objecter ms_handle_connect 0x7f75c0068910 2023-09-30T02:11:31.011 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.010+0000 7f75e18b7700 1 --2- 172.21.15.59:0/514637314 >> v2:172.21.15.110:6800/115870 conn(0x7f75c0068910 0x7f75c006adc0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f75d8009db0 tx=0x7f75d80091a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:31.243 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.244+0000 7f75d1ffb700 1 -- 172.21.15.59:0/514637314 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f75d4052530 con 0x7f75dc1420c0 2023-09-30T02:11:31.243 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.244+0000 7f75d1ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:11:31.243 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.244+0000 7f75d1ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:11:31.443 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.444+0000 7f75e3b1b700 1 -- 172.21.15.59:0/514637314 --> v2:172.21.15.110:6800/115870 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f75dc064a00 con 0x7f75c0068910 2023-09-30T02:11:31.447 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.448+0000 7f75d1ffb700 1 -- 172.21.15.59:0/514637314 <== mgr.4108 v2:172.21.15.110:6800/115870 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+70500 (secure 0 0 0) 0x7f75dc064a00 con 0x7f75c0068910 2023-09-30T02:11:31.448 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:11:31.450 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.450+0000 7f75e3b1b700 10 librados: watch_flush enter 2023-09-30T02:11:31.450 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.450+0000 7f75e3b1b700 10 librados: watch_flush exit 2023-09-30T02:11:31.450 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.450+0000 7f75e3b1b700 20 client.4861.objecter shutdown clearing up homeless session... 2023-09-30T02:11:31.450 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.450+0000 7f75e3b1b700 10 client.4861.objecter successfully canceled tick 2023-09-30T02:11:31.450 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.450+0000 7f75e3b1b700 1 -- 172.21.15.59:0/514637314 >> v2:172.21.15.110:6800/115870 conn(0x7f75c0068910 msgr2=0x7f75c006adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:31.450 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.450+0000 7f75e3b1b700 1 --2- 172.21.15.59:0/514637314 >> v2:172.21.15.110:6800/115870 conn(0x7f75c0068910 0x7f75c006adc0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f75d8009db0 tx=0x7f75d80091a0 comp rx=0 tx=0).stop 2023-09-30T02:11:31.450 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.450+0000 7f75e3b1b700 10 monclient: shutdown 2023-09-30T02:11:31.450 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.450+0000 7f75e3b1b700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:11:31.450 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.450+0000 7f75e3b1b700 1 -- 172.21.15.59:0/514637314 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f75dc1420c0 msgr2=0x7f75dc0a78a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:31.451 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.450+0000 7f75e3b1b700 1 --2- 172.21.15.59:0/514637314 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f75dc1420c0 0x7f75dc0a78a0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f75d400cbc0 tx=0x7f75d40019c0 comp rx=0 tx=0).stop 2023-09-30T02:11:31.451 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.450+0000 7f75e3b1b700 1 -- 172.21.15.59:0/514637314 shutdown_connections 2023-09-30T02:11:31.451 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.450+0000 7f75e3b1b700 1 --2- 172.21.15.59:0/514637314 >> v2:172.21.15.110:6800/115870 conn(0x7f75c0068910 0x7f75c006adc0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:31.451 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.450+0000 7f75e3b1b700 1 --2- 172.21.15.59:0/514637314 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f75dc1420c0 0x7f75dc0a78a0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:31.451 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.450+0000 7f75e3b1b700 1 --2- 172.21.15.59:0/514637314 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f75dc0a5b70 0x7f75dc141100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:31.451 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.450+0000 7f75e3b1b700 1 --2- 172.21.15.59:0/514637314 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f75dc0a5f40 0x7f75dc151870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:31.451 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.450+0000 7f75e3b1b700 1 -- 172.21.15.59:0/514637314 >> 172.21.15.59:0/514637314 conn(0x7f75dc0631f0 msgr2=0x7f75dc137a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:11:31.451 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.450+0000 7f75e3b1b700 1 -- 172.21.15.59:0/514637314 shutdown_connections 2023-09-30T02:11:31.451 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.450+0000 7f75e3b1b700 1 -- 172.21.15.59:0/514637314 wait complete. 2023-09-30T02:11:31.451 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.450+0000 7f75e3b1b700 1 librados: shutdown 2023-09-30T02:11:31.452 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2023-09-30T02:11:31.463 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":193,"stamp":"2023-09-30T02:11:29.647188+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":43464,"kb_used_data":2448,"kb_used_omap":0,"kb_used_meta":40576,"kb_avail":754931256,"statfs":{"total":773094113280,"available":773049606144,"internally_reserved":0,"allocated":2506752,"data_stored":1396000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":41549824},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002809"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T02:06:18.350818+0000","last_change":"2023-09-30T02:06:18.210390+0000","last_active":"2023-09-30T02:06:18.350818+0000","last_peered":"2023-09-30T02:06:18.350818+0000","last_clean":"2023-09-30T02:06:18.350818+0000","last_became_active":"2023-09-30T02:06:18.198153+0000","last_became_peered":"2023-09-30T02:06:18.198153+0000","last_unstale":"2023-09-30T02:06:18.350818+0000","last_undegraded":"2023-09-30T02:06:18.350818+0000","last_fullsized":"2023-09-30T02:06:18.350818+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_clean_scrub_stamp":"2023-09-30T02:06:15.800912+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:42:32.739379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T02:06:18.437839+0000","last_change":"2023-09-30T02:06:18.262708+0000","last_active":"2023-09-30T02:06:18.437839+0000","last_peered":"2023-09-30T02:06:18.437839+0000","last_clean":"2023-09-30T02:06:18.437839+0000","last_became_active":"2023-09-30T02:06:18.224464+0000","last_became_peered":"2023-09-30T02:06:18.224464+0000","last_unstale":"2023-09-30T02:06:18.437839+0000","last_undegraded":"2023-09-30T02:06:18.437839+0000","last_fullsized":"2023-09-30T02:06:18.437839+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_clean_scrub_stamp":"2023-09-30T02:06:15.800912+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:13:39.084627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T02:06:18.364779+0000","last_change":"2023-09-30T02:06:17.910244+0000","last_active":"2023-09-30T02:06:18.364779+0000","last_peered":"2023-09-30T02:06:18.364779+0000","last_clean":"2023-09-30T02:06:18.364779+0000","last_became_active":"2023-09-30T02:06:17.855689+0000","last_became_peered":"2023-09-30T02:06:17.855689+0000","last_unstale":"2023-09-30T02:06:18.364779+0000","last_undegraded":"2023-09-30T02:06:18.364779+0000","last_fullsized":"2023-09-30T02:06:18.364779+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_clean_scrub_stamp":"2023-09-30T02:06:15.800912+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:18:08.185701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T02:06:18.359472+0000","last_change":"2023-09-30T02:06:17.806354+0000","last_active":"2023-09-30T02:06:18.359472+0000","last_peered":"2023-09-30T02:06:18.359472+0000","last_clean":"2023-09-30T02:06:18.359472+0000","last_became_active":"2023-09-30T02:06:17.706615+0000","last_became_peered":"2023-09-30T02:06:17.706615+0000","last_unstale":"2023-09-30T02:06:18.359472+0000","last_undegraded":"2023-09-30T02:06:18.359472+0000","last_fullsized":"2023-09-30T02:06:18.359472+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_clean_scrub_stamp":"2023-09-30T02:06:15.800912+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:08:20.786129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"25'76","reported_seq":117,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-09-30T02:06:21.647351+0000","last_change":"2023-09-30T02:06:11.645599+0000","last_active":"2023-09-30T02:06:21.647351+0000","last_peered":"2023-09-30T02:06:21.647351+0000","last_clean":"2023-09-30T02:06:21.647351+0000","last_became_active":"2023-09-30T02:06:11.592631+0000","last_became_peered":"2023-09-30T02:06:11.592631+0000","last_unstale":"2023-09-30T02:06:21.647351+0000","last_undegraded":"2023-09-30T02:06:21.647351+0000","last_fullsized":"2023-09-30T02:06:21.647351+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:06:06.666529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:06:06.666529+0000","last_clean_scrub_stamp":"2023-09-30T02:06:06.666529+0000","objects_scrubbed":0,"log_size":76,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:16:35.859435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T02:06:18.343393+0000","last_change":"2023-09-30T02:06:18.102959+0000","last_active":"2023-09-30T02:06:18.343393+0000","last_peered":"2023-09-30T02:06:18.343393+0000","last_clean":"2023-09-30T02:06:18.343393+0000","last_became_active":"2023-09-30T02:06:18.091407+0000","last_became_peered":"2023-09-30T02:06:18.091407+0000","last_unstale":"2023-09-30T02:06:18.343393+0000","last_undegraded":"2023-09-30T02:06:18.343393+0000","last_fullsized":"2023-09-30T02:06:18.343393+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_clean_scrub_stamp":"2023-09-30T02:06:15.800912+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:34:28.390487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T02:06:18.250861+0000","last_change":"2023-09-30T02:06:17.465018+0000","last_active":"2023-09-30T02:06:18.250861+0000","last_peered":"2023-09-30T02:06:18.250861+0000","last_clean":"2023-09-30T02:06:18.250861+0000","last_became_active":"2023-09-30T02:06:17.420574+0000","last_became_peered":"2023-09-30T02:06:17.420574+0000","last_unstale":"2023-09-30T02:06:18.250861+0000","last_undegraded":"2023-09-30T02:06:18.250861+0000","last_fullsized":"2023-09-30T02:06:18.250861+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_clean_scrub_stamp":"2023-09-30T02:06:15.800912+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:46:28.414555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T02:06:18.354965+0000","last_change":"2023-09-30T02:06:17.896913+0000","last_active":"2023-09-30T02:06:18.354965+0000","last_peered":"2023-09-30T02:06:18.354965+0000","last_clean":"2023-09-30T02:06:18.354965+0000","last_became_active":"2023-09-30T02:06:17.890282+0000","last_became_peered":"2023-09-30T02:06:17.890282+0000","last_unstale":"2023-09-30T02:06:18.354965+0000","last_undegraded":"2023-09-30T02:06:18.354965+0000","last_fullsized":"2023-09-30T02:06:18.354965+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_clean_scrub_stamp":"2023-09-30T02:06:15.800912+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:35:55.671531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-09-30T02:06:17.900193+0000","last_change":"2023-09-30T02:06:17.877329+0000","last_active":"2023-09-30T02:06:17.900193+0000","last_peered":"2023-09-30T02:06:17.900193+0000","last_clean":"2023-09-30T02:06:17.900193+0000","last_became_active":"2023-09-30T02:06:17.836124+0000","last_became_peered":"2023-09-30T02:06:17.836124+0000","last_unstale":"2023-09-30T02:06:17.900193+0000","last_undegraded":"2023-09-30T02:06:17.900193+0000","last_fullsized":"2023-09-30T02:06:17.900193+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_clean_scrub_stamp":"2023-09-30T02:06:15.800912+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T14:04:06.678156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":19,"seq":81604378691,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5208,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366632,"statfs":{"total":96636764160,"available":96631431168,"internally_reserved":0,"allocated":212992,"data_stored":75040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":12.859999999999999,"5min":16.204000000000001,"15min":26.364000000000001},"min":{"1min":8.8520000000000003,"5min":7.1429999999999998,"15min":7.1429999999999998},"max":{"1min":17.263000000000002,"5min":414.25400000000002,"15min":414.25400000000002},"last":17.838999999999999},{"interface":"front","average":{"1min":12.984,"5min":17.393999999999998,"15min":29.042999999999999},"min":{"1min":7.1289999999999996,"5min":7.1289999999999996,"15min":7.1289999999999996},"max":{"1min":21.555,"5min":421.178,"15min":421.178},"last":12.563000000000001}]},{"osd":1,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":13.353,"5min":15.817,"15min":24.888999999999999},"min":{"1min":8.9610000000000003,"5min":6.8200000000000003,"15min":6.8200000000000003},"max":{"1min":16.489999999999998,"5min":326.60899999999998,"15min":326.60899999999998},"last":11.785},{"interface":"front","average":{"1min":14.826000000000001,"5min":19.149999999999999,"15min":31.222999999999999},"min":{"1min":8.5099999999999998,"5min":8.5079999999999991,"15min":8.5079999999999991},"max":{"1min":24.559999999999999,"5min":418.35500000000002,"15min":418.35500000000002},"last":9.3290000000000006}]},{"osd":2,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":16.120000000000001,"5min":21.582000000000001,"15min":33.603999999999999},"min":{"1min":9.2850000000000001,"5min":8.3960000000000008,"15min":8.3960000000000008},"max":{"1min":23.693999999999999,"5min":423.46699999999998,"15min":423.46699999999998},"last":13.65},{"interface":"front","average":{"1min":14.952,"5min":19.960999999999999,"15min":33.359999999999999},"min":{"1min":9.0749999999999993,"5min":8.4000000000000004,"15min":8.4000000000000004},"max":{"1min":23.123000000000001,"5min":438.697,"15min":438.697},"last":16.091000000000001}]},{"osd":4,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":15.225,"5min":19.898,"15min":32.472000000000001},"min":{"1min":9.5370000000000008,"5min":8.4030000000000005,"15min":8.4030000000000005},"max":{"1min":26.390000000000001,"5min":443.36399999999998,"15min":443.36399999999998},"last":13.970000000000001},{"interface":"front","average":{"1min":17.562999999999999,"5min":22.780999999999999,"15min":35.773000000000003},"min":{"1min":10.282,"5min":10.282,"15min":10.282},"max":{"1min":23.670000000000002,"5min":453.20699999999999,"15min":453.20699999999999},"last":20.038}]},{"osd":5,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":14.875,"5min":20.896999999999998,"15min":33.405000000000001},"min":{"1min":8.2430000000000003,"5min":8.141,"15min":8.141},"max":{"1min":22.359000000000002,"5min":443.94600000000003,"15min":443.94600000000003},"last":18.710000000000001},{"interface":"front","average":{"1min":19.251000000000001,"5min":22.654,"15min":32.534999999999997},"min":{"1min":11.301,"5min":10.746,"15min":10.746},"max":{"1min":25.305,"5min":335.11799999999999,"15min":335.11799999999999},"last":22.116}]},{"osd":6,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":18.507000000000001,"5min":22.931999999999999,"15min":36.216999999999999},"min":{"1min":12.955,"5min":9.4009999999999998,"15min":9.4009999999999998},"max":{"1min":28.02,"5min":432.34699999999998,"15min":432.34699999999998},"last":23.873999999999999},{"interface":"front","average":{"1min":21.754000000000001,"5min":24.712,"15min":34.631999999999998},"min":{"1min":18.292999999999999,"5min":13.018000000000001,"15min":13.018000000000001},"max":{"1min":27.356999999999999,"5min":334.178,"15min":334.178},"last":21.161000000000001}]},{"osd":7,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":17.149000000000001,"5min":23.225000000000001,"15min":35.642000000000003},"min":{"1min":10.933999999999999,"5min":10.928000000000001,"15min":10.928000000000001},"max":{"1min":23.986999999999998,"5min":461.42000000000002,"15min":461.42000000000002},"last":16.986000000000001},{"interface":"front","average":{"1min":22.254000000000001,"5min":26.744,"15min":39.024000000000001},"min":{"1min":17.257000000000001,"5min":15.686,"15min":15.686},"max":{"1min":25.774000000000001,"5min":431.27800000000002,"15min":431.27800000000002},"last":23.109999999999999}]}]},{"osd":1,"up_from":18,"seq":77309411394,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6044,"kb_used_data":600,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":94365796,"statfs":{"total":96636764160,"available":96630575104,"internally_reserved":0,"allocated":614400,"data_stored":472880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":13.101000000000001,"5min":13.603999999999999,"15min":13.84},"min":{"1min":8.7560000000000002,"5min":7.1639999999999997,"15min":7.1639999999999997},"max":{"1min":19.596,"5min":24.626999999999999,"15min":24.626999999999999},"last":12.608000000000001},{"interface":"front","average":{"1min":11.407,"5min":12.699,"15min":14.554},"min":{"1min":6.4859999999999998,"5min":6.4859999999999998,"15min":6.4859999999999998},"max":{"1min":17.785,"5min":38.213999999999999,"15min":38.213999999999999},"last":11.879}]},{"osd":2,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":12.917999999999999,"5min":13.146000000000001,"15min":13.359999999999999},"min":{"1min":8.4949999999999992,"5min":6.9509999999999996,"15min":6.9509999999999996},"max":{"1min":20.673999999999999,"5min":39.070999999999998,"15min":39.070999999999998},"last":11.021000000000001},{"interface":"front","average":{"1min":14.161,"5min":15.342000000000001,"15min":16.507999999999999},"min":{"1min":8.6050000000000004,"5min":8.6050000000000004,"15min":8.6050000000000004},"max":{"1min":20.436,"5min":28.981999999999999,"15min":28.981999999999999},"last":8.6050000000000004}]},{"osd":3,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":18.742999999999999,"5min":28.812999999999999,"15min":56.773000000000003},"min":{"1min":13.654999999999999,"5min":9.2110000000000003,"15min":9.2110000000000003},"max":{"1min":23.431999999999999,"5min":789.26099999999997,"15min":789.26099999999997},"last":14.082000000000001},{"interface":"front","average":{"1min":16.210999999999999,"5min":26.545999999999999,"15min":55.965000000000003},"min":{"1min":7.8869999999999996,"5min":7.8869999999999996,"15min":7.8869999999999996},"max":{"1min":24.469000000000001,"5min":827.53899999999999,"15min":827.53899999999999},"last":9.9890000000000008}]},{"osd":4,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":14.420999999999999,"5min":21.696999999999999,"15min":34.310000000000002},"min":{"1min":11.004,"5min":8.7029999999999994,"15min":8.7029999999999994},"max":{"1min":20.788,"5min":390.31700000000001,"15min":390.31700000000001},"last":14.851000000000001},{"interface":"front","average":{"1min":17.638000000000002,"5min":21.739000000000001,"15min":34.366999999999997},"min":{"1min":10.234,"5min":10.234,"15min":10.234},"max":{"1min":24.907,"5min":381.255,"15min":381.255},"last":18.593}]},{"osd":5,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":13.234999999999999,"5min":14.538,"15min":15.808999999999999},"min":{"1min":9.2680000000000007,"5min":7.6849999999999996,"15min":7.6849999999999996},"max":{"1min":21.724,"5min":31.677,"15min":31.677},"last":17.120999999999999},{"interface":"front","average":{"1min":20.417000000000002,"5min":20.757000000000001,"15min":21.004999999999999},"min":{"1min":9.9860000000000007,"5min":9.9860000000000007,"15min":9.9860000000000007},"max":{"1min":26.379000000000001,"5min":31.449999999999999,"15min":31.449999999999999},"last":15.631}]},{"osd":6,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":18.373999999999999,"5min":19.23,"15min":21.132000000000001},"min":{"1min":12.045,"5min":9.7560000000000002,"15min":9.7560000000000002},"max":{"1min":24.495999999999999,"5min":41.316000000000003,"15min":41.316000000000003},"last":17.870999999999999},{"interface":"front","average":{"1min":16.523,"5min":17.629999999999999,"15min":19.443999999999999},"min":{"1min":10.407,"5min":9.2699999999999996,"15min":9.2699999999999996},"max":{"1min":22.161999999999999,"5min":53.012,"15min":53.012},"last":21.864999999999998}]},{"osd":7,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":19.762,"5min":24.777999999999999,"15min":37.265999999999998},"min":{"1min":14.295999999999999,"5min":9.5310000000000006,"15min":9.5310000000000006},"max":{"1min":23.609000000000002,"5min":332.072,"15min":332.072},"last":21.032},{"interface":"front","average":{"1min":17.829000000000001,"5min":20.178999999999998,"15min":24.689},"min":{"1min":11.316000000000001,"5min":8.141,"15min":8.141},"max":{"1min":23.631,"5min":108.00700000000001,"15min":108.00700000000001},"last":19.300000000000001}]}]},{"osd":7,"up_from":14,"seq":60129542212,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5272,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":94366568,"statfs":{"total":96636764160,"available":96631365632,"internally_reserved":0,"allocated":212992,"data_stored":75040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:11:21 2023","interfaces":[{"interface":"back","average":{"1min":8.6750000000000007,"5min":8.9369999999999994,"15min":8.9199999999999999},"min":{"1min":6.2140000000000004,"5min":5.3120000000000003,"15min":5.3120000000000003},"max":{"1min":16.916,"5min":19.491,"15min":19.491},"last":9.1519999999999992},{"interface":"front","average":{"1min":10.153,"5min":9.9879999999999995,"15min":9.7560000000000002},"min":{"1min":7.3099999999999996,"5min":6.2850000000000001,"15min":6.2850000000000001},"max":{"1min":19.263000000000002,"5min":20.952000000000002,"15min":20.952000000000002},"last":19.263000000000002}]},{"osd":1,"last update":"Sat Sep 30 02:11:21 2023","interfaces":[{"interface":"back","average":{"1min":10.753,"5min":11.709,"15min":14.192},"min":{"1min":6.6630000000000003,"5min":6.6630000000000003,"15min":6.6630000000000003},"max":{"1min":18.591000000000001,"5min":89.454999999999998,"15min":89.454999999999998},"last":11.798999999999999},{"interface":"front","average":{"1min":11.548999999999999,"5min":12.834,"15min":14.573},"min":{"1min":8.0909999999999993,"5min":7.0700000000000003,"15min":7.0700000000000003},"max":{"1min":19.093,"5min":83.695999999999998,"15min":83.695999999999998},"last":10.634}]},{"osd":2,"last update":"Sat Sep 30 02:11:21 2023","interfaces":[{"interface":"back","average":{"1min":13.856,"5min":13.919,"15min":13.823},"min":{"1min":8.1240000000000006,"5min":7.4669999999999996,"15min":7.4669999999999996},"max":{"1min":20.259,"5min":21.370999999999999,"15min":21.370999999999999},"last":20.259},{"interface":"front","average":{"1min":14.368,"5min":14.385,"15min":15.127000000000001},"min":{"1min":10.176,"5min":7.8920000000000003,"15min":7.8920000000000003},"max":{"1min":19.542999999999999,"5min":21.712,"15min":21.712},"last":13.757999999999999}]},{"osd":3,"last update":"Sat Sep 30 02:11:21 2023","interfaces":[{"interface":"back","average":{"1min":15.417999999999999,"5min":15.926,"15min":15.646000000000001},"min":{"1min":11.319000000000001,"5min":7.1710000000000003,"15min":7.1710000000000003},"max":{"1min":21.238,"5min":24.276,"15min":24.276},"last":21.238},{"interface":"front","average":{"1min":16.335999999999999,"5min":16.087,"15min":16.295000000000002},"min":{"1min":12.477,"5min":7.9630000000000001,"15min":7.9630000000000001},"max":{"1min":22.021999999999998,"5min":24.716000000000001,"15min":24.716000000000001},"last":15.807}]},{"osd":4,"last update":"Sat Sep 30 02:11:12 2023","interfaces":[{"interface":"back","average":{"1min":16.693999999999999,"5min":17.527000000000001,"15min":19.007999999999999},"min":{"1min":9.4380000000000006,"5min":5.7039999999999997,"15min":5.7039999999999997},"max":{"1min":23.076000000000001,"5min":99.298000000000002,"15min":99.298000000000002},"last":14.891999999999999},{"interface":"front","average":{"1min":12.294,"5min":15.105,"15min":17.399000000000001},"min":{"1min":7.1509999999999998,"5min":7.1509999999999998,"15min":7.1509999999999998},"max":{"1min":17.899000000000001,"5min":86.938999999999993,"15min":86.938999999999993},"last":13.117000000000001}]},{"osd":5,"last update":"Sat Sep 30 02:11:12 2023","interfaces":[{"interface":"back","average":{"1min":15.673999999999999,"5min":17.282,"15min":19.91},"min":{"1min":8.6509999999999998,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":21.527000000000001,"5min":118.675,"15min":118.675},"last":22.318000000000001},{"interface":"front","average":{"1min":18.716999999999999,"5min":19.459,"15min":21.693999999999999},"min":{"1min":16.239999999999998,"5min":8.7330000000000005,"15min":8.7330000000000005},"max":{"1min":23.338999999999999,"5min":114.14400000000001,"15min":114.14400000000001},"last":17.053000000000001}]},{"osd":6,"last update":"Sat Sep 30 02:11:12 2023","interfaces":[{"interface":"back","average":{"1min":19.045000000000002,"5min":20.443999999999999,"15min":23.024000000000001},"min":{"1min":8.4359999999999999,"5min":8.4359999999999999,"15min":8.4359999999999999},"max":{"1min":24.417000000000002,"5min":111.048,"15min":111.048},"last":17.846},{"interface":"front","average":{"1min":18.745000000000001,"5min":19.427,"15min":22.183},"min":{"1min":12.542,"5min":9.8140000000000001,"15min":9.8140000000000001},"max":{"1min":23.036000000000001,"5min":115.241,"15min":115.241},"last":23.251999999999999}]}]},{"osd":0,"up_from":18,"seq":77309411395,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5208,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366632,"statfs":{"total":96636764160,"available":96631431168,"internally_reserved":0,"allocated":212992,"data_stored":75040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 30 02:11:19 2023","interfaces":[{"interface":"back","average":{"1min":10.754,"5min":16.196000000000002,"15min":29.155000000000001},"min":{"1min":7.2089999999999996,"5min":7.2089999999999996,"15min":7.2089999999999996},"max":{"1min":19.498999999999999,"5min":425.49599999999998,"15min":425.49599999999998},"last":12.119999999999999},{"interface":"front","average":{"1min":13.093,"5min":20.338000000000001,"15min":39.884},"min":{"1min":8.1210000000000004,"5min":7.9269999999999996,"15min":7.9269999999999996},"max":{"1min":18.349,"5min":667.81100000000004,"15min":667.81100000000004},"last":9.6829999999999998}]},{"osd":2,"last update":"Sat Sep 30 02:11:19 2023","interfaces":[{"interface":"back","average":{"1min":12.025,"5min":14.532999999999999,"15min":18.928999999999998},"min":{"1min":8.2110000000000003,"5min":7.718,"15min":7.718},"max":{"1min":16.954000000000001,"5min":162.50700000000001,"15min":162.50700000000001},"last":9.0099999999999998},{"interface":"front","average":{"1min":17.457000000000001,"5min":17.991,"15min":20.966999999999999},"min":{"1min":10.747,"5min":8.5079999999999991,"15min":8.5079999999999991},"max":{"1min":23.617000000000001,"5min":131.863,"15min":131.863},"last":14.253}]},{"osd":3,"last update":"Sat Sep 30 02:11:19 2023","interfaces":[{"interface":"back","average":{"1min":13.837,"5min":14.763,"15min":15.271000000000001},"min":{"1min":8.1020000000000003,"5min":7.4859999999999998,"15min":7.4859999999999998},"max":{"1min":22.571999999999999,"5min":24.763000000000002,"15min":24.763000000000002},"last":7.7300000000000004},{"interface":"front","average":{"1min":18.533999999999999,"5min":18.309999999999999,"15min":18.783000000000001},"min":{"1min":10.459,"5min":9.8569999999999993,"15min":9.8569999999999993},"max":{"1min":23.57,"5min":27.309000000000001,"15min":27.309000000000001},"last":11.268000000000001}]},{"osd":4,"last update":"Sat Sep 30 02:11:19 2023","interfaces":[{"interface":"back","average":{"1min":16.027999999999999,"5min":16.099,"15min":19.271000000000001},"min":{"1min":9.9179999999999993,"5min":9.0700000000000003,"15min":9.0700000000000003},"max":{"1min":22.122,"5min":102.456,"15min":102.456},"last":15.006},{"interface":"front","average":{"1min":13.881,"5min":17.271000000000001,"15min":20.933},"min":{"1min":8.7929999999999993,"5min":8.7929999999999993,"15min":8.7929999999999993},"max":{"1min":19.463999999999999,"5min":98.293999999999997,"15min":98.293999999999997},"last":12.898999999999999}]},{"osd":5,"last update":"Sat Sep 30 02:11:19 2023","interfaces":[{"interface":"back","average":{"1min":20.268999999999998,"5min":20.559999999999999,"15min":22.850000000000001},"min":{"1min":14.006,"5min":10.726000000000001,"15min":10.726000000000001},"max":{"1min":28.995000000000001,"5min":79.566999999999993,"15min":79.566999999999993},"last":19.181999999999999},{"interface":"front","average":{"1min":16.812999999999999,"5min":18.582999999999998,"15min":21.466999999999999},"min":{"1min":9.7599999999999998,"5min":9.7599999999999998,"15min":9.7599999999999998},"max":{"1min":24.213000000000001,"5min":109.80500000000001,"15min":109.80500000000001},"last":17.227}]},{"osd":6,"last update":"Sat Sep 30 02:11:19 2023","interfaces":[{"interface":"back","average":{"1min":18.077999999999999,"5min":20.367000000000001,"15min":26.062999999999999},"min":{"1min":11.140000000000001,"5min":9.2829999999999995,"15min":9.2829999999999995},"max":{"1min":24.276,"5min":161.01499999999999,"15min":161.01499999999999},"last":13.56},{"interface":"front","average":{"1min":22.562000000000001,"5min":22.975000000000001,"15min":26.202999999999999},"min":{"1min":13.904,"5min":13.163,"15min":13.163},"max":{"1min":27.786000000000001,"5min":118.71599999999999,"15min":118.71599999999999},"last":17.969000000000001}]},{"osd":7,"last update":"Sat Sep 30 02:11:19 2023","interfaces":[{"interface":"back","average":{"1min":20.117999999999999,"5min":20.393000000000001,"15min":23.483000000000001},"min":{"1min":11.741,"5min":9.8810000000000002,"15min":9.8810000000000002},"max":{"1min":26.491,"5min":111.86,"15min":111.86},"last":19.989000000000001},{"interface":"front","average":{"1min":21.664000000000001,"5min":22.388999999999999,"15min":26.010999999999999},"min":{"1min":15.186999999999999,"5min":10.237,"15min":10.237},"max":{"1min":27.012,"5min":126.889,"15min":126.889},"last":21.155000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574915,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5208,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366632,"statfs":{"total":96636764160,"available":96631431168,"internally_reserved":0,"allocated":212992,"data_stored":75040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:11:20 2023","interfaces":[{"interface":"back","average":{"1min":9.6969999999999992,"5min":9.8569999999999993,"15min":9.7870000000000008},"min":{"1min":6.5830000000000002,"5min":5.774,"15min":5.774},"max":{"1min":16.530999999999999,"5min":19.709,"15min":19.709},"last":6.5439999999999996},{"interface":"front","average":{"1min":10.919,"5min":10.390000000000001,"15min":11.138999999999999},"min":{"1min":6.1239999999999997,"5min":5.0419999999999998,"15min":5.0419999999999998},"max":{"1min":19.603000000000002,"5min":23.134,"15min":23.134},"last":9.2300000000000004}]},{"osd":1,"last update":"Sat Sep 30 02:11:20 2023","interfaces":[{"interface":"back","average":{"1min":11.829000000000001,"5min":12.981999999999999,"15min":14.561999999999999},"min":{"1min":7.351,"5min":6.3010000000000002,"15min":6.3010000000000002},"max":{"1min":17.946999999999999,"5min":27.242999999999999,"15min":27.242999999999999},"last":15.981},{"interface":"front","average":{"1min":11.532999999999999,"5min":11.537000000000001,"15min":12.59},"min":{"1min":7.9770000000000003,"5min":7.4989999999999997,"15min":7.4989999999999997},"max":{"1min":18.957000000000001,"5min":24.388999999999999,"15min":24.388999999999999},"last":10.726000000000001}]},{"osd":2,"last update":"Sat Sep 30 02:11:20 2023","interfaces":[{"interface":"back","average":{"1min":14.006,"5min":14.630000000000001,"15min":15.132},"min":{"1min":9.6690000000000005,"5min":7.7039999999999997,"15min":7.7039999999999997},"max":{"1min":19.521000000000001,"5min":24.510999999999999,"15min":24.510999999999999},"last":11.791},{"interface":"front","average":{"1min":12.595000000000001,"5min":13.534000000000001,"15min":14.086},"min":{"1min":7.9699999999999998,"5min":7.9699999999999998,"15min":7.9699999999999998},"max":{"1min":20.085000000000001,"5min":25.033999999999999,"15min":25.033999999999999},"last":7.7389999999999999}]},{"osd":3,"last update":"Sat Sep 30 02:11:20 2023","interfaces":[{"interface":"back","average":{"1min":14.676,"5min":19.765999999999998,"15min":31.925000000000001},"min":{"1min":8.3770000000000007,"5min":8.2089999999999996,"15min":8.2089999999999996},"max":{"1min":20.157,"5min":540.52099999999996,"15min":540.52099999999996},"last":18.699000000000002},{"interface":"front","average":{"1min":15.218,"5min":21.224,"15min":35.917999999999999},"min":{"1min":9.4489999999999998,"5min":7.9649999999999999,"15min":7.9649999999999999},"max":{"1min":19.452999999999999,"5min":644.31600000000003,"15min":644.31600000000003},"last":13.864000000000001}]},{"osd":4,"last update":"Sat Sep 30 02:11:15 2023","interfaces":[{"interface":"back","average":{"1min":17.922000000000001,"5min":18.664999999999999,"15min":22.556999999999999},"min":{"1min":13.959,"5min":9.0969999999999995,"15min":9.0969999999999995},"max":{"1min":21.425999999999998,"5min":179.749,"15min":179.749},"last":14.513},{"interface":"front","average":{"1min":15.129,"5min":15.641999999999999,"15min":19.817},"min":{"1min":8.7550000000000008,"5min":7.6070000000000002,"15min":7.6070000000000002},"max":{"1min":19.663,"5min":174.52799999999999,"15min":174.52799999999999},"last":14.923999999999999}]},{"osd":5,"last update":"Sat Sep 30 02:11:15 2023","interfaces":[{"interface":"back","average":{"1min":16.963000000000001,"5min":16.859000000000002,"15min":16.969000000000001},"min":{"1min":9.6210000000000004,"5min":8.2379999999999995,"15min":8.2379999999999995},"max":{"1min":21.975999999999999,"5min":36.709000000000003,"15min":36.709000000000003},"last":19.891999999999999},{"interface":"front","average":{"1min":15.141999999999999,"5min":16.25,"15min":16.556000000000001},"min":{"1min":9.0809999999999995,"5min":5.0279999999999996,"15min":5.0279999999999996},"max":{"1min":21.393999999999998,"5min":27.364999999999998,"15min":27.364999999999998},"last":17.754000000000001}]},{"osd":7,"last update":"Sat Sep 30 02:11:09 2023","interfaces":[{"interface":"back","average":{"1min":19.030000000000001,"5min":19.597000000000001,"15min":22.396999999999998},"min":{"1min":13.644,"5min":10.367000000000001,"15min":10.367000000000001},"max":{"1min":22.771000000000001,"5min":105.801,"15min":105.801},"last":20.815000000000001},{"interface":"front","average":{"1min":17.553000000000001,"5min":19.193999999999999,"15min":20.960000000000001},"min":{"1min":7.3120000000000003,"5min":7.3120000000000003,"15min":7.3120000000000003},"max":{"1min":23.283000000000001,"5min":90.156000000000006,"15min":90.156000000000006},"last":15.582000000000001}]}]},{"osd":4,"up_from":15,"seq":64424509508,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5208,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366632,"statfs":{"total":96636764160,"available":96631431168,"internally_reserved":0,"allocated":212992,"data_stored":75040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:11:17 2023","interfaces":[{"interface":"back","average":{"1min":11.865,"5min":10.567,"15min":9.4399999999999995},"min":{"1min":6.3970000000000002,"5min":5.2699999999999996,"15min":5.2699999999999996},"max":{"1min":20.655000000000001,"5min":24.532,"15min":24.532},"last":6.8970000000000002},{"interface":"front","average":{"1min":11.137,"5min":10.974,"15min":11.926},"min":{"1min":7.0190000000000001,"5min":5.4530000000000003,"15min":5.4530000000000003},"max":{"1min":21.841000000000001,"5min":25.225000000000001,"15min":25.225000000000001},"last":12.039}]},{"osd":1,"last update":"Sat Sep 30 02:11:17 2023","interfaces":[{"interface":"back","average":{"1min":12.827,"5min":13.271000000000001,"15min":13.897},"min":{"1min":8.3960000000000008,"5min":6.2830000000000004,"15min":6.2830000000000004},"max":{"1min":17.864000000000001,"5min":28.917000000000002,"15min":28.917000000000002},"last":7.907},{"interface":"front","average":{"1min":10.146000000000001,"5min":11.148,"15min":11.789999999999999},"min":{"1min":7.4059999999999997,"5min":7.0250000000000004,"15min":7.0250000000000004},"max":{"1min":19.279,"5min":30.922999999999998,"15min":30.922999999999998},"last":8.6850000000000005}]},{"osd":2,"last update":"Sat Sep 30 02:11:17 2023","interfaces":[{"interface":"back","average":{"1min":14.563000000000001,"5min":13.641999999999999,"15min":14.407999999999999},"min":{"1min":7.8300000000000001,"5min":7.1609999999999996,"15min":7.1609999999999996},"max":{"1min":19.428000000000001,"5min":21.175000000000001,"15min":21.175000000000001},"last":18.225000000000001},{"interface":"front","average":{"1min":12.637,"5min":13.872,"15min":14.563000000000001},"min":{"1min":6.7060000000000004,"5min":6.7060000000000004,"15min":6.7060000000000004},"max":{"1min":18.747,"5min":27.533999999999999,"15min":27.533999999999999},"last":10.913}]},{"osd":3,"last update":"Sat Sep 30 02:11:23 2023","interfaces":[{"interface":"back","average":{"1min":16.146000000000001,"5min":16.042999999999999,"15min":15.887},"min":{"1min":9.4719999999999995,"5min":9.4719999999999995,"15min":9.4719999999999995},"max":{"1min":23.526,"5min":27.632999999999999,"15min":27.632999999999999},"last":19.196999999999999},{"interface":"front","average":{"1min":15.352,"5min":16.143999999999998,"15min":17.184999999999999},"min":{"1min":8.2599999999999998,"5min":7.9269999999999996,"15min":7.9269999999999996},"max":{"1min":23.113,"5min":30.495000000000001,"15min":30.495000000000001},"last":13.912000000000001}]},{"osd":5,"last update":"Sat Sep 30 02:11:11 2023","interfaces":[{"interface":"back","average":{"1min":16.295999999999999,"5min":18.059000000000001,"15min":20.965},"min":{"1min":12.298,"5min":7.9080000000000004,"15min":7.9080000000000004},"max":{"1min":21.044,"5min":99.784000000000006,"15min":99.784000000000006},"last":17.027999999999999},{"interface":"front","average":{"1min":13.371,"5min":15.058,"15min":17.594999999999999},"min":{"1min":8.3030000000000008,"5min":7.609,"15min":7.609},"max":{"1min":19.103000000000002,"5min":98.634,"15min":98.634},"last":14.597}]},{"osd":6,"last update":"Sat Sep 30 02:11:11 2023","interfaces":[{"interface":"back","average":{"1min":16.995000000000001,"5min":16.553999999999998,"15min":16.952999999999999},"min":{"1min":7.8949999999999996,"5min":7.8949999999999996,"15min":7.8949999999999996},"max":{"1min":22.498999999999999,"5min":27.277999999999999,"15min":27.277999999999999},"last":20.065999999999999},{"interface":"front","average":{"1min":15.564,"5min":16.422000000000001,"15min":16.088999999999999},"min":{"1min":9.109,"5min":6.5810000000000004,"15min":6.5810000000000004},"max":{"1min":20.640999999999998,"5min":28.640000000000001,"15min":28.640000000000001},"last":12.805}]},{"osd":7,"last update":"Sat Sep 30 02:11:11 2023","interfaces":[{"interface":"back","average":{"1min":17.225000000000001,"5min":18.161000000000001,"15min":18.513000000000002},"min":{"1min":7.79,"5min":7.79,"15min":7.79},"max":{"1min":23.047000000000001,"5min":31.143000000000001,"15min":31.143000000000001},"last":20.954999999999998},{"interface":"front","average":{"1min":17.949000000000002,"5min":18.007000000000001,"15min":18.579000000000001},"min":{"1min":10.542999999999999,"5min":8.6509999999999998,"15min":8.6509999999999998},"max":{"1min":22.966999999999999,"5min":30.529,"15min":30.529},"last":16.082000000000001}]}]},{"osd":5,"up_from":15,"seq":64424509508,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6108,"kb_used_data":600,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":94365732,"statfs":{"total":96636764160,"available":96630509568,"internally_reserved":0,"allocated":614400,"data_stored":472880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:11:18 2023","interfaces":[{"interface":"back","average":{"1min":10.455,"5min":10.214,"15min":11.779999999999999},"min":{"1min":6.1040000000000001,"5min":5.7350000000000003,"15min":5.7350000000000003},"max":{"1min":19.498999999999999,"5min":22.452999999999999,"15min":22.452999999999999},"last":18.425000000000001},{"interface":"front","average":{"1min":10.673999999999999,"5min":10.722,"15min":11.548},"min":{"1min":6.6929999999999996,"5min":5.8879999999999999,"15min":5.8879999999999999},"max":{"1min":20.34,"5min":22.821999999999999,"15min":22.821999999999999},"last":10.420999999999999}]},{"osd":1,"last update":"Sat Sep 30 02:11:18 2023","interfaces":[{"interface":"back","average":{"1min":14.335000000000001,"5min":13.829000000000001,"15min":15.194000000000001},"min":{"1min":8.9589999999999996,"5min":7.29,"15min":7.29},"max":{"1min":22.402999999999999,"5min":47.451999999999998,"15min":47.451999999999998},"last":15.948},{"interface":"front","average":{"1min":11.304,"5min":11.545,"15min":12.558},"min":{"1min":7.774,"5min":6.6130000000000004,"15min":6.6130000000000004},"max":{"1min":15.734999999999999,"5min":48.841999999999999,"15min":48.841999999999999},"last":9.3559999999999999}]},{"osd":2,"last update":"Sat Sep 30 02:11:18 2023","interfaces":[{"interface":"back","average":{"1min":15.868,"5min":13.461,"15min":11.835000000000001},"min":{"1min":8.2829999999999995,"5min":6.8250000000000002,"15min":6.8250000000000002},"max":{"1min":27.501000000000001,"5min":27.501000000000001,"15min":27.501000000000001},"last":19.318999999999999},{"interface":"front","average":{"1min":12.898999999999999,"5min":14.695,"15min":15.513},"min":{"1min":7.8150000000000004,"5min":7.5369999999999999,"15min":7.5369999999999999},"max":{"1min":19.411000000000001,"5min":26.576000000000001,"15min":26.576000000000001},"last":8.6630000000000003}]},{"osd":3,"last update":"Sat Sep 30 02:10:25 2023","interfaces":[{"interface":"back","average":{"1min":15.916,"5min":15.304,"15min":15.119},"min":{"1min":7.2619999999999996,"5min":7.2619999999999996,"15min":7.2619999999999996},"max":{"1min":20.603000000000002,"5min":26.895,"15min":26.895},"last":12.452999999999999},{"interface":"front","average":{"1min":17.449999999999999,"5min":17.664999999999999,"15min":18.010999999999999},"min":{"1min":12.377000000000001,"5min":11.140000000000001,"15min":11.140000000000001},"max":{"1min":21.434000000000001,"5min":24.702999999999999,"15min":24.702999999999999},"last":16.960000000000001}]},{"osd":4,"last update":"Sat Sep 30 02:11:18 2023","interfaces":[{"interface":"back","average":{"1min":15.975,"5min":17.355,"15min":18.113},"min":{"1min":10.063000000000001,"5min":8.8160000000000007,"15min":8.8160000000000007},"max":{"1min":22.620999999999999,"5min":26.710000000000001,"15min":26.710000000000001},"last":12.776},{"interface":"front","average":{"1min":16.137,"5min":16.138999999999999,"15min":15.467000000000001},"min":{"1min":10.363,"5min":6.0270000000000001,"15min":6.0270000000000001},"max":{"1min":23.376000000000001,"5min":28.344999999999999,"15min":28.344999999999999},"last":20.321000000000002}]},{"osd":6,"last update":"Sat Sep 30 02:11:18 2023","interfaces":[{"interface":"back","average":{"1min":17.373000000000001,"5min":16.178999999999998,"15min":14.978999999999999},"min":{"1min":10.907999999999999,"5min":7.0679999999999996,"15min":7.0679999999999996},"max":{"1min":22.890999999999998,"5min":23.573,"15min":23.573},"last":13.380000000000001},{"interface":"front","average":{"1min":14.722,"5min":16.218,"15min":16.617000000000001},"min":{"1min":7.8550000000000004,"5min":7.5540000000000003,"15min":7.5540000000000003},"max":{"1min":20.41,"5min":27.763999999999999,"15min":27.763999999999999},"last":14.332000000000001}]},{"osd":7,"last update":"Sat Sep 30 02:11:18 2023","interfaces":[{"interface":"back","average":{"1min":17.943999999999999,"5min":18.265999999999998,"15min":19.606000000000002},"min":{"1min":12.4,"5min":10.584,"15min":10.584},"max":{"1min":22.768000000000001,"5min":28.951000000000001,"15min":28.951000000000001},"last":20.969999999999999},{"interface":"front","average":{"1min":17.033999999999999,"5min":16.844000000000001,"15min":16.053999999999998},"min":{"1min":11.109,"5min":7.0490000000000004,"15min":7.0490000000000004},"max":{"1min":21.635999999999999,"5min":25.151,"15min":25.151},"last":17.356000000000002}]}]},{"osd":2,"up_from":18,"seq":77309411394,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5208,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366632,"statfs":{"total":96636764160,"available":96631431168,"internally_reserved":0,"allocated":212992,"data_stored":75040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:11:21 2023","interfaces":[{"interface":"back","average":{"1min":12.927,"5min":13.202999999999999,"15min":13.711},"min":{"1min":7.1929999999999996,"5min":6.665,"15min":6.665},"max":{"1min":20.434000000000001,"5min":69.003,"15min":69.003},"last":8.2379999999999995},{"interface":"front","average":{"1min":13.492000000000001,"5min":14.566000000000001,"15min":16.558},"min":{"1min":8.6880000000000006,"5min":8.1329999999999991,"15min":8.1329999999999991},"max":{"1min":25.254000000000001,"5min":91.331999999999994,"15min":91.331999999999994},"last":13.752000000000001}]},{"osd":1,"last update":"Sat Sep 30 02:11:21 2023","interfaces":[{"interface":"back","average":{"1min":14.648999999999999,"5min":16.484999999999999,"15min":20.882000000000001},"min":{"1min":9.4800000000000004,"5min":7.9630000000000001,"15min":7.9630000000000001},"max":{"1min":21.640999999999998,"5min":145.952,"15min":145.952},"last":16.172999999999998},{"interface":"front","average":{"1min":15.581,"5min":17.215,"15min":21.452000000000002},"min":{"1min":9.6679999999999993,"5min":8.7469999999999999,"15min":8.7469999999999999},"max":{"1min":25.678000000000001,"5min":155.72900000000001,"15min":155.72900000000001},"last":14.782}]},{"osd":3,"last update":"Sat Sep 30 02:11:21 2023","interfaces":[{"interface":"back","average":{"1min":16.968,"5min":16.724,"15min":17.524000000000001},"min":{"1min":10.601000000000001,"5min":10.124000000000001,"15min":10.124000000000001},"max":{"1min":32.027000000000001,"5min":39.378999999999998,"15min":39.378999999999998},"last":20.689},{"interface":"front","average":{"1min":17.798999999999999,"5min":18.437000000000001,"15min":18.405000000000001},"min":{"1min":8.9250000000000007,"5min":8.9250000000000007,"15min":8.9250000000000007},"max":{"1min":24.704999999999998,"5min":36.649000000000001,"15min":36.649000000000001},"last":16.843}]},{"osd":4,"last update":"Sat Sep 30 02:11:21 2023","interfaces":[{"interface":"back","average":{"1min":18.395,"5min":18.151,"15min":18.896999999999998},"min":{"1min":11.792999999999999,"5min":8.9359999999999999,"15min":8.9359999999999999},"max":{"1min":29.699000000000002,"5min":39.186999999999998,"15min":39.186999999999998},"last":21.863},{"interface":"front","average":{"1min":11.427,"5min":11.253,"15min":12.537000000000001},"min":{"1min":6.7329999999999997,"5min":6.7329999999999997,"15min":6.7329999999999997},"max":{"1min":23.036999999999999,"5min":31.32,"15min":31.32},"last":11.134}]},{"osd":5,"last update":"Sat Sep 30 02:11:21 2023","interfaces":[{"interface":"back","average":{"1min":19.495000000000001,"5min":18.518999999999998,"15min":19.271000000000001},"min":{"1min":12.682,"5min":8.3499999999999996,"15min":8.3499999999999996},"max":{"1min":26.695,"5min":43.409999999999997,"15min":43.409999999999997},"last":18.423999999999999},{"interface":"front","average":{"1min":20.788,"5min":21.66,"15min":22.609999999999999},"min":{"1min":11.974,"5min":11.974,"15min":11.974},"max":{"1min":33.695999999999998,"5min":40.646000000000001,"15min":40.646000000000001},"last":22.545000000000002}]},{"osd":6,"last update":"Sat Sep 30 02:11:21 2023","interfaces":[{"interface":"back","average":{"1min":13.474,"5min":14.634,"15min":15.917999999999999},"min":{"1min":9.8520000000000003,"5min":7.4829999999999997,"15min":7.4829999999999997},"max":{"1min":21.18,"5min":34.045000000000002,"15min":34.045000000000002},"last":17.640999999999998},{"interface":"front","average":{"1min":19.893999999999998,"5min":19.949000000000002,"15min":21.067},"min":{"1min":12.353999999999999,"5min":9.1600000000000001,"15min":9.1600000000000001},"max":{"1min":24.425000000000001,"5min":47.997999999999998,"15min":47.997999999999998},"last":19.922000000000001}]},{"osd":7,"last update":"Sat Sep 30 02:11:21 2023","interfaces":[{"interface":"back","average":{"1min":21.699999999999999,"5min":22.599,"15min":24.295000000000002},"min":{"1min":15.781000000000001,"5min":13.542999999999999,"15min":13.542999999999999},"max":{"1min":31.061,"5min":44.585000000000001,"15min":44.585000000000001},"last":23.716999999999999},{"interface":"front","average":{"1min":15.755000000000001,"5min":16.690999999999999,"15min":17.437000000000001},"min":{"1min":8.8230000000000004,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":26.234000000000002,"5min":38.295000000000002,"15min":38.295000000000002},"last":14.118}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T02:11:31.465 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T02:11:31.597 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.597+0000 7fef4512f700 10 monclient: get_monmap_and_config 2023-09-30T02:11:31.597 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.597+0000 7fef4512f700 10 monclient: build_initial_monmap 2023-09-30T02:11:31.597 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.597+0000 7fef4512f700 10 monclient: monmap: 2023-09-30T02:11:31.597 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:11:31.597 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:31.597 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:11:31.598102+0000 2023-09-30T02:11:31.597 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:11:31.598102+0000 2023-09-30T02:11:31.597 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:11:31.597 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:31.598 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:11:31.598 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:11:31.598 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:11:31.598 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:11:31.598 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:11:31.598 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:31.600 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.598+0000 7fef4512f700 1 Processor -- start 2023-09-30T02:11:31.600 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.598+0000 7fef4512f700 1 -- start start 2023-09-30T02:11:31.600 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.598+0000 7fef4512f700 10 monclient: init 2023-09-30T02:11:31.600 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.599+0000 7fef4512f700 10 monclient: _reopen_session rank -1 2023-09-30T02:11:31.601 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.599+0000 7fef4512f700 10 monclient: _add_conns ranks=[1,4,3,0,2] 2023-09-30T02:11:31.601 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.599+0000 7fef4512f700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fef400a5b70 addr v1:172.21.15.110:6789/0 2023-09-30T02:11:31.601 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.599+0000 7fef4512f700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fef40142290 0x7fef400b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:31.601 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.599+0000 7fef4512f700 10 monclient(hunting): picked mon.noname-c con 0x7fef40142290 addr v2:172.21.15.110:3300/0 2023-09-30T02:11:31.601 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.599+0000 7fef4512f700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fef400a5f40 0x7fef4013af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:31.601 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.599+0000 7fef4512f700 10 monclient(hunting): picked mon.noname-a con 0x7fef400a5f40 addr v2:172.21.15.59:3300/0 2023-09-30T02:11:31.601 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.599+0000 7fef4512f700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef400a75a0 con 0x7fef400a5b70 2023-09-30T02:11:31.602 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.599+0000 7fef4512f700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:31.602 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.599+0000 7fef4512f700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef400a6360 con 0x7fef400a5f40 2023-09-30T02:11:31.602 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.599+0000 7fef4512f700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:31.602 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.599+0000 7fef4512f700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef400af3f0 con 0x7fef40142290 2023-09-30T02:11:31.602 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.599+0000 7fef4512f700 10 monclient(hunting): _renew_subs 2023-09-30T02:11:31.602 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.599+0000 7fef4512f700 10 monclient(hunting): authenticate will time out at 2136.901282s 2023-09-30T02:11:31.602 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.599+0000 7fef3e59c700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fef40142290 0x7fef400b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:31.602 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.600+0000 7fef3f59e700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fef400a5f40 0x7fef4013af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:31.602 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.600+0000 7fef3e59c700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fef40142290 0x7fef400b1500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:60456/0 (socket says 172.21.15.59:60456) 2023-09-30T02:11:31.602 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.600+0000 7fef3e59c700 1 -- 172.21.15.59:0/57473707 learned_addr learned my addr 172.21.15.59:0/57473707 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:11:31.602 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.600+0000 7fef3e59c700 10 monclient(hunting): get_auth_request con 0x7fef40142290 auth_method 0 2023-09-30T02:11:31.602 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.600+0000 7fef3e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:31.603 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.600+0000 7fef3e59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:31.603 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.600+0000 7fef3e59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:31.603 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.600+0000 7fef3f59e700 10 monclient(hunting): get_auth_request con 0x7fef400a5f40 auth_method 0 2023-09-30T02:11:31.603 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.600+0000 7fef3f59e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:31.603 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.600+0000 7fef3f59e700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:31.603 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.600+0000 7fef3f59e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:31.603 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.601+0000 7fef3e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:31.604 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.601+0000 7fef3e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:31.604 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.601+0000 7fef3e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:31.604 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.601+0000 7fef3f59e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:31.604 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.601+0000 7fef3f59e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:31.604 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.601+0000 7fef3f59e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:31.604 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.603+0000 7fef3e59c700 10 monclient(hunting): handle_auth_done global_id 4864 payload 274 2023-09-30T02:11:31.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.603+0000 7fef3e59c700 1 -- 172.21.15.59:0/57473707 >> v2:172.21.15.59:3300/0 conn(0x7fef400a5f40 msgr2=0x7fef4013af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:11:31.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.603+0000 7fef3e59c700 1 --2- 172.21.15.59:0/57473707 >> v2:172.21.15.59:3300/0 conn(0x7fef400a5f40 0x7fef4013af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:31.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.603+0000 7fef3e59c700 1 -- 172.21.15.59:0/57473707 >> v1:172.21.15.110:6789/0 conn(0x7fef400a5b70 legacy=0x7fef40141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:31.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.603+0000 7fef3e59c700 10 monclient: _finish_hunting 0 2023-09-30T02:11:31.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.603+0000 7fef3e59c700 1 monclient: found mon.noname-c 2023-09-30T02:11:31.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.603+0000 7fef3e59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:11:31.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.603+0000 7fef3e59c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3300/0 2023-09-30T02:11:31.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.603+0000 7fef3e59c700 1 -- 172.21.15.59:0/57473707 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef4013b470 con 0x7fef40142290 2023-09-30T02:11:31.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.606+0000 7fef3e59c700 1 --2- 172.21.15.59:0/57473707 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fef40142290 0x7fef400b1500 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fef400b1fc0 tx=0x7fef34009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=366942869778c97d server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:31.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.611+0000 7fef3dd9b700 1 -- 172.21.15.59:0/57473707 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef34009e70 con 0x7fef40142290 2023-09-30T02:11:31.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.611+0000 7fef3dd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:31.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.611+0000 7fef3dd9b700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:11:31.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.611+0000 7fef3dd9b700 10 monclient: dump: 2023-09-30T02:11:31.611 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:31.611 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:31.612 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:31.612 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:31.612 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:31.612 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:31.612 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:31.612 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:31.612 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:31.612 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:31.612 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.611+0000 7fef3dd9b700 10 monclient: _finish_auth 0 2023-09-30T02:11:31.612 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.611+0000 7fef3dd9b700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:11:31.612 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.611+0000 7fef3dd9b700 1 -- 172.21.15.59:0/57473707 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef3400b860 con 0x7fef40142290 2023-09-30T02:11:31.612 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.611+0000 7fef3dd9b700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:11:31.613 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.611+0000 7fef3dd9b700 1 -- 172.21.15.59:0/57473707 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef3400bc00 con 0x7fef40142290 2023-09-30T02:11:31.613 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.611+0000 7fef4512f700 5 monclient: authenticate success, global_id 4864 2023-09-30T02:11:31.613 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.611+0000 7fef4512f700 10 monclient: get_monmap_and_config success 2023-09-30T02:11:31.613 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.611+0000 7fef4512f700 4 set_mon_vals no callback set 2023-09-30T02:11:31.613 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.611+0000 7fef3dd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:31.613 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.611+0000 7fef3dd9b700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:11:31.613 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.611+0000 7fef3dd9b700 10 monclient: dump: 2023-09-30T02:11:31.613 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:31.613 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:31.613 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:31.613 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:31.613 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:31.614 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:31.614 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:31.614 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:31.614 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:31.614 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:31.617 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.612+0000 7fef4512f700 10 monclient: shutdown 2023-09-30T02:11:31.618 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.612+0000 7fef4512f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:11:31.618 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.612+0000 7fef4512f700 1 -- 172.21.15.59:0/57473707 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fef40142290 msgr2=0x7fef400b1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:31.618 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.612+0000 7fef4512f700 1 --2- 172.21.15.59:0/57473707 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fef40142290 0x7fef400b1500 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fef400b1fc0 tx=0x7fef34009250 comp rx=0 tx=0).stop 2023-09-30T02:11:31.618 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.612+0000 7fef4512f700 1 -- 172.21.15.59:0/57473707 shutdown_connections 2023-09-30T02:11:31.618 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.612+0000 7fef4512f700 1 --2- 172.21.15.59:0/57473707 >> v2:172.21.15.59:3300/0 conn(0x7fef400a5f40 0x7fef4013af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:31.618 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.612+0000 7fef4512f700 1 --2- 172.21.15.59:0/57473707 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fef40142290 0x7fef400b1500 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:31.618 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.612+0000 7fef4512f700 1 -- 172.21.15.59:0/57473707 >> 172.21.15.59:0/57473707 conn(0x7fef400631f0 msgr2=0x7fef400635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:11:31.618 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.612+0000 7fef4512f700 1 -- 172.21.15.59:0/57473707 shutdown_connections 2023-09-30T02:11:31.618 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.612+0000 7fef4512f700 1 -- 172.21.15.59:0/57473707 wait complete. 2023-09-30T02:11:31.618 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.613+0000 7fef4512f700 10 monclient: build_initial_monmap 2023-09-30T02:11:31.618 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.613+0000 7fef4512f700 10 monclient: monmap: 2023-09-30T02:11:31.619 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:11:31.619 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:11:31.619 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:11:31.619 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:11:31.619 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:11:31.619 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:11:31.619 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:11:31.619 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:11:31.619 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:11:31.619 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:31.619 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.613+0000 7fef4512f700 1 librados: starting msgr at 2023-09-30T02:11:31.619 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.613+0000 7fef4512f700 1 librados: starting objecter 2023-09-30T02:11:31.620 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.614+0000 7fef4512f700 1 Processor -- start 2023-09-30T02:11:31.620 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.614+0000 7fef4512f700 1 -- start start 2023-09-30T02:11:31.620 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.614+0000 7fef4512f700 1 librados: setting wanted keys 2023-09-30T02:11:31.620 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.614+0000 7fef4512f700 1 librados: calling monclient init 2023-09-30T02:11:31.620 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.614+0000 7fef4512f700 10 monclient: init 2023-09-30T02:11:31.620 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.614+0000 7fef4512f700 10 monclient: _reopen_session rank -1 2023-09-30T02:11:31.620 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.614+0000 7fef4512f700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T02:11:31.620 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.614+0000 7fef4512f700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fef400a5b70 0x7fef4013da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:31.620 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.614+0000 7fef4512f700 10 monclient(hunting): picked mon.noname-b con 0x7fef400a5b70 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:11:31.620 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.614+0000 7fef4512f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef400a5f40 0x7fef4013fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:31.620 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.614+0000 7fef4512f700 10 monclient(hunting): picked mon.noname-a con 0x7fef400a5f40 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:11:31.620 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.614+0000 7fef4512f700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fef40142290 0x7fef4014e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:31.621 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.614+0000 7fef4512f700 10 monclient(hunting): picked mon.noname-c con 0x7fef40142290 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:11:31.621 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.614+0000 7fef4512f700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:31.621 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.614+0000 7fef4512f700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef4013e1f0 con 0x7fef400a5f40 2023-09-30T02:11:31.621 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.614+0000 7fef4512f700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:31.621 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.614+0000 7fef4512f700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef4014e3c0 con 0x7fef400a5b70 2023-09-30T02:11:31.621 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.614+0000 7fef4512f700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:31.621 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.614+0000 7fef4512f700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef4014e050 con 0x7fef40142290 2023-09-30T02:11:31.621 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.614+0000 7fef4512f700 10 monclient(hunting): _renew_subs 2023-09-30T02:11:31.621 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.614+0000 7fef4512f700 10 monclient(hunting): authenticate will time out at 2136.916724s 2023-09-30T02:11:31.621 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.615+0000 7fef3ed9d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fef400a5b70 0x7fef4013da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:31.621 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.615+0000 7fef3e59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef400a5f40 0x7fef4013fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:31.621 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.615+0000 7fef3f59e700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fef40142290 0x7fef4014e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:31.622 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.615+0000 7fef3ed9d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fef400a5b70 0x7fef4013da80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:60464/0 (socket says 172.21.15.59:60464) 2023-09-30T02:11:31.622 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.615+0000 7fef3ed9d700 1 -- 172.21.15.59:0/2303409760 learned_addr learned my addr 172.21.15.59:0/2303409760 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:11:31.622 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.615+0000 7fef3ed9d700 10 monclient(hunting): get_auth_request con 0x7fef400a5b70 auth_method 0 2023-09-30T02:11:31.622 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.615+0000 7fef3ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:31.622 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.615+0000 7fef3ed9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:31.622 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.615+0000 7fef3ed9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:31.622 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.615+0000 7fef3f59e700 10 monclient(hunting): get_auth_request con 0x7fef40142290 auth_method 0 2023-09-30T02:11:31.622 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.615+0000 7fef3f59e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:31.622 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.615+0000 7fef3f59e700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:31.622 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.615+0000 7fef3f59e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:31.622 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.615+0000 7fef3e59c700 10 monclient(hunting): get_auth_request con 0x7fef400a5f40 auth_method 0 2023-09-30T02:11:31.622 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.615+0000 7fef3e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:31.623 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.615+0000 7fef3e59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:31.623 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.615+0000 7fef3e59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:31.623 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.616+0000 7fef3ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:31.623 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.616+0000 7fef3ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:31.623 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.616+0000 7fef3ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:31.623 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.616+0000 7fef3f59e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:31.623 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.616+0000 7fef3f59e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:31.623 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.616+0000 7fef3f59e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:31.623 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.616+0000 7fef3e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:31.623 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.616+0000 7fef3e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:31.623 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.616+0000 7fef3e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:31.625 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.619+0000 7fef3ed9d700 10 monclient(hunting): handle_auth_done global_id 4867 payload 824 2023-09-30T02:11:31.625 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.619+0000 7fef3ed9d700 1 -- 172.21.15.59:0/2303409760 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fef40142290 msgr2=0x7fef4014e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:31.625 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.619+0000 7fef3ed9d700 1 --2- 172.21.15.59:0/2303409760 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fef40142290 0x7fef4014e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:31.625 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.619+0000 7fef3ed9d700 1 -- 172.21.15.59:0/2303409760 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef400a5f40 msgr2=0x7fef4013fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:31.625 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.619+0000 7fef3ed9d700 1 --2- 172.21.15.59:0/2303409760 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef400a5f40 0x7fef4013fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:31.626 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.619+0000 7fef3ed9d700 10 monclient: _finish_hunting 0 2023-09-30T02:11:31.626 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.619+0000 7fef3ed9d700 1 monclient: found mon.noname-b 2023-09-30T02:11:31.626 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.619+0000 7fef3ed9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:11:31.626 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.619+0000 7fef3ed9d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:11:31.626 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.619+0000 7fef3ed9d700 1 -- 172.21.15.59:0/2303409760 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef340094f0 con 0x7fef400a5b70 2023-09-30T02:11:31.626 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.621+0000 7fef3ed9d700 1 --2- 172.21.15.59:0/2303409760 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fef400a5b70 0x7fef4013da80 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fef3000c590 tx=0x7fef3000c8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:31.626 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.621+0000 7fef26ffd700 10 client.?.objecter ms_handle_connect 0x7fef400a5b70 2023-09-30T02:11:31.626 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.621+0000 7fef26ffd700 10 client.?.objecter resend_mon_ops 2023-09-30T02:11:31.626 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.625+0000 7fef26ffd700 1 -- 172.21.15.59:0/2303409760 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef30011590 con 0x7fef400a5b70 2023-09-30T02:11:31.626 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.625+0000 7fef26ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:31.626 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.625+0000 7fef26ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:11:31.626 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.625+0000 7fef26ffd700 10 monclient: dump: 2023-09-30T02:11:31.627 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:31.627 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:31.627 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:31.627 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:31.628 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:31.628 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:31.628 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:31.628 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:31.628 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:31.628 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:31.628 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.625+0000 7fef26ffd700 10 monclient: _finish_auth 0 2023-09-30T02:11:31.628 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.625+0000 7fef26ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:11:31.628 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.625+0000 7fef26ffd700 1 -- 172.21.15.59:0/2303409760 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef30011710 con 0x7fef400a5b70 2023-09-30T02:11:31.628 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.625+0000 7fef4512f700 5 monclient: authenticate success, global_id 4867 2023-09-30T02:11:31.628 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.625+0000 7fef4512f700 10 monclient: _renew_subs 2023-09-30T02:11:31.628 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.625+0000 7fef4512f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:11:31.629 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.625+0000 7fef4512f700 1 -- 172.21.15.59:0/2303409760 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef4014ed90 con 0x7fef400a5b70 2023-09-30T02:11:31.629 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.625+0000 7fef26ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:11:31.629 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.625+0000 7fef26ffd700 1 -- 172.21.15.59:0/2303409760 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef30011ad0 con 0x7fef400a5b70 2023-09-30T02:11:31.629 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.625+0000 7fef26ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:31.629 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.625+0000 7fef26ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:11:31.629 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.625+0000 7fef26ffd700 10 monclient: dump: 2023-09-30T02:11:31.629 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:31.629 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:31.629 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:31.629 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:31.629 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:31.629 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:31.630 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:31.630 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:31.630 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:31.630 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:31.630 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.625+0000 7fef3d59a700 4 set_mon_vals no callback set 2023-09-30T02:11:31.630 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.625+0000 7fef4512f700 10 client.4867.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:11:31.630 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.625+0000 7fef4512f700 10 monclient: _renew_subs 2023-09-30T02:11:31.630 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.625+0000 7fef4512f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:11:31.630 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.625+0000 7fef4512f700 1 -- 172.21.15.59:0/2303409760 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef4013c1f0 con 0x7fef400a5b70 2023-09-30T02:11:31.630 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.625+0000 7fef4512f700 1 librados: init done 2023-09-30T02:11:31.630 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.626+0000 7fef4512f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:11:31.630 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.626+0000 7fef4512f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:11:31.631 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.626+0000 7fef4512f700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:11:31.631 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.626+0000 7fef4512f700 1 -- 172.21.15.59:0/2303409760 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef4014e6a0 con 0x7fef400a5b70 2023-09-30T02:11:31.874 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.875+0000 7fef26ffd700 1 -- 172.21.15.59:0/2303409760 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fef3001b3b0 con 0x7fef400a5b70 2023-09-30T02:11:31.875 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.875+0000 7fef26ffd700 1 --2- 172.21.15.59:0/2303409760 >> v2:172.21.15.110:6800/115870 conn(0x7fef2c068960 0x7fef2c06ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:31.875 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.875+0000 7fef26ffd700 1 -- 172.21.15.59:0/2303409760 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fef30083770 con 0x7fef400a5b70 2023-09-30T02:11:31.875 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.875+0000 7fef26ffd700 10 client.4867.objecter ms_dispatch 0x7fef40001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:11:31.875 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.875+0000 7fef26ffd700 3 client.4867.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:11:31.875 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.875+0000 7fef26ffd700 3 client.4867.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:11:31.876 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.875+0000 7fef26ffd700 20 client.4867.objecter dump_active .. 0 homeless 2023-09-30T02:11:31.876 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.875+0000 7fef3e59c700 1 --2- 172.21.15.59:0/2303409760 >> v2:172.21.15.110:6800/115870 conn(0x7fef2c068960 0x7fef2c06ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:31.876 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.875+0000 7fef3e59c700 10 monclient: get_auth_request con 0x7fef2c068960 auth_method 0 2023-09-30T02:11:31.876 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.876+0000 7fef26ffd700 10 client.4867.objecter ms_handle_connect 0x7fef2c068960 2023-09-30T02:11:31.876 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.876+0000 7fef3e59c700 1 --2- 172.21.15.59:0/2303409760 >> v2:172.21.15.110:6800/115870 conn(0x7fef2c068960 0x7fef2c06ae10 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fef400b1fc0 tx=0x7fef34017000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:31.883 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.883+0000 7fef26ffd700 1 -- 172.21.15.59:0/2303409760 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fef30051090 con 0x7fef400a5b70 2023-09-30T02:11:31.883 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.883+0000 7fef26ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:11:31.883 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:31.883+0000 7fef26ffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:11:32.086 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.087+0000 7fef4512f700 1 -- 172.21.15.59:0/2303409760 --> v2:172.21.15.110:6800/115870 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fef40064a00 con 0x7fef2c068960 2023-09-30T02:11:32.090 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.090+0000 7fef26ffd700 1 -- 172.21.15.59:0/2303409760 <== mgr.4108 v2:172.21.15.110:6800/115870 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+70474 (secure 0 0 0) 0x7fef40064a00 con 0x7fef2c068960 2023-09-30T02:11:32.091 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:11:32.092 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.093+0000 7fef4512f700 10 librados: watch_flush enter 2023-09-30T02:11:32.093 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.093+0000 7fef4512f700 10 librados: watch_flush exit 2023-09-30T02:11:32.093 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.093+0000 7fef4512f700 20 client.4867.objecter shutdown clearing up homeless session... 2023-09-30T02:11:32.093 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.093+0000 7fef4512f700 10 client.4867.objecter successfully canceled tick 2023-09-30T02:11:32.093 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.093+0000 7fef4512f700 1 -- 172.21.15.59:0/2303409760 >> v2:172.21.15.110:6800/115870 conn(0x7fef2c068960 msgr2=0x7fef2c06ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:32.093 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.093+0000 7fef4512f700 1 --2- 172.21.15.59:0/2303409760 >> v2:172.21.15.110:6800/115870 conn(0x7fef2c068960 0x7fef2c06ae10 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fef400b1fc0 tx=0x7fef34017000 comp rx=0 tx=0).stop 2023-09-30T02:11:32.093 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.093+0000 7fef4512f700 10 monclient: shutdown 2023-09-30T02:11:32.093 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.093+0000 7fef4512f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:11:32.093 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.093+0000 7fef4512f700 1 -- 172.21.15.59:0/2303409760 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fef400a5b70 msgr2=0x7fef4013da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:32.093 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.093+0000 7fef4512f700 1 --2- 172.21.15.59:0/2303409760 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fef400a5b70 0x7fef4013da80 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fef3000c590 tx=0x7fef3000c8a0 comp rx=0 tx=0).stop 2023-09-30T02:11:32.093 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.093+0000 7fef4512f700 1 -- 172.21.15.59:0/2303409760 shutdown_connections 2023-09-30T02:11:32.094 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.093+0000 7fef4512f700 1 --2- 172.21.15.59:0/2303409760 >> v2:172.21.15.110:6800/115870 conn(0x7fef2c068960 0x7fef2c06ae10 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:32.094 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.093+0000 7fef4512f700 1 --2- 172.21.15.59:0/2303409760 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fef400a5b70 0x7fef4013da80 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:32.094 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.093+0000 7fef4512f700 1 --2- 172.21.15.59:0/2303409760 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fef40142290 0x7fef4014e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:32.094 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.093+0000 7fef4512f700 1 --2- 172.21.15.59:0/2303409760 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fef400a5f40 0x7fef4013fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:32.094 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.093+0000 7fef4512f700 1 -- 172.21.15.59:0/2303409760 >> 172.21.15.59:0/2303409760 conn(0x7fef400631f0 msgr2=0x7fef400b4250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:11:32.094 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.093+0000 7fef4512f700 1 -- 172.21.15.59:0/2303409760 shutdown_connections 2023-09-30T02:11:32.094 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.093+0000 7fef4512f700 1 -- 172.21.15.59:0/2303409760 wait complete. 2023-09-30T02:11:32.094 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.093+0000 7fef4512f700 1 librados: shutdown 2023-09-30T02:11:32.094 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2023-09-30T02:11:32.106 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":194,"stamp":"2023-09-30T02:11:31.647496+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":43464,"kb_used_data":2448,"kb_used_omap":0,"kb_used_meta":40576,"kb_avail":754931256,"statfs":{"total":773094113280,"available":773049606144,"internally_reserved":0,"allocated":2506752,"data_stored":1396000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":41549824},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002601"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T02:06:18.350818+0000","last_change":"2023-09-30T02:06:18.210390+0000","last_active":"2023-09-30T02:06:18.350818+0000","last_peered":"2023-09-30T02:06:18.350818+0000","last_clean":"2023-09-30T02:06:18.350818+0000","last_became_active":"2023-09-30T02:06:18.198153+0000","last_became_peered":"2023-09-30T02:06:18.198153+0000","last_unstale":"2023-09-30T02:06:18.350818+0000","last_undegraded":"2023-09-30T02:06:18.350818+0000","last_fullsized":"2023-09-30T02:06:18.350818+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_clean_scrub_stamp":"2023-09-30T02:06:15.800912+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:42:32.739379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T02:06:18.437839+0000","last_change":"2023-09-30T02:06:18.262708+0000","last_active":"2023-09-30T02:06:18.437839+0000","last_peered":"2023-09-30T02:06:18.437839+0000","last_clean":"2023-09-30T02:06:18.437839+0000","last_became_active":"2023-09-30T02:06:18.224464+0000","last_became_peered":"2023-09-30T02:06:18.224464+0000","last_unstale":"2023-09-30T02:06:18.437839+0000","last_undegraded":"2023-09-30T02:06:18.437839+0000","last_fullsized":"2023-09-30T02:06:18.437839+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_clean_scrub_stamp":"2023-09-30T02:06:15.800912+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:13:39.084627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T02:06:18.364779+0000","last_change":"2023-09-30T02:06:17.910244+0000","last_active":"2023-09-30T02:06:18.364779+0000","last_peered":"2023-09-30T02:06:18.364779+0000","last_clean":"2023-09-30T02:06:18.364779+0000","last_became_active":"2023-09-30T02:06:17.855689+0000","last_became_peered":"2023-09-30T02:06:17.855689+0000","last_unstale":"2023-09-30T02:06:18.364779+0000","last_undegraded":"2023-09-30T02:06:18.364779+0000","last_fullsized":"2023-09-30T02:06:18.364779+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_clean_scrub_stamp":"2023-09-30T02:06:15.800912+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:18:08.185701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T02:06:18.359472+0000","last_change":"2023-09-30T02:06:17.806354+0000","last_active":"2023-09-30T02:06:18.359472+0000","last_peered":"2023-09-30T02:06:18.359472+0000","last_clean":"2023-09-30T02:06:18.359472+0000","last_became_active":"2023-09-30T02:06:17.706615+0000","last_became_peered":"2023-09-30T02:06:17.706615+0000","last_unstale":"2023-09-30T02:06:18.359472+0000","last_undegraded":"2023-09-30T02:06:18.359472+0000","last_fullsized":"2023-09-30T02:06:18.359472+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_clean_scrub_stamp":"2023-09-30T02:06:15.800912+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:08:20.786129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"25'76","reported_seq":117,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-09-30T02:06:21.647351+0000","last_change":"2023-09-30T02:06:11.645599+0000","last_active":"2023-09-30T02:06:21.647351+0000","last_peered":"2023-09-30T02:06:21.647351+0000","last_clean":"2023-09-30T02:06:21.647351+0000","last_became_active":"2023-09-30T02:06:11.592631+0000","last_became_peered":"2023-09-30T02:06:11.592631+0000","last_unstale":"2023-09-30T02:06:21.647351+0000","last_undegraded":"2023-09-30T02:06:21.647351+0000","last_fullsized":"2023-09-30T02:06:21.647351+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:06:06.666529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:06:06.666529+0000","last_clean_scrub_stamp":"2023-09-30T02:06:06.666529+0000","objects_scrubbed":0,"log_size":76,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:16:35.859435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T02:06:18.343393+0000","last_change":"2023-09-30T02:06:18.102959+0000","last_active":"2023-09-30T02:06:18.343393+0000","last_peered":"2023-09-30T02:06:18.343393+0000","last_clean":"2023-09-30T02:06:18.343393+0000","last_became_active":"2023-09-30T02:06:18.091407+0000","last_became_peered":"2023-09-30T02:06:18.091407+0000","last_unstale":"2023-09-30T02:06:18.343393+0000","last_undegraded":"2023-09-30T02:06:18.343393+0000","last_fullsized":"2023-09-30T02:06:18.343393+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_clean_scrub_stamp":"2023-09-30T02:06:15.800912+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:34:28.390487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T02:06:18.250861+0000","last_change":"2023-09-30T02:06:17.465018+0000","last_active":"2023-09-30T02:06:18.250861+0000","last_peered":"2023-09-30T02:06:18.250861+0000","last_clean":"2023-09-30T02:06:18.250861+0000","last_became_active":"2023-09-30T02:06:17.420574+0000","last_became_peered":"2023-09-30T02:06:17.420574+0000","last_unstale":"2023-09-30T02:06:18.250861+0000","last_undegraded":"2023-09-30T02:06:18.250861+0000","last_fullsized":"2023-09-30T02:06:18.250861+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_clean_scrub_stamp":"2023-09-30T02:06:15.800912+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:46:28.414555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T02:06:18.354965+0000","last_change":"2023-09-30T02:06:17.896913+0000","last_active":"2023-09-30T02:06:18.354965+0000","last_peered":"2023-09-30T02:06:18.354965+0000","last_clean":"2023-09-30T02:06:18.354965+0000","last_became_active":"2023-09-30T02:06:17.890282+0000","last_became_peered":"2023-09-30T02:06:17.890282+0000","last_unstale":"2023-09-30T02:06:18.354965+0000","last_undegraded":"2023-09-30T02:06:18.354965+0000","last_fullsized":"2023-09-30T02:06:18.354965+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_clean_scrub_stamp":"2023-09-30T02:06:15.800912+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:35:55.671531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-09-30T02:06:17.900193+0000","last_change":"2023-09-30T02:06:17.877329+0000","last_active":"2023-09-30T02:06:17.900193+0000","last_peered":"2023-09-30T02:06:17.900193+0000","last_clean":"2023-09-30T02:06:17.900193+0000","last_became_active":"2023-09-30T02:06:17.836124+0000","last_became_peered":"2023-09-30T02:06:17.836124+0000","last_unstale":"2023-09-30T02:06:17.900193+0000","last_undegraded":"2023-09-30T02:06:17.900193+0000","last_fullsized":"2023-09-30T02:06:17.900193+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_clean_scrub_stamp":"2023-09-30T02:06:15.800912+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T14:04:06.678156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":19,"seq":81604378692,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5208,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366632,"statfs":{"total":96636764160,"available":96631431168,"internally_reserved":0,"allocated":212992,"data_stored":75040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":12.859999999999999,"5min":16.204000000000001,"15min":26.364000000000001},"min":{"1min":8.8520000000000003,"5min":7.1429999999999998,"15min":7.1429999999999998},"max":{"1min":17.263000000000002,"5min":414.25400000000002,"15min":414.25400000000002},"last":11.553000000000001},{"interface":"front","average":{"1min":12.984,"5min":17.393999999999998,"15min":29.042999999999999},"min":{"1min":7.1289999999999996,"5min":7.1289999999999996,"15min":7.1289999999999996},"max":{"1min":21.555,"5min":421.178,"15min":421.178},"last":10.301}]},{"osd":1,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":13.353,"5min":15.817,"15min":24.888999999999999},"min":{"1min":8.9610000000000003,"5min":6.8200000000000003,"15min":6.8200000000000003},"max":{"1min":16.489999999999998,"5min":326.60899999999998,"15min":326.60899999999998},"last":9.7940000000000005},{"interface":"front","average":{"1min":14.826000000000001,"5min":19.149999999999999,"15min":31.222999999999999},"min":{"1min":8.5099999999999998,"5min":8.5079999999999991,"15min":8.5079999999999991},"max":{"1min":24.559999999999999,"5min":418.35500000000002,"15min":418.35500000000002},"last":21.640000000000001}]},{"osd":2,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":16.120000000000001,"5min":21.582000000000001,"15min":33.603999999999999},"min":{"1min":9.2850000000000001,"5min":8.3960000000000008,"15min":8.3960000000000008},"max":{"1min":23.693999999999999,"5min":423.46699999999998,"15min":423.46699999999998},"last":22.791},{"interface":"front","average":{"1min":14.952,"5min":19.960999999999999,"15min":33.359999999999999},"min":{"1min":9.0749999999999993,"5min":8.4000000000000004,"15min":8.4000000000000004},"max":{"1min":23.123000000000001,"5min":438.697,"15min":438.697},"last":14.054}]},{"osd":4,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":15.225,"5min":19.898,"15min":32.472000000000001},"min":{"1min":9.5370000000000008,"5min":8.4030000000000005,"15min":8.4030000000000005},"max":{"1min":26.390000000000001,"5min":443.36399999999998,"15min":443.36399999999998},"last":12.300000000000001},{"interface":"front","average":{"1min":17.562999999999999,"5min":22.780999999999999,"15min":35.773000000000003},"min":{"1min":10.282,"5min":10.282,"15min":10.282},"max":{"1min":23.670000000000002,"5min":453.20699999999999,"15min":453.20699999999999},"last":20.591999999999999}]},{"osd":5,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":14.875,"5min":20.896999999999998,"15min":33.405000000000001},"min":{"1min":8.2430000000000003,"5min":8.141,"15min":8.141},"max":{"1min":22.359000000000002,"5min":443.94600000000003,"15min":443.94600000000003},"last":15.303000000000001},{"interface":"front","average":{"1min":19.251000000000001,"5min":22.654,"15min":32.534999999999997},"min":{"1min":11.301,"5min":10.746,"15min":10.746},"max":{"1min":25.305,"5min":335.11799999999999,"15min":335.11799999999999},"last":18.221}]},{"osd":6,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":18.507000000000001,"5min":22.931999999999999,"15min":36.216999999999999},"min":{"1min":12.955,"5min":9.4009999999999998,"15min":9.4009999999999998},"max":{"1min":28.02,"5min":432.34699999999998,"15min":432.34699999999998},"last":23.867999999999999},{"interface":"front","average":{"1min":21.754000000000001,"5min":24.712,"15min":34.631999999999998},"min":{"1min":18.292999999999999,"5min":13.018000000000001,"15min":13.018000000000001},"max":{"1min":27.356999999999999,"5min":334.178,"15min":334.178},"last":16.396000000000001}]},{"osd":7,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":17.149000000000001,"5min":23.225000000000001,"15min":35.642000000000003},"min":{"1min":10.933999999999999,"5min":10.928000000000001,"15min":10.928000000000001},"max":{"1min":23.986999999999998,"5min":461.42000000000002,"15min":461.42000000000002},"last":17.193999999999999},{"interface":"front","average":{"1min":22.254000000000001,"5min":26.744,"15min":39.024000000000001},"min":{"1min":17.257000000000001,"5min":15.686,"15min":15.686},"max":{"1min":25.774000000000001,"5min":431.27800000000002,"15min":431.27800000000002},"last":19.611999999999998}]}]},{"osd":1,"up_from":18,"seq":77309411395,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6044,"kb_used_data":600,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":94365796,"statfs":{"total":96636764160,"available":96630575104,"internally_reserved":0,"allocated":614400,"data_stored":472880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":13.101000000000001,"5min":13.603999999999999,"15min":13.84},"min":{"1min":8.7560000000000002,"5min":7.1639999999999997,"15min":7.1639999999999997},"max":{"1min":19.596,"5min":24.626999999999999,"15min":24.626999999999999},"last":14.039999999999999},{"interface":"front","average":{"1min":11.407,"5min":12.699,"15min":14.554},"min":{"1min":6.4859999999999998,"5min":6.4859999999999998,"15min":6.4859999999999998},"max":{"1min":17.785,"5min":38.213999999999999,"15min":38.213999999999999},"last":15.19}]},{"osd":2,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":12.917999999999999,"5min":13.146000000000001,"15min":13.359999999999999},"min":{"1min":8.4949999999999992,"5min":6.9509999999999996,"15min":6.9509999999999996},"max":{"1min":20.673999999999999,"5min":39.070999999999998,"15min":39.070999999999998},"last":14.342000000000001},{"interface":"front","average":{"1min":14.161,"5min":15.342000000000001,"15min":16.507999999999999},"min":{"1min":8.6050000000000004,"5min":8.6050000000000004,"15min":8.6050000000000004},"max":{"1min":20.436,"5min":28.981999999999999,"15min":28.981999999999999},"last":21.966000000000001}]},{"osd":3,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":18.742999999999999,"5min":28.812999999999999,"15min":56.773000000000003},"min":{"1min":13.654999999999999,"5min":9.2110000000000003,"15min":9.2110000000000003},"max":{"1min":23.431999999999999,"5min":789.26099999999997,"15min":789.26099999999997},"last":24.800999999999998},{"interface":"front","average":{"1min":16.210999999999999,"5min":26.545999999999999,"15min":55.965000000000003},"min":{"1min":7.8869999999999996,"5min":7.8869999999999996,"15min":7.8869999999999996},"max":{"1min":24.469000000000001,"5min":827.53899999999999,"15min":827.53899999999999},"last":11.007}]},{"osd":4,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":14.420999999999999,"5min":21.696999999999999,"15min":34.310000000000002},"min":{"1min":11.004,"5min":8.7029999999999994,"15min":8.7029999999999994},"max":{"1min":20.788,"5min":390.31700000000001,"15min":390.31700000000001},"last":18.489999999999998},{"interface":"front","average":{"1min":17.638000000000002,"5min":21.739000000000001,"15min":34.366999999999997},"min":{"1min":10.234,"5min":10.234,"15min":10.234},"max":{"1min":24.907,"5min":381.255,"15min":381.255},"last":11.484}]},{"osd":5,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":13.234999999999999,"5min":14.538,"15min":15.808999999999999},"min":{"1min":9.2680000000000007,"5min":7.6849999999999996,"15min":7.6849999999999996},"max":{"1min":21.724,"5min":31.677,"15min":31.677},"last":19.353999999999999},{"interface":"front","average":{"1min":20.417000000000002,"5min":20.757000000000001,"15min":21.004999999999999},"min":{"1min":9.9860000000000007,"5min":9.9860000000000007,"15min":9.9860000000000007},"max":{"1min":26.379000000000001,"5min":31.449999999999999,"15min":31.449999999999999},"last":13.403}]},{"osd":6,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":18.373999999999999,"5min":19.23,"15min":21.132000000000001},"min":{"1min":12.045,"5min":9.7560000000000002,"15min":9.7560000000000002},"max":{"1min":24.495999999999999,"5min":41.316000000000003,"15min":41.316000000000003},"last":20.221},{"interface":"front","average":{"1min":16.523,"5min":17.629999999999999,"15min":19.443999999999999},"min":{"1min":10.407,"5min":9.2699999999999996,"15min":9.2699999999999996},"max":{"1min":22.161999999999999,"5min":53.012,"15min":53.012},"last":21.260999999999999}]},{"osd":7,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":19.762,"5min":24.777999999999999,"15min":37.265999999999998},"min":{"1min":14.295999999999999,"5min":9.5310000000000006,"15min":9.5310000000000006},"max":{"1min":23.609000000000002,"5min":332.072,"15min":332.072},"last":17.216999999999999},{"interface":"front","average":{"1min":17.829000000000001,"5min":20.178999999999998,"15min":24.689},"min":{"1min":11.316000000000001,"5min":8.141,"15min":8.141},"max":{"1min":23.631,"5min":108.00700000000001,"15min":108.00700000000001},"last":23.73}]}]},{"osd":7,"up_from":14,"seq":60129542213,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5272,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":94366568,"statfs":{"total":96636764160,"available":96631365632,"internally_reserved":0,"allocated":212992,"data_stored":75040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:11:21 2023","interfaces":[{"interface":"back","average":{"1min":8.6750000000000007,"5min":8.9369999999999994,"15min":8.9199999999999999},"min":{"1min":6.2140000000000004,"5min":5.3120000000000003,"15min":5.3120000000000003},"max":{"1min":16.916,"5min":19.491,"15min":19.491},"last":7.3099999999999996},{"interface":"front","average":{"1min":10.153,"5min":9.9879999999999995,"15min":9.7560000000000002},"min":{"1min":7.3099999999999996,"5min":6.2850000000000001,"15min":6.2850000000000001},"max":{"1min":19.263000000000002,"5min":20.952000000000002,"15min":20.952000000000002},"last":10.08}]},{"osd":1,"last update":"Sat Sep 30 02:11:21 2023","interfaces":[{"interface":"back","average":{"1min":10.753,"5min":11.709,"15min":14.192},"min":{"1min":6.6630000000000003,"5min":6.6630000000000003,"15min":6.6630000000000003},"max":{"1min":18.591000000000001,"5min":89.454999999999998,"15min":89.454999999999998},"last":11.307},{"interface":"front","average":{"1min":11.548999999999999,"5min":12.834,"15min":14.573},"min":{"1min":8.0909999999999993,"5min":7.0700000000000003,"15min":7.0700000000000003},"max":{"1min":19.093,"5min":83.695999999999998,"15min":83.695999999999998},"last":20.039000000000001}]},{"osd":2,"last update":"Sat Sep 30 02:11:21 2023","interfaces":[{"interface":"back","average":{"1min":13.856,"5min":13.919,"15min":13.823},"min":{"1min":8.1240000000000006,"5min":7.4669999999999996,"15min":7.4669999999999996},"max":{"1min":20.259,"5min":21.370999999999999,"15min":21.370999999999999},"last":14.94},{"interface":"front","average":{"1min":14.368,"5min":14.385,"15min":15.127000000000001},"min":{"1min":10.176,"5min":7.8920000000000003,"15min":7.8920000000000003},"max":{"1min":19.542999999999999,"5min":21.712,"15min":21.712},"last":13.404999999999999}]},{"osd":3,"last update":"Sat Sep 30 02:11:21 2023","interfaces":[{"interface":"back","average":{"1min":15.417999999999999,"5min":15.926,"15min":15.646000000000001},"min":{"1min":11.319000000000001,"5min":7.1710000000000003,"15min":7.1710000000000003},"max":{"1min":21.238,"5min":24.276,"15min":24.276},"last":21.707999999999998},{"interface":"front","average":{"1min":16.335999999999999,"5min":16.087,"15min":16.295000000000002},"min":{"1min":12.477,"5min":7.9630000000000001,"15min":7.9630000000000001},"max":{"1min":22.021999999999998,"5min":24.716000000000001,"15min":24.716000000000001},"last":17.811}]},{"osd":4,"last update":"Sat Sep 30 02:11:12 2023","interfaces":[{"interface":"back","average":{"1min":16.693999999999999,"5min":17.527000000000001,"15min":19.007999999999999},"min":{"1min":9.4380000000000006,"5min":5.7039999999999997,"15min":5.7039999999999997},"max":{"1min":23.076000000000001,"5min":99.298000000000002,"15min":99.298000000000002},"last":15.726000000000001},{"interface":"front","average":{"1min":12.294,"5min":15.105,"15min":17.399000000000001},"min":{"1min":7.1509999999999998,"5min":7.1509999999999998,"15min":7.1509999999999998},"max":{"1min":17.899000000000001,"5min":86.938999999999993,"15min":86.938999999999993},"last":18.771000000000001}]},{"osd":5,"last update":"Sat Sep 30 02:11:12 2023","interfaces":[{"interface":"back","average":{"1min":15.673999999999999,"5min":17.282,"15min":19.91},"min":{"1min":8.6509999999999998,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":21.527000000000001,"5min":118.675,"15min":118.675},"last":16.376999999999999},{"interface":"front","average":{"1min":18.716999999999999,"5min":19.459,"15min":21.693999999999999},"min":{"1min":16.239999999999998,"5min":8.7330000000000005,"15min":8.7330000000000005},"max":{"1min":23.338999999999999,"5min":114.14400000000001,"15min":114.14400000000001},"last":22.946999999999999}]},{"osd":6,"last update":"Sat Sep 30 02:11:12 2023","interfaces":[{"interface":"back","average":{"1min":19.045000000000002,"5min":20.443999999999999,"15min":23.024000000000001},"min":{"1min":8.4359999999999999,"5min":8.4359999999999999,"15min":8.4359999999999999},"max":{"1min":24.417000000000002,"5min":111.048,"15min":111.048},"last":21.039000000000001},{"interface":"front","average":{"1min":18.745000000000001,"5min":19.427,"15min":22.183},"min":{"1min":12.542,"5min":9.8140000000000001,"15min":9.8140000000000001},"max":{"1min":23.036000000000001,"5min":115.241,"15min":115.241},"last":13.801}]}]},{"osd":0,"up_from":18,"seq":77309411396,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5208,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366632,"statfs":{"total":96636764160,"available":96631431168,"internally_reserved":0,"allocated":212992,"data_stored":75040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 30 02:11:19 2023","interfaces":[{"interface":"back","average":{"1min":10.754,"5min":16.196000000000002,"15min":29.155000000000001},"min":{"1min":7.2089999999999996,"5min":7.2089999999999996,"15min":7.2089999999999996},"max":{"1min":19.498999999999999,"5min":425.49599999999998,"15min":425.49599999999998},"last":14.003},{"interface":"front","average":{"1min":13.093,"5min":20.338000000000001,"15min":39.884},"min":{"1min":8.1210000000000004,"5min":7.9269999999999996,"15min":7.9269999999999996},"max":{"1min":18.349,"5min":667.81100000000004,"15min":667.81100000000004},"last":13.289999999999999}]},{"osd":2,"last update":"Sat Sep 30 02:11:19 2023","interfaces":[{"interface":"back","average":{"1min":12.025,"5min":14.532999999999999,"15min":18.928999999999998},"min":{"1min":8.2110000000000003,"5min":7.718,"15min":7.718},"max":{"1min":16.954000000000001,"5min":162.50700000000001,"15min":162.50700000000001},"last":11.449999999999999},{"interface":"front","average":{"1min":17.457000000000001,"5min":17.991,"15min":20.966999999999999},"min":{"1min":10.747,"5min":8.5079999999999991,"15min":8.5079999999999991},"max":{"1min":23.617000000000001,"5min":131.863,"15min":131.863},"last":14.81}]},{"osd":3,"last update":"Sat Sep 30 02:11:19 2023","interfaces":[{"interface":"back","average":{"1min":13.837,"5min":14.763,"15min":15.271000000000001},"min":{"1min":8.1020000000000003,"5min":7.4859999999999998,"15min":7.4859999999999998},"max":{"1min":22.571999999999999,"5min":24.763000000000002,"15min":24.763000000000002},"last":9.7050000000000001},{"interface":"front","average":{"1min":18.533999999999999,"5min":18.309999999999999,"15min":18.783000000000001},"min":{"1min":10.459,"5min":9.8569999999999993,"15min":9.8569999999999993},"max":{"1min":23.57,"5min":27.309000000000001,"15min":27.309000000000001},"last":18.625}]},{"osd":4,"last update":"Sat Sep 30 02:11:19 2023","interfaces":[{"interface":"back","average":{"1min":16.027999999999999,"5min":16.099,"15min":19.271000000000001},"min":{"1min":9.9179999999999993,"5min":9.0700000000000003,"15min":9.0700000000000003},"max":{"1min":22.122,"5min":102.456,"15min":102.456},"last":15.573},{"interface":"front","average":{"1min":13.881,"5min":17.271000000000001,"15min":20.933},"min":{"1min":8.7929999999999993,"5min":8.7929999999999993,"15min":8.7929999999999993},"max":{"1min":19.463999999999999,"5min":98.293999999999997,"15min":98.293999999999997},"last":17.890000000000001}]},{"osd":5,"last update":"Sat Sep 30 02:11:19 2023","interfaces":[{"interface":"back","average":{"1min":20.268999999999998,"5min":20.559999999999999,"15min":22.850000000000001},"min":{"1min":14.006,"5min":10.726000000000001,"15min":10.726000000000001},"max":{"1min":28.995000000000001,"5min":79.566999999999993,"15min":79.566999999999993},"last":17.129999999999999},{"interface":"front","average":{"1min":16.812999999999999,"5min":18.582999999999998,"15min":21.466999999999999},"min":{"1min":9.7599999999999998,"5min":9.7599999999999998,"15min":9.7599999999999998},"max":{"1min":24.213000000000001,"5min":109.80500000000001,"15min":109.80500000000001},"last":8.8889999999999993}]},{"osd":6,"last update":"Sat Sep 30 02:11:19 2023","interfaces":[{"interface":"back","average":{"1min":18.077999999999999,"5min":20.367000000000001,"15min":26.062999999999999},"min":{"1min":11.140000000000001,"5min":9.2829999999999995,"15min":9.2829999999999995},"max":{"1min":24.276,"5min":161.01499999999999,"15min":161.01499999999999},"last":12.545},{"interface":"front","average":{"1min":22.562000000000001,"5min":22.975000000000001,"15min":26.202999999999999},"min":{"1min":13.904,"5min":13.163,"15min":13.163},"max":{"1min":27.786000000000001,"5min":118.71599999999999,"15min":118.71599999999999},"last":22.385000000000002}]},{"osd":7,"last update":"Sat Sep 30 02:11:19 2023","interfaces":[{"interface":"back","average":{"1min":20.117999999999999,"5min":20.393000000000001,"15min":23.483000000000001},"min":{"1min":11.741,"5min":9.8810000000000002,"15min":9.8810000000000002},"max":{"1min":26.491,"5min":111.86,"15min":111.86},"last":16.395},{"interface":"front","average":{"1min":21.664000000000001,"5min":22.388999999999999,"15min":26.010999999999999},"min":{"1min":15.186999999999999,"5min":10.237,"15min":10.237},"max":{"1min":27.012,"5min":126.889,"15min":126.889},"last":21.527999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574916,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5208,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366632,"statfs":{"total":96636764160,"available":96631431168,"internally_reserved":0,"allocated":212992,"data_stored":75040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:11:20 2023","interfaces":[{"interface":"back","average":{"1min":9.6969999999999992,"5min":9.8569999999999993,"15min":9.7870000000000008},"min":{"1min":6.5830000000000002,"5min":5.774,"15min":5.774},"max":{"1min":16.530999999999999,"5min":19.709,"15min":19.709},"last":6.8959999999999999},{"interface":"front","average":{"1min":10.919,"5min":10.390000000000001,"15min":11.138999999999999},"min":{"1min":6.1239999999999997,"5min":5.0419999999999998,"15min":5.0419999999999998},"max":{"1min":19.603000000000002,"5min":23.134,"15min":23.134},"last":10.864000000000001}]},{"osd":1,"last update":"Sat Sep 30 02:11:20 2023","interfaces":[{"interface":"back","average":{"1min":11.829000000000001,"5min":12.981999999999999,"15min":14.561999999999999},"min":{"1min":7.351,"5min":6.3010000000000002,"15min":6.3010000000000002},"max":{"1min":17.946999999999999,"5min":27.242999999999999,"15min":27.242999999999999},"last":14.702},{"interface":"front","average":{"1min":11.532999999999999,"5min":11.537000000000001,"15min":12.59},"min":{"1min":7.9770000000000003,"5min":7.4989999999999997,"15min":7.4989999999999997},"max":{"1min":18.957000000000001,"5min":24.388999999999999,"15min":24.388999999999999},"last":18.056999999999999}]},{"osd":2,"last update":"Sat Sep 30 02:11:20 2023","interfaces":[{"interface":"back","average":{"1min":14.006,"5min":14.630000000000001,"15min":15.132},"min":{"1min":9.6690000000000005,"5min":7.7039999999999997,"15min":7.7039999999999997},"max":{"1min":19.521000000000001,"5min":24.510999999999999,"15min":24.510999999999999},"last":14.407999999999999},{"interface":"front","average":{"1min":12.595000000000001,"5min":13.534000000000001,"15min":14.086},"min":{"1min":7.9699999999999998,"5min":7.9699999999999998,"15min":7.9699999999999998},"max":{"1min":20.085000000000001,"5min":25.033999999999999,"15min":25.033999999999999},"last":13.747999999999999}]},{"osd":3,"last update":"Sat Sep 30 02:11:20 2023","interfaces":[{"interface":"back","average":{"1min":14.676,"5min":19.765999999999998,"15min":31.925000000000001},"min":{"1min":8.3770000000000007,"5min":8.2089999999999996,"15min":8.2089999999999996},"max":{"1min":20.157,"5min":540.52099999999996,"15min":540.52099999999996},"last":17.163},{"interface":"front","average":{"1min":15.218,"5min":21.224,"15min":35.917999999999999},"min":{"1min":9.4489999999999998,"5min":7.9649999999999999,"15min":7.9649999999999999},"max":{"1min":19.452999999999999,"5min":644.31600000000003,"15min":644.31600000000003},"last":21.873999999999999}]},{"osd":4,"last update":"Sat Sep 30 02:11:15 2023","interfaces":[{"interface":"back","average":{"1min":17.922000000000001,"5min":18.664999999999999,"15min":22.556999999999999},"min":{"1min":13.959,"5min":9.0969999999999995,"15min":9.0969999999999995},"max":{"1min":21.425999999999998,"5min":179.749,"15min":179.749},"last":23.646000000000001},{"interface":"front","average":{"1min":15.129,"5min":15.641999999999999,"15min":19.817},"min":{"1min":8.7550000000000008,"5min":7.6070000000000002,"15min":7.6070000000000002},"max":{"1min":19.663,"5min":174.52799999999999,"15min":174.52799999999999},"last":19.271000000000001}]},{"osd":5,"last update":"Sat Sep 30 02:11:15 2023","interfaces":[{"interface":"back","average":{"1min":16.963000000000001,"5min":16.859000000000002,"15min":16.969000000000001},"min":{"1min":9.6210000000000004,"5min":8.2379999999999995,"15min":8.2379999999999995},"max":{"1min":21.975999999999999,"5min":36.709000000000003,"15min":36.709000000000003},"last":19.513999999999999},{"interface":"front","average":{"1min":15.141999999999999,"5min":16.25,"15min":16.556000000000001},"min":{"1min":9.0809999999999995,"5min":5.0279999999999996,"15min":5.0279999999999996},"max":{"1min":21.393999999999998,"5min":27.364999999999998,"15min":27.364999999999998},"last":20.292000000000002}]},{"osd":7,"last update":"Sat Sep 30 02:11:09 2023","interfaces":[{"interface":"back","average":{"1min":19.030000000000001,"5min":19.597000000000001,"15min":22.396999999999998},"min":{"1min":13.644,"5min":10.367000000000001,"15min":10.367000000000001},"max":{"1min":22.771000000000001,"5min":105.801,"15min":105.801},"last":20.815000000000001},{"interface":"front","average":{"1min":17.553000000000001,"5min":19.193999999999999,"15min":20.960000000000001},"min":{"1min":7.3120000000000003,"5min":7.3120000000000003,"15min":7.3120000000000003},"max":{"1min":23.283000000000001,"5min":90.156000000000006,"15min":90.156000000000006},"last":15.582000000000001}]}]},{"osd":4,"up_from":15,"seq":64424509509,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5208,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366632,"statfs":{"total":96636764160,"available":96631431168,"internally_reserved":0,"allocated":212992,"data_stored":75040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:11:17 2023","interfaces":[{"interface":"back","average":{"1min":11.865,"5min":10.567,"15min":9.4399999999999995},"min":{"1min":6.3970000000000002,"5min":5.2699999999999996,"15min":5.2699999999999996},"max":{"1min":20.655000000000001,"5min":24.532,"15min":24.532},"last":11.471},{"interface":"front","average":{"1min":11.137,"5min":10.974,"15min":11.926},"min":{"1min":7.0190000000000001,"5min":5.4530000000000003,"15min":5.4530000000000003},"max":{"1min":21.841000000000001,"5min":25.225000000000001,"15min":25.225000000000001},"last":7.9080000000000004}]},{"osd":1,"last update":"Sat Sep 30 02:11:17 2023","interfaces":[{"interface":"back","average":{"1min":12.827,"5min":13.271000000000001,"15min":13.897},"min":{"1min":8.3960000000000008,"5min":6.2830000000000004,"15min":6.2830000000000004},"max":{"1min":17.864000000000001,"5min":28.917000000000002,"15min":28.917000000000002},"last":15.699999999999999},{"interface":"front","average":{"1min":10.146000000000001,"5min":11.148,"15min":11.789999999999999},"min":{"1min":7.4059999999999997,"5min":7.0250000000000004,"15min":7.0250000000000004},"max":{"1min":19.279,"5min":30.922999999999998,"15min":30.922999999999998},"last":9.0760000000000005}]},{"osd":2,"last update":"Sat Sep 30 02:11:17 2023","interfaces":[{"interface":"back","average":{"1min":14.563000000000001,"5min":13.641999999999999,"15min":14.407999999999999},"min":{"1min":7.8300000000000001,"5min":7.1609999999999996,"15min":7.1609999999999996},"max":{"1min":19.428000000000001,"5min":21.175000000000001,"15min":21.175000000000001},"last":13.936999999999999},{"interface":"front","average":{"1min":12.637,"5min":13.872,"15min":14.563000000000001},"min":{"1min":6.7060000000000004,"5min":6.7060000000000004,"15min":6.7060000000000004},"max":{"1min":18.747,"5min":27.533999999999999,"15min":27.533999999999999},"last":12.906000000000001}]},{"osd":3,"last update":"Sat Sep 30 02:11:23 2023","interfaces":[{"interface":"back","average":{"1min":16.146000000000001,"5min":16.042999999999999,"15min":15.887},"min":{"1min":9.4719999999999995,"5min":9.4719999999999995,"15min":9.4719999999999995},"max":{"1min":23.526,"5min":27.632999999999999,"15min":27.632999999999999},"last":21.678000000000001},{"interface":"front","average":{"1min":15.352,"5min":16.143999999999998,"15min":17.184999999999999},"min":{"1min":8.2599999999999998,"5min":7.9269999999999996,"15min":7.9269999999999996},"max":{"1min":23.113,"5min":30.495000000000001,"15min":30.495000000000001},"last":18.611000000000001}]},{"osd":5,"last update":"Sat Sep 30 02:11:11 2023","interfaces":[{"interface":"back","average":{"1min":16.295999999999999,"5min":18.059000000000001,"15min":20.965},"min":{"1min":12.298,"5min":7.9080000000000004,"15min":7.9080000000000004},"max":{"1min":21.044,"5min":99.784000000000006,"15min":99.784000000000006},"last":24.344000000000001},{"interface":"front","average":{"1min":13.371,"5min":15.058,"15min":17.594999999999999},"min":{"1min":8.3030000000000008,"5min":7.609,"15min":7.609},"max":{"1min":19.103000000000002,"5min":98.634,"15min":98.634},"last":16.399999999999999}]},{"osd":6,"last update":"Sat Sep 30 02:11:11 2023","interfaces":[{"interface":"back","average":{"1min":16.995000000000001,"5min":16.553999999999998,"15min":16.952999999999999},"min":{"1min":7.8949999999999996,"5min":7.8949999999999996,"15min":7.8949999999999996},"max":{"1min":22.498999999999999,"5min":27.277999999999999,"15min":27.277999999999999},"last":19.588999999999999},{"interface":"front","average":{"1min":15.564,"5min":16.422000000000001,"15min":16.088999999999999},"min":{"1min":9.109,"5min":6.5810000000000004,"15min":6.5810000000000004},"max":{"1min":20.640999999999998,"5min":28.640000000000001,"15min":28.640000000000001},"last":14.936}]},{"osd":7,"last update":"Sat Sep 30 02:11:11 2023","interfaces":[{"interface":"back","average":{"1min":17.225000000000001,"5min":18.161000000000001,"15min":18.513000000000002},"min":{"1min":7.79,"5min":7.79,"15min":7.79},"max":{"1min":23.047000000000001,"5min":31.143000000000001,"15min":31.143000000000001},"last":20.795999999999999},{"interface":"front","average":{"1min":17.949000000000002,"5min":18.007000000000001,"15min":18.579000000000001},"min":{"1min":10.542999999999999,"5min":8.6509999999999998,"15min":8.6509999999999998},"max":{"1min":22.966999999999999,"5min":30.529,"15min":30.529},"last":23.006}]}]},{"osd":5,"up_from":15,"seq":64424509509,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6108,"kb_used_data":600,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":94365732,"statfs":{"total":96636764160,"available":96630509568,"internally_reserved":0,"allocated":614400,"data_stored":472880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:11:18 2023","interfaces":[{"interface":"back","average":{"1min":10.455,"5min":10.214,"15min":11.779999999999999},"min":{"1min":6.1040000000000001,"5min":5.7350000000000003,"15min":5.7350000000000003},"max":{"1min":19.498999999999999,"5min":22.452999999999999,"15min":22.452999999999999},"last":14.721},{"interface":"front","average":{"1min":10.673999999999999,"5min":10.722,"15min":11.548},"min":{"1min":6.6929999999999996,"5min":5.8879999999999999,"15min":5.8879999999999999},"max":{"1min":20.34,"5min":22.821999999999999,"15min":22.821999999999999},"last":8.5670000000000002}]},{"osd":1,"last update":"Sat Sep 30 02:11:18 2023","interfaces":[{"interface":"back","average":{"1min":14.335000000000001,"5min":13.829000000000001,"15min":15.194000000000001},"min":{"1min":8.9589999999999996,"5min":7.29,"15min":7.29},"max":{"1min":22.402999999999999,"5min":47.451999999999998,"15min":47.451999999999998},"last":8.3119999999999994},{"interface":"front","average":{"1min":11.304,"5min":11.545,"15min":12.558},"min":{"1min":7.774,"5min":6.6130000000000004,"15min":6.6130000000000004},"max":{"1min":15.734999999999999,"5min":48.841999999999999,"15min":48.841999999999999},"last":8.0190000000000001}]},{"osd":2,"last update":"Sat Sep 30 02:11:18 2023","interfaces":[{"interface":"back","average":{"1min":15.868,"5min":13.461,"15min":11.835000000000001},"min":{"1min":8.2829999999999995,"5min":6.8250000000000002,"15min":6.8250000000000002},"max":{"1min":27.501000000000001,"5min":27.501000000000001,"15min":27.501000000000001},"last":17.762},{"interface":"front","average":{"1min":12.898999999999999,"5min":14.695,"15min":15.513},"min":{"1min":7.8150000000000004,"5min":7.5369999999999999,"15min":7.5369999999999999},"max":{"1min":19.411000000000001,"5min":26.576000000000001,"15min":26.576000000000001},"last":15.177}]},{"osd":3,"last update":"Sat Sep 30 02:11:30 2023","interfaces":[{"interface":"back","average":{"1min":14.194000000000001,"5min":15.137,"15min":15.063000000000001},"min":{"1min":7.6980000000000004,"5min":7.2619999999999996,"15min":7.2619999999999996},"max":{"1min":21.891999999999999,"5min":26.895,"15min":26.895},"last":9.9550000000000001},{"interface":"front","average":{"1min":15.656000000000001,"5min":17.158999999999999,"15min":17.841999999999999},"min":{"1min":11.603,"5min":11.140000000000001,"15min":11.140000000000001},"max":{"1min":19.952000000000002,"5min":24.702999999999999,"15min":24.702999999999999},"last":13.757}]},{"osd":4,"last update":"Sat Sep 30 02:11:18 2023","interfaces":[{"interface":"back","average":{"1min":15.975,"5min":17.355,"15min":18.113},"min":{"1min":10.063000000000001,"5min":8.8160000000000007,"15min":8.8160000000000007},"max":{"1min":22.620999999999999,"5min":26.710000000000001,"15min":26.710000000000001},"last":18.731000000000002},{"interface":"front","average":{"1min":16.137,"5min":16.138999999999999,"15min":15.467000000000001},"min":{"1min":10.363,"5min":6.0270000000000001,"15min":6.0270000000000001},"max":{"1min":23.376000000000001,"5min":28.344999999999999,"15min":28.344999999999999},"last":13.065}]},{"osd":6,"last update":"Sat Sep 30 02:11:18 2023","interfaces":[{"interface":"back","average":{"1min":17.373000000000001,"5min":16.178999999999998,"15min":14.978999999999999},"min":{"1min":10.907999999999999,"5min":7.0679999999999996,"15min":7.0679999999999996},"max":{"1min":22.890999999999998,"5min":23.573,"15min":23.573},"last":16.010000000000002},{"interface":"front","average":{"1min":14.722,"5min":16.218,"15min":16.617000000000001},"min":{"1min":7.8550000000000004,"5min":7.5540000000000003,"15min":7.5540000000000003},"max":{"1min":20.41,"5min":27.763999999999999,"15min":27.763999999999999},"last":16.571000000000002}]},{"osd":7,"last update":"Sat Sep 30 02:11:18 2023","interfaces":[{"interface":"back","average":{"1min":17.943999999999999,"5min":18.265999999999998,"15min":19.606000000000002},"min":{"1min":12.4,"5min":10.584,"15min":10.584},"max":{"1min":22.768000000000001,"5min":28.951000000000001,"15min":28.951000000000001},"last":20.495999999999999},{"interface":"front","average":{"1min":17.033999999999999,"5min":16.844000000000001,"15min":16.053999999999998},"min":{"1min":11.109,"5min":7.0490000000000004,"15min":7.0490000000000004},"max":{"1min":21.635999999999999,"5min":25.151,"15min":25.151},"last":19.292999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411395,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5208,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366632,"statfs":{"total":96636764160,"available":96631431168,"internally_reserved":0,"allocated":212992,"data_stored":75040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:11:21 2023","interfaces":[{"interface":"back","average":{"1min":12.927,"5min":13.202999999999999,"15min":13.711},"min":{"1min":7.1929999999999996,"5min":6.665,"15min":6.665},"max":{"1min":20.434000000000001,"5min":69.003,"15min":69.003},"last":9.2360000000000007},{"interface":"front","average":{"1min":13.492000000000001,"5min":14.566000000000001,"15min":16.558},"min":{"1min":8.6880000000000006,"5min":8.1329999999999991,"15min":8.1329999999999991},"max":{"1min":25.254000000000001,"5min":91.331999999999994,"15min":91.331999999999994},"last":16.228000000000002}]},{"osd":1,"last update":"Sat Sep 30 02:11:21 2023","interfaces":[{"interface":"back","average":{"1min":14.648999999999999,"5min":16.484999999999999,"15min":20.882000000000001},"min":{"1min":9.4800000000000004,"5min":7.9630000000000001,"15min":7.9630000000000001},"max":{"1min":21.640999999999998,"5min":145.952,"15min":145.952},"last":13.198},{"interface":"front","average":{"1min":15.581,"5min":17.215,"15min":21.452000000000002},"min":{"1min":9.6679999999999993,"5min":8.7469999999999999,"15min":8.7469999999999999},"max":{"1min":25.678000000000001,"5min":155.72900000000001,"15min":155.72900000000001},"last":11.943}]},{"osd":3,"last update":"Sat Sep 30 02:11:21 2023","interfaces":[{"interface":"back","average":{"1min":16.968,"5min":16.724,"15min":17.524000000000001},"min":{"1min":10.601000000000001,"5min":10.124000000000001,"15min":10.124000000000001},"max":{"1min":32.027000000000001,"5min":39.378999999999998,"15min":39.378999999999998},"last":19.701000000000001},{"interface":"front","average":{"1min":17.798999999999999,"5min":18.437000000000001,"15min":18.405000000000001},"min":{"1min":8.9250000000000007,"5min":8.9250000000000007,"15min":8.9250000000000007},"max":{"1min":24.704999999999998,"5min":36.649000000000001,"15min":36.649000000000001},"last":10.896000000000001}]},{"osd":4,"last update":"Sat Sep 30 02:11:21 2023","interfaces":[{"interface":"back","average":{"1min":18.395,"5min":18.151,"15min":18.896999999999998},"min":{"1min":11.792999999999999,"5min":8.9359999999999999,"15min":8.9359999999999999},"max":{"1min":29.699000000000002,"5min":39.186999999999998,"15min":39.186999999999998},"last":20.402000000000001},{"interface":"front","average":{"1min":11.427,"5min":11.253,"15min":12.537000000000001},"min":{"1min":6.7329999999999997,"5min":6.7329999999999997,"15min":6.7329999999999997},"max":{"1min":23.036999999999999,"5min":31.32,"15min":31.32},"last":9.5489999999999995}]},{"osd":5,"last update":"Sat Sep 30 02:11:21 2023","interfaces":[{"interface":"back","average":{"1min":19.495000000000001,"5min":18.518999999999998,"15min":19.271000000000001},"min":{"1min":12.682,"5min":8.3499999999999996,"15min":8.3499999999999996},"max":{"1min":26.695,"5min":43.409999999999997,"15min":43.409999999999997},"last":18.501999999999999},{"interface":"front","average":{"1min":20.788,"5min":21.66,"15min":22.609999999999999},"min":{"1min":11.974,"5min":11.974,"15min":11.974},"max":{"1min":33.695999999999998,"5min":40.646000000000001,"15min":40.646000000000001},"last":17.030000000000001}]},{"osd":6,"last update":"Sat Sep 30 02:11:21 2023","interfaces":[{"interface":"back","average":{"1min":13.474,"5min":14.634,"15min":15.917999999999999},"min":{"1min":9.8520000000000003,"5min":7.4829999999999997,"15min":7.4829999999999997},"max":{"1min":21.18,"5min":34.045000000000002,"15min":34.045000000000002},"last":13.928000000000001},{"interface":"front","average":{"1min":19.893999999999998,"5min":19.949000000000002,"15min":21.067},"min":{"1min":12.353999999999999,"5min":9.1600000000000001,"15min":9.1600000000000001},"max":{"1min":24.425000000000001,"5min":47.997999999999998,"15min":47.997999999999998},"last":12.933}]},{"osd":7,"last update":"Sat Sep 30 02:11:21 2023","interfaces":[{"interface":"back","average":{"1min":21.699999999999999,"5min":22.599,"15min":24.295000000000002},"min":{"1min":15.781000000000001,"5min":13.542999999999999,"15min":13.542999999999999},"max":{"1min":31.061,"5min":44.585000000000001,"15min":44.585000000000001},"last":21.584},{"interface":"front","average":{"1min":15.755000000000001,"5min":16.690999999999999,"15min":17.437000000000001},"min":{"1min":8.8230000000000004,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":26.234000000000002,"5min":38.295000000000002,"15min":38.295000000000002},"last":15.538}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T02:11:32.108 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-09-30T02:11:32.108 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T02:11:32.241 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.241+0000 7fa67364d700 10 monclient: get_monmap_and_config 2023-09-30T02:11:32.241 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.241+0000 7fa67364d700 10 monclient: build_initial_monmap 2023-09-30T02:11:32.241 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.241+0000 7fa67364d700 10 monclient: monmap: 2023-09-30T02:11:32.241 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:11:32.241 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:32.241 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:11:32.242090+0000 2023-09-30T02:11:32.241 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:11:32.242090+0000 2023-09-30T02:11:32.242 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:11:32.242 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:32.242 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:11:32.242 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:11:32.242 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:11:32.242 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:11:32.242 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:11:32.242 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:32.244 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.242+0000 7fa67364d700 1 Processor -- start 2023-09-30T02:11:32.244 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.242+0000 7fa67364d700 1 -- start start 2023-09-30T02:11:32.244 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.242+0000 7fa67364d700 10 monclient: init 2023-09-30T02:11:32.244 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.243+0000 7fa67364d700 10 monclient: _reopen_session rank -1 2023-09-30T02:11:32.244 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.243+0000 7fa67364d700 10 monclient: _add_conns ranks=[1,3,2,0,4] 2023-09-30T02:11:32.244 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.243+0000 7fa67364d700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa66c0a5b70 addr v1:172.21.15.110:6789/0 2023-09-30T02:11:32.244 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.243+0000 7fa67364d700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa66c142290 0x7fa66c0b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:32.245 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.243+0000 7fa67364d700 10 monclient(hunting): picked mon.noname-a con 0x7fa66c142290 addr v2:172.21.15.59:3300/0 2023-09-30T02:11:32.245 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.243+0000 7fa67364d700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa66c0a5f40 0x7fa66c13af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:32.245 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.243+0000 7fa67364d700 10 monclient(hunting): picked mon.noname-b con 0x7fa66c0a5f40 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:11:32.245 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.243+0000 7fa67364d700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa66c0a6380 con 0x7fa66c0a5b70 2023-09-30T02:11:32.245 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.243+0000 7fa67364d700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:32.245 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.243+0000 7fa67364d700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa66c0a75a0 con 0x7fa66c142290 2023-09-30T02:11:32.245 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.243+0000 7fa67364d700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:32.245 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.243+0000 7fa67364d700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa66c0af3f0 con 0x7fa66c0a5f40 2023-09-30T02:11:32.245 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.243+0000 7fa67364d700 10 monclient(hunting): _renew_subs 2023-09-30T02:11:32.245 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.243+0000 7fa67364d700 10 monclient(hunting): authenticate will time out at 2137.545394s 2023-09-30T02:11:32.245 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.244+0000 7fa671bea700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa66c0a5f40 0x7fa66c13af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:32.245 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.244+0000 7fa6713e9700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7fa66c0a5b70 0x7fa66c141b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.59:42868/0 (socket says 172.21.15.59:42868) 2023-09-30T02:11:32.245 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.244+0000 7fa6713e9700 1 -- 172.21.15.59:0/1584819959 learned_addr learned my addr 172.21.15.59:0/1584819959 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:11:32.246 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.244+0000 7fa671bea700 10 monclient(hunting): get_auth_request con 0x7fa66c0a5f40 auth_method 0 2023-09-30T02:11:32.246 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.244+0000 7fa671bea700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:32.246 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.244+0000 7fa671bea700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:32.246 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.244+0000 7fa671bea700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:32.246 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.245+0000 7fa671bea700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:32.246 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.245+0000 7fa671bea700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:32.246 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.245+0000 7fa671bea700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:32.247 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.246+0000 7fa670be8700 1 --2- 172.21.15.59:0/1584819959 >> v2:172.21.15.59:3300/0 conn(0x7fa66c142290 0x7fa66c0b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:32.248 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.247+0000 7fa663fff700 1 -- 172.21.15.59:0/1584819959 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 190850118 0 0) 0x7fa66c0a6380 con 0x7fa66c0a5b70 2023-09-30T02:11:32.248 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.247+0000 7fa663fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:32.248 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.247+0000 7fa663fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:32.248 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.247+0000 7fa663fff700 10 monclient(hunting): my global_id is 4870 2023-09-30T02:11:32.248 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.247+0000 7fa663fff700 1 -- 172.21.15.59:0/1584819959 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa658003570 con 0x7fa66c0a5b70 2023-09-30T02:11:32.248 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.247+0000 7fa670be8700 10 monclient(hunting): get_auth_request con 0x7fa66c142290 auth_method 0 2023-09-30T02:11:32.248 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.247+0000 7fa670be8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:32.248 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.247+0000 7fa670be8700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:32.248 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.247+0000 7fa670be8700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:32.248 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.247+0000 7fa671bea700 10 monclient(hunting): handle_auth_done global_id 4745 payload 274 2023-09-30T02:11:32.249 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.247+0000 7fa671bea700 1 -- 172.21.15.59:0/1584819959 >> v2:172.21.15.59:3300/0 conn(0x7fa66c142290 msgr2=0x7fa66c0b1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:11:32.249 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.247+0000 7fa671bea700 1 --2- 172.21.15.59:0/1584819959 >> v2:172.21.15.59:3300/0 conn(0x7fa66c142290 0x7fa66c0b1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:32.249 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.247+0000 7fa671bea700 1 -- 172.21.15.59:0/1584819959 >> v1:172.21.15.110:6789/0 conn(0x7fa66c0a5b70 legacy=0x7fa66c141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:32.249 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.247+0000 7fa671bea700 10 monclient: _finish_hunting 0 2023-09-30T02:11:32.249 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.247+0000 7fa671bea700 1 monclient: found mon.noname-b 2023-09-30T02:11:32.249 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.247+0000 7fa671bea700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:11:32.249 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.247+0000 7fa671bea700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.59:3301/0 2023-09-30T02:11:32.249 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.247+0000 7fa671bea700 1 -- 172.21.15.59:0/1584819959 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa66c13b470 con 0x7fa66c0a5f40 2023-09-30T02:11:32.250 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.250+0000 7fa671bea700 1 --2- 172.21.15.59:0/1584819959 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa66c0a5f40 0x7fa66c13af30 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fa664009e60 tx=0x7fa664009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b136d190c71889e2 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:32.255 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.255+0000 7fa663fff700 1 -- 172.21.15.59:0/1584819959 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa66400e070 con 0x7fa66c0a5f40 2023-09-30T02:11:32.255 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.255+0000 7fa663fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:32.255 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.255+0000 7fa663fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:11:32.255 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.255+0000 7fa663fff700 10 monclient: dump: 2023-09-30T02:11:32.255 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:32.255 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:32.255 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:32.255 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:32.256 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:32.256 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:32.256 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:32.256 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:32.256 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:32.256 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:32.256 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.255+0000 7fa663fff700 10 monclient: _finish_auth 0 2023-09-30T02:11:32.256 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.255+0000 7fa663fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:11:32.256 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.255+0000 7fa663fff700 1 -- 172.21.15.59:0/1584819959 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa66400bb60 con 0x7fa66c0a5f40 2023-09-30T02:11:32.256 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.255+0000 7fa67364d700 5 monclient: authenticate success, global_id 4745 2023-09-30T02:11:32.256 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.255+0000 7fa67364d700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:11:32.257 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.255+0000 7fa663fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:11:32.257 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.255+0000 7fa67364d700 10 monclient: get_monmap_and_config success 2023-09-30T02:11:32.257 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.255+0000 7fa67364d700 4 set_mon_vals no callback set 2023-09-30T02:11:32.257 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.255+0000 7fa663fff700 1 -- 172.21.15.59:0/1584819959 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa664012430 con 0x7fa66c0a5f40 2023-09-30T02:11:32.257 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.255+0000 7fa663fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:32.257 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.255+0000 7fa663fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:11:32.257 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.255+0000 7fa663fff700 10 monclient: dump: 2023-09-30T02:11:32.257 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:32.257 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:32.257 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:32.257 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:32.258 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:32.258 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:32.258 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:32.258 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:32.258 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:32.258 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:32.261 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.256+0000 7fa67364d700 10 monclient: shutdown 2023-09-30T02:11:32.261 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.256+0000 7fa67364d700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:11:32.261 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.256+0000 7fa67364d700 1 -- 172.21.15.59:0/1584819959 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa66c0a5f40 msgr2=0x7fa66c13af30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:32.261 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.256+0000 7fa67364d700 1 --2- 172.21.15.59:0/1584819959 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa66c0a5f40 0x7fa66c13af30 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fa664009e60 tx=0x7fa664009780 comp rx=0 tx=0).stop 2023-09-30T02:11:32.261 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.256+0000 7fa67364d700 1 -- 172.21.15.59:0/1584819959 shutdown_connections 2023-09-30T02:11:32.261 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.256+0000 7fa67364d700 1 --2- 172.21.15.59:0/1584819959 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa66c0a5f40 0x7fa66c13af30 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:32.261 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.256+0000 7fa67364d700 1 --2- 172.21.15.59:0/1584819959 >> v2:172.21.15.59:3300/0 conn(0x7fa66c142290 0x7fa66c0b1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:32.262 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.256+0000 7fa67364d700 1 -- 172.21.15.59:0/1584819959 >> 172.21.15.59:0/1584819959 conn(0x7fa66c0631f0 msgr2=0x7fa66c0635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:11:32.262 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.256+0000 7fa67364d700 1 -- 172.21.15.59:0/1584819959 shutdown_connections 2023-09-30T02:11:32.262 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.256+0000 7fa67364d700 1 -- 172.21.15.59:0/1584819959 wait complete. 2023-09-30T02:11:32.262 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.257+0000 7fa67364d700 10 monclient: build_initial_monmap 2023-09-30T02:11:32.262 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.257+0000 7fa67364d700 10 monclient: monmap: 2023-09-30T02:11:32.262 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:11:32.262 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:11:32.262 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:11:32.262 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:11:32.262 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:11:32.263 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:11:32.263 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:11:32.263 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:11:32.263 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:11:32.263 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:32.263 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.257+0000 7fa67364d700 1 librados: starting msgr at 2023-09-30T02:11:32.263 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.257+0000 7fa67364d700 1 librados: starting objecter 2023-09-30T02:11:32.263 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.258+0000 7fa67364d700 1 Processor -- start 2023-09-30T02:11:32.264 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.258+0000 7fa67364d700 1 -- start start 2023-09-30T02:11:32.264 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.258+0000 7fa67364d700 1 librados: setting wanted keys 2023-09-30T02:11:32.264 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.258+0000 7fa67364d700 1 librados: calling monclient init 2023-09-30T02:11:32.264 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.258+0000 7fa67364d700 10 monclient: init 2023-09-30T02:11:32.264 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.259+0000 7fa67364d700 10 monclient: _reopen_session rank -1 2023-09-30T02:11:32.264 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.259+0000 7fa67364d700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T02:11:32.264 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.259+0000 7fa67364d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa66c0a5b70 0x7fa66c0a7d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:32.264 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.259+0000 7fa67364d700 10 monclient(hunting): picked mon.noname-b con 0x7fa66c0a5b70 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:11:32.264 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.259+0000 7fa67364d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa66c0a5f40 0x7fa66c13e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:32.264 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.259+0000 7fa67364d700 10 monclient(hunting): picked mon.noname-a con 0x7fa66c0a5f40 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:11:32.264 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.259+0000 7fa67364d700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa66c142290 0x7fa66c137880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:32.264 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.259+0000 7fa67364d700 10 monclient(hunting): picked mon.noname-c con 0x7fa66c142290 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:11:32.265 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.259+0000 7fa67364d700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:32.265 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.259+0000 7fa67364d700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa66c0a85a0 con 0x7fa66c0a5f40 2023-09-30T02:11:32.265 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.259+0000 7fa67364d700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:32.265 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.259+0000 7fa67364d700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa66c0ae890 con 0x7fa66c0a5b70 2023-09-30T02:11:32.265 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.259+0000 7fa67364d700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:32.265 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.259+0000 7fa67364d700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa66c0a92c0 con 0x7fa66c142290 2023-09-30T02:11:32.265 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.259+0000 7fa67364d700 10 monclient(hunting): _renew_subs 2023-09-30T02:11:32.265 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.259+0000 7fa67364d700 10 monclient(hunting): authenticate will time out at 2137.561174s 2023-09-30T02:11:32.265 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.260+0000 7fa6713e9700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa66c0a5b70 0x7fa66c0a7d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:32.265 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.260+0000 7fa6713e9700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa66c0a5b70 0x7fa66c0a7d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:60480/0 (socket says 172.21.15.59:60480) 2023-09-30T02:11:32.265 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.260+0000 7fa6713e9700 1 -- 172.21.15.59:0/2593478254 learned_addr learned my addr 172.21.15.59:0/2593478254 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:11:32.265 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.260+0000 7fa6713e9700 10 monclient(hunting): get_auth_request con 0x7fa66c0a5b70 auth_method 0 2023-09-30T02:11:32.266 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.260+0000 7fa6713e9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:32.266 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.260+0000 7fa6713e9700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:32.266 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.260+0000 7fa6713e9700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:32.266 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.260+0000 7fa670be8700 1 --2- 172.21.15.59:0/2593478254 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa66c0a5f40 0x7fa66c13e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:32.266 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.261+0000 7fa6713e9700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:32.266 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.261+0000 7fa6713e9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:32.266 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.261+0000 7fa6713e9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:32.266 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.261+0000 7fa670be8700 10 monclient(hunting): get_auth_request con 0x7fa66c0a5f40 auth_method 0 2023-09-30T02:11:32.266 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.261+0000 7fa670be8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:32.266 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.261+0000 7fa670be8700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:32.266 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.261+0000 7fa670be8700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:32.266 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.262+0000 7fa670be8700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:32.267 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.262+0000 7fa670be8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:32.267 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.262+0000 7fa670be8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:32.269 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.264+0000 7fa671bea700 1 --2- 172.21.15.59:0/2593478254 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa66c142290 0x7fa66c137880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:32.269 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.264+0000 7fa671bea700 10 monclient(hunting): get_auth_request con 0x7fa66c142290 auth_method 0 2023-09-30T02:11:32.270 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.264+0000 7fa671bea700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:32.270 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.264+0000 7fa671bea700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:32.270 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.264+0000 7fa671bea700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:32.270 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.264+0000 7fa6713e9700 10 monclient(hunting): handle_auth_done global_id 4873 payload 824 2023-09-30T02:11:32.270 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.264+0000 7fa6713e9700 1 -- 172.21.15.59:0/2593478254 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa66c142290 msgr2=0x7fa66c137880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:32.270 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.264+0000 7fa6713e9700 1 --2- 172.21.15.59:0/2593478254 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa66c142290 0x7fa66c137880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:32.270 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.264+0000 7fa6713e9700 1 -- 172.21.15.59:0/2593478254 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa66c0a5f40 msgr2=0x7fa66c13e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:32.270 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.264+0000 7fa6713e9700 1 --2- 172.21.15.59:0/2593478254 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa66c0a5f40 0x7fa66c13e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:32.270 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.264+0000 7fa6713e9700 10 monclient: _finish_hunting 0 2023-09-30T02:11:32.270 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.264+0000 7fa6713e9700 1 monclient: found mon.noname-b 2023-09-30T02:11:32.270 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.265+0000 7fa6713e9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:11:32.271 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.265+0000 7fa6713e9700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:11:32.271 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.265+0000 7fa6713e9700 1 -- 172.21.15.59:0/2593478254 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa664009a20 con 0x7fa66c0a5b70 2023-09-30T02:11:32.271 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.268+0000 7fa6713e9700 1 --2- 172.21.15.59:0/2593478254 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa66c0a5b70 0x7fa66c0a7d60 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fa66800c7c0 tx=0x7fa668007e00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:32.271 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.268+0000 7fa6617fa700 10 client.?.objecter ms_handle_connect 0x7fa66c0a5b70 2023-09-30T02:11:32.271 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.268+0000 7fa6617fa700 10 client.?.objecter resend_mon_ops 2023-09-30T02:11:32.272 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.272+0000 7fa6617fa700 1 -- 172.21.15.59:0/2593478254 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa668005b30 con 0x7fa66c0a5b70 2023-09-30T02:11:32.272 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.272+0000 7fa6617fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:32.273 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.272+0000 7fa6617fa700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:11:32.273 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.272+0000 7fa6617fa700 10 monclient: dump: 2023-09-30T02:11:32.273 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:32.273 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:32.273 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:32.273 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:32.273 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:32.273 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:32.273 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:32.273 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:32.273 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:32.273 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:32.273 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.272+0000 7fa6617fa700 10 monclient: _finish_auth 0 2023-09-30T02:11:32.274 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.272+0000 7fa6617fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:11:32.274 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.272+0000 7fa6617fa700 1 -- 172.21.15.59:0/2593478254 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa668005c90 con 0x7fa66c0a5b70 2023-09-30T02:11:32.274 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.272+0000 7fa67364d700 5 monclient: authenticate success, global_id 4873 2023-09-30T02:11:32.274 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.272+0000 7fa67364d700 10 monclient: _renew_subs 2023-09-30T02:11:32.274 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.272+0000 7fa67364d700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:11:32.274 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.272+0000 7fa67364d700 1 -- 172.21.15.59:0/2593478254 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa66c0a82a0 con 0x7fa66c0a5b70 2023-09-30T02:11:32.274 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.272+0000 7fa6617fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:11:32.274 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.272+0000 7fa6617fa700 1 -- 172.21.15.59:0/2593478254 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa668007320 con 0x7fa66c0a5b70 2023-09-30T02:11:32.274 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.272+0000 7fa6617fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:32.274 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.272+0000 7fa6617fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:11:32.274 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.272+0000 7fa6617fa700 10 monclient: dump: 2023-09-30T02:11:32.275 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:32.275 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:32.275 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:32.275 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:32.275 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:32.275 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:32.275 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:32.275 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:32.275 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:32.275 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:32.275 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.272+0000 7fa6637fe700 4 set_mon_vals no callback set 2023-09-30T02:11:32.275 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.272+0000 7fa67364d700 10 client.4873.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:11:32.275 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.272+0000 7fa67364d700 10 monclient: _renew_subs 2023-09-30T02:11:32.276 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.272+0000 7fa67364d700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:11:32.276 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.272+0000 7fa67364d700 1 -- 172.21.15.59:0/2593478254 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa66c13f030 con 0x7fa66c0a5b70 2023-09-30T02:11:32.276 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.272+0000 7fa67364d700 1 librados: init done 2023-09-30T02:11:32.276 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.273+0000 7fa67364d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:11:32.276 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.273+0000 7fa67364d700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:11:32.276 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.273+0000 7fa67364d700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:11:32.276 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.273+0000 7fa67364d700 1 -- 172.21.15.59:0/2593478254 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa66c0a7330 con 0x7fa66c0a5b70 2023-09-30T02:11:32.340 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.341+0000 7fa6617fa700 1 -- 172.21.15.59:0/2593478254 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fa66801b3b0 con 0x7fa66c0a5b70 2023-09-30T02:11:32.341 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.341+0000 7fa6617fa700 1 --2- 172.21.15.59:0/2593478254 >> v2:172.21.15.110:6800/115870 conn(0x7fa650068890 0x7fa65006ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:32.341 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.341+0000 7fa6617fa700 1 -- 172.21.15.59:0/2593478254 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fa6680837e0 con 0x7fa66c0a5b70 2023-09-30T02:11:32.341 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.341+0000 7fa6617fa700 10 client.4873.objecter ms_dispatch 0x7fa66c001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:11:32.341 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.341+0000 7fa6617fa700 3 client.4873.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:11:32.342 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.341+0000 7fa6617fa700 3 client.4873.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:11:32.342 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.341+0000 7fa6617fa700 20 client.4873.objecter dump_active .. 0 homeless 2023-09-30T02:11:32.342 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.342+0000 7fa670be8700 1 --2- 172.21.15.59:0/2593478254 >> v2:172.21.15.110:6800/115870 conn(0x7fa650068890 0x7fa65006ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:32.342 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.342+0000 7fa670be8700 10 monclient: get_auth_request con 0x7fa650068890 auth_method 0 2023-09-30T02:11:32.342 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.342+0000 7fa670be8700 1 --2- 172.21.15.59:0/2593478254 >> v2:172.21.15.110:6800/115870 conn(0x7fa650068890 0x7fa65006ad40 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa66c1388a0 tx=0x7fa65c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:32.342 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.342+0000 7fa6617fa700 10 client.4873.objecter ms_handle_connect 0x7fa650068890 2023-09-30T02:11:32.542 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.542+0000 7fa6617fa700 1 -- 172.21.15.59:0/2593478254 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa668051080 con 0x7fa66c0a5b70 2023-09-30T02:11:32.542 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.542+0000 7fa6617fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:11:32.542 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.542+0000 7fa6617fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:11:32.742 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.743+0000 7fa67364d700 1 -- 172.21.15.59:0/2593478254 --> v2:172.21.15.110:6800/115870 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa66c064a00 con 0x7fa650068890 2023-09-30T02:11:32.746 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.746+0000 7fa6617fa700 1 -- 172.21.15.59:0/2593478254 <== mgr.4108 v2:172.21.15.110:6800/115870 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+70474 (secure 0 0 0) 0x7fa66c064a00 con 0x7fa650068890 2023-09-30T02:11:32.747 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:11:32.750 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.748+0000 7fa67364d700 10 librados: watch_flush enter 2023-09-30T02:11:32.750 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.748+0000 7fa67364d700 10 librados: watch_flush exit 2023-09-30T02:11:32.750 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.748+0000 7fa67364d700 20 client.4873.objecter shutdown clearing up homeless session... 2023-09-30T02:11:32.750 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.748+0000 7fa67364d700 10 client.4873.objecter successfully canceled tick 2023-09-30T02:11:32.751 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.748+0000 7fa67364d700 1 -- 172.21.15.59:0/2593478254 >> v2:172.21.15.110:6800/115870 conn(0x7fa650068890 msgr2=0x7fa65006ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:32.751 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.748+0000 7fa67364d700 1 --2- 172.21.15.59:0/2593478254 >> v2:172.21.15.110:6800/115870 conn(0x7fa650068890 0x7fa65006ad40 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa66c1388a0 tx=0x7fa65c008040 comp rx=0 tx=0).stop 2023-09-30T02:11:32.751 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.749+0000 7fa67364d700 10 monclient: shutdown 2023-09-30T02:11:32.752 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.749+0000 7fa67364d700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:11:32.752 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.749+0000 7fa67364d700 1 -- 172.21.15.59:0/2593478254 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa66c0a5b70 msgr2=0x7fa66c0a7d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:32.752 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.749+0000 7fa67364d700 1 --2- 172.21.15.59:0/2593478254 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa66c0a5b70 0x7fa66c0a7d60 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fa66800c7c0 tx=0x7fa668007e00 comp rx=0 tx=0).stop 2023-09-30T02:11:32.752 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.749+0000 7fa67364d700 1 -- 172.21.15.59:0/2593478254 shutdown_connections 2023-09-30T02:11:32.752 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.749+0000 7fa67364d700 1 --2- 172.21.15.59:0/2593478254 >> v2:172.21.15.110:6800/115870 conn(0x7fa650068890 0x7fa65006ad40 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:32.753 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.749+0000 7fa67364d700 1 --2- 172.21.15.59:0/2593478254 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa66c0a5b70 0x7fa66c0a7d60 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:32.753 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.749+0000 7fa67364d700 1 --2- 172.21.15.59:0/2593478254 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa66c142290 0x7fa66c137880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:32.753 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.749+0000 7fa67364d700 1 --2- 172.21.15.59:0/2593478254 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa66c0a5f40 0x7fa66c13e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:32.753 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.749+0000 7fa67364d700 1 -- 172.21.15.59:0/2593478254 >> 172.21.15.59:0/2593478254 conn(0x7fa66c0631f0 msgr2=0x7fa66c141b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:11:32.753 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.749+0000 7fa67364d700 1 -- 172.21.15.59:0/2593478254 shutdown_connections 2023-09-30T02:11:32.753 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.749+0000 7fa67364d700 1 -- 172.21.15.59:0/2593478254 wait complete. 2023-09-30T02:11:32.753 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.749+0000 7fa67364d700 1 librados: shutdown 2023-09-30T02:11:32.754 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2023-09-30T02:11:32.761 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":194,"stamp":"2023-09-30T02:11:31.647496+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":43464,"kb_used_data":2448,"kb_used_omap":0,"kb_used_meta":40576,"kb_avail":754931256,"statfs":{"total":773094113280,"available":773049606144,"internally_reserved":0,"allocated":2506752,"data_stored":1396000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":41549824},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002601"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T02:06:18.350818+0000","last_change":"2023-09-30T02:06:18.210390+0000","last_active":"2023-09-30T02:06:18.350818+0000","last_peered":"2023-09-30T02:06:18.350818+0000","last_clean":"2023-09-30T02:06:18.350818+0000","last_became_active":"2023-09-30T02:06:18.198153+0000","last_became_peered":"2023-09-30T02:06:18.198153+0000","last_unstale":"2023-09-30T02:06:18.350818+0000","last_undegraded":"2023-09-30T02:06:18.350818+0000","last_fullsized":"2023-09-30T02:06:18.350818+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_clean_scrub_stamp":"2023-09-30T02:06:15.800912+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:42:32.739379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T02:06:18.437839+0000","last_change":"2023-09-30T02:06:18.262708+0000","last_active":"2023-09-30T02:06:18.437839+0000","last_peered":"2023-09-30T02:06:18.437839+0000","last_clean":"2023-09-30T02:06:18.437839+0000","last_became_active":"2023-09-30T02:06:18.224464+0000","last_became_peered":"2023-09-30T02:06:18.224464+0000","last_unstale":"2023-09-30T02:06:18.437839+0000","last_undegraded":"2023-09-30T02:06:18.437839+0000","last_fullsized":"2023-09-30T02:06:18.437839+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_clean_scrub_stamp":"2023-09-30T02:06:15.800912+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:13:39.084627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T02:06:18.364779+0000","last_change":"2023-09-30T02:06:17.910244+0000","last_active":"2023-09-30T02:06:18.364779+0000","last_peered":"2023-09-30T02:06:18.364779+0000","last_clean":"2023-09-30T02:06:18.364779+0000","last_became_active":"2023-09-30T02:06:17.855689+0000","last_became_peered":"2023-09-30T02:06:17.855689+0000","last_unstale":"2023-09-30T02:06:18.364779+0000","last_undegraded":"2023-09-30T02:06:18.364779+0000","last_fullsized":"2023-09-30T02:06:18.364779+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_clean_scrub_stamp":"2023-09-30T02:06:15.800912+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:18:08.185701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T02:06:18.359472+0000","last_change":"2023-09-30T02:06:17.806354+0000","last_active":"2023-09-30T02:06:18.359472+0000","last_peered":"2023-09-30T02:06:18.359472+0000","last_clean":"2023-09-30T02:06:18.359472+0000","last_became_active":"2023-09-30T02:06:17.706615+0000","last_became_peered":"2023-09-30T02:06:17.706615+0000","last_unstale":"2023-09-30T02:06:18.359472+0000","last_undegraded":"2023-09-30T02:06:18.359472+0000","last_fullsized":"2023-09-30T02:06:18.359472+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_clean_scrub_stamp":"2023-09-30T02:06:15.800912+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:08:20.786129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"25'76","reported_seq":117,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-09-30T02:06:21.647351+0000","last_change":"2023-09-30T02:06:11.645599+0000","last_active":"2023-09-30T02:06:21.647351+0000","last_peered":"2023-09-30T02:06:21.647351+0000","last_clean":"2023-09-30T02:06:21.647351+0000","last_became_active":"2023-09-30T02:06:11.592631+0000","last_became_peered":"2023-09-30T02:06:11.592631+0000","last_unstale":"2023-09-30T02:06:21.647351+0000","last_undegraded":"2023-09-30T02:06:21.647351+0000","last_fullsized":"2023-09-30T02:06:21.647351+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:06:06.666529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:06:06.666529+0000","last_clean_scrub_stamp":"2023-09-30T02:06:06.666529+0000","objects_scrubbed":0,"log_size":76,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:16:35.859435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T02:06:18.343393+0000","last_change":"2023-09-30T02:06:18.102959+0000","last_active":"2023-09-30T02:06:18.343393+0000","last_peered":"2023-09-30T02:06:18.343393+0000","last_clean":"2023-09-30T02:06:18.343393+0000","last_became_active":"2023-09-30T02:06:18.091407+0000","last_became_peered":"2023-09-30T02:06:18.091407+0000","last_unstale":"2023-09-30T02:06:18.343393+0000","last_undegraded":"2023-09-30T02:06:18.343393+0000","last_fullsized":"2023-09-30T02:06:18.343393+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_clean_scrub_stamp":"2023-09-30T02:06:15.800912+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:34:28.390487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T02:06:18.250861+0000","last_change":"2023-09-30T02:06:17.465018+0000","last_active":"2023-09-30T02:06:18.250861+0000","last_peered":"2023-09-30T02:06:18.250861+0000","last_clean":"2023-09-30T02:06:18.250861+0000","last_became_active":"2023-09-30T02:06:17.420574+0000","last_became_peered":"2023-09-30T02:06:17.420574+0000","last_unstale":"2023-09-30T02:06:18.250861+0000","last_undegraded":"2023-09-30T02:06:18.250861+0000","last_fullsized":"2023-09-30T02:06:18.250861+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_clean_scrub_stamp":"2023-09-30T02:06:15.800912+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:46:28.414555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T02:06:18.354965+0000","last_change":"2023-09-30T02:06:17.896913+0000","last_active":"2023-09-30T02:06:18.354965+0000","last_peered":"2023-09-30T02:06:18.354965+0000","last_clean":"2023-09-30T02:06:18.354965+0000","last_became_active":"2023-09-30T02:06:17.890282+0000","last_became_peered":"2023-09-30T02:06:17.890282+0000","last_unstale":"2023-09-30T02:06:18.354965+0000","last_undegraded":"2023-09-30T02:06:18.354965+0000","last_fullsized":"2023-09-30T02:06:18.354965+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_clean_scrub_stamp":"2023-09-30T02:06:15.800912+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:35:55.671531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-09-30T02:06:17.900193+0000","last_change":"2023-09-30T02:06:17.877329+0000","last_active":"2023-09-30T02:06:17.900193+0000","last_peered":"2023-09-30T02:06:17.900193+0000","last_clean":"2023-09-30T02:06:17.900193+0000","last_became_active":"2023-09-30T02:06:17.836124+0000","last_became_peered":"2023-09-30T02:06:17.836124+0000","last_unstale":"2023-09-30T02:06:17.900193+0000","last_undegraded":"2023-09-30T02:06:17.900193+0000","last_fullsized":"2023-09-30T02:06:17.900193+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_clean_scrub_stamp":"2023-09-30T02:06:15.800912+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T14:04:06.678156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":19,"seq":81604378692,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5208,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366632,"statfs":{"total":96636764160,"available":96631431168,"internally_reserved":0,"allocated":212992,"data_stored":75040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":12.859999999999999,"5min":16.204000000000001,"15min":26.364000000000001},"min":{"1min":8.8520000000000003,"5min":7.1429999999999998,"15min":7.1429999999999998},"max":{"1min":17.263000000000002,"5min":414.25400000000002,"15min":414.25400000000002},"last":11.553000000000001},{"interface":"front","average":{"1min":12.984,"5min":17.393999999999998,"15min":29.042999999999999},"min":{"1min":7.1289999999999996,"5min":7.1289999999999996,"15min":7.1289999999999996},"max":{"1min":21.555,"5min":421.178,"15min":421.178},"last":10.301}]},{"osd":1,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":13.353,"5min":15.817,"15min":24.888999999999999},"min":{"1min":8.9610000000000003,"5min":6.8200000000000003,"15min":6.8200000000000003},"max":{"1min":16.489999999999998,"5min":326.60899999999998,"15min":326.60899999999998},"last":9.7940000000000005},{"interface":"front","average":{"1min":14.826000000000001,"5min":19.149999999999999,"15min":31.222999999999999},"min":{"1min":8.5099999999999998,"5min":8.5079999999999991,"15min":8.5079999999999991},"max":{"1min":24.559999999999999,"5min":418.35500000000002,"15min":418.35500000000002},"last":21.640000000000001}]},{"osd":2,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":16.120000000000001,"5min":21.582000000000001,"15min":33.603999999999999},"min":{"1min":9.2850000000000001,"5min":8.3960000000000008,"15min":8.3960000000000008},"max":{"1min":23.693999999999999,"5min":423.46699999999998,"15min":423.46699999999998},"last":22.791},{"interface":"front","average":{"1min":14.952,"5min":19.960999999999999,"15min":33.359999999999999},"min":{"1min":9.0749999999999993,"5min":8.4000000000000004,"15min":8.4000000000000004},"max":{"1min":23.123000000000001,"5min":438.697,"15min":438.697},"last":14.054}]},{"osd":4,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":15.225,"5min":19.898,"15min":32.472000000000001},"min":{"1min":9.5370000000000008,"5min":8.4030000000000005,"15min":8.4030000000000005},"max":{"1min":26.390000000000001,"5min":443.36399999999998,"15min":443.36399999999998},"last":12.300000000000001},{"interface":"front","average":{"1min":17.562999999999999,"5min":22.780999999999999,"15min":35.773000000000003},"min":{"1min":10.282,"5min":10.282,"15min":10.282},"max":{"1min":23.670000000000002,"5min":453.20699999999999,"15min":453.20699999999999},"last":20.591999999999999}]},{"osd":5,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":14.875,"5min":20.896999999999998,"15min":33.405000000000001},"min":{"1min":8.2430000000000003,"5min":8.141,"15min":8.141},"max":{"1min":22.359000000000002,"5min":443.94600000000003,"15min":443.94600000000003},"last":15.303000000000001},{"interface":"front","average":{"1min":19.251000000000001,"5min":22.654,"15min":32.534999999999997},"min":{"1min":11.301,"5min":10.746,"15min":10.746},"max":{"1min":25.305,"5min":335.11799999999999,"15min":335.11799999999999},"last":18.221}]},{"osd":6,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":18.507000000000001,"5min":22.931999999999999,"15min":36.216999999999999},"min":{"1min":12.955,"5min":9.4009999999999998,"15min":9.4009999999999998},"max":{"1min":28.02,"5min":432.34699999999998,"15min":432.34699999999998},"last":23.867999999999999},{"interface":"front","average":{"1min":21.754000000000001,"5min":24.712,"15min":34.631999999999998},"min":{"1min":18.292999999999999,"5min":13.018000000000001,"15min":13.018000000000001},"max":{"1min":27.356999999999999,"5min":334.178,"15min":334.178},"last":16.396000000000001}]},{"osd":7,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":17.149000000000001,"5min":23.225000000000001,"15min":35.642000000000003},"min":{"1min":10.933999999999999,"5min":10.928000000000001,"15min":10.928000000000001},"max":{"1min":23.986999999999998,"5min":461.42000000000002,"15min":461.42000000000002},"last":17.193999999999999},{"interface":"front","average":{"1min":22.254000000000001,"5min":26.744,"15min":39.024000000000001},"min":{"1min":17.257000000000001,"5min":15.686,"15min":15.686},"max":{"1min":25.774000000000001,"5min":431.27800000000002,"15min":431.27800000000002},"last":19.611999999999998}]}]},{"osd":1,"up_from":18,"seq":77309411395,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6044,"kb_used_data":600,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":94365796,"statfs":{"total":96636764160,"available":96630575104,"internally_reserved":0,"allocated":614400,"data_stored":472880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":13.101000000000001,"5min":13.603999999999999,"15min":13.84},"min":{"1min":8.7560000000000002,"5min":7.1639999999999997,"15min":7.1639999999999997},"max":{"1min":19.596,"5min":24.626999999999999,"15min":24.626999999999999},"last":14.039999999999999},{"interface":"front","average":{"1min":11.407,"5min":12.699,"15min":14.554},"min":{"1min":6.4859999999999998,"5min":6.4859999999999998,"15min":6.4859999999999998},"max":{"1min":17.785,"5min":38.213999999999999,"15min":38.213999999999999},"last":15.19}]},{"osd":2,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":12.917999999999999,"5min":13.146000000000001,"15min":13.359999999999999},"min":{"1min":8.4949999999999992,"5min":6.9509999999999996,"15min":6.9509999999999996},"max":{"1min":20.673999999999999,"5min":39.070999999999998,"15min":39.070999999999998},"last":14.342000000000001},{"interface":"front","average":{"1min":14.161,"5min":15.342000000000001,"15min":16.507999999999999},"min":{"1min":8.6050000000000004,"5min":8.6050000000000004,"15min":8.6050000000000004},"max":{"1min":20.436,"5min":28.981999999999999,"15min":28.981999999999999},"last":21.966000000000001}]},{"osd":3,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":18.742999999999999,"5min":28.812999999999999,"15min":56.773000000000003},"min":{"1min":13.654999999999999,"5min":9.2110000000000003,"15min":9.2110000000000003},"max":{"1min":23.431999999999999,"5min":789.26099999999997,"15min":789.26099999999997},"last":24.800999999999998},{"interface":"front","average":{"1min":16.210999999999999,"5min":26.545999999999999,"15min":55.965000000000003},"min":{"1min":7.8869999999999996,"5min":7.8869999999999996,"15min":7.8869999999999996},"max":{"1min":24.469000000000001,"5min":827.53899999999999,"15min":827.53899999999999},"last":11.007}]},{"osd":4,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":14.420999999999999,"5min":21.696999999999999,"15min":34.310000000000002},"min":{"1min":11.004,"5min":8.7029999999999994,"15min":8.7029999999999994},"max":{"1min":20.788,"5min":390.31700000000001,"15min":390.31700000000001},"last":18.489999999999998},{"interface":"front","average":{"1min":17.638000000000002,"5min":21.739000000000001,"15min":34.366999999999997},"min":{"1min":10.234,"5min":10.234,"15min":10.234},"max":{"1min":24.907,"5min":381.255,"15min":381.255},"last":11.484}]},{"osd":5,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":13.234999999999999,"5min":14.538,"15min":15.808999999999999},"min":{"1min":9.2680000000000007,"5min":7.6849999999999996,"15min":7.6849999999999996},"max":{"1min":21.724,"5min":31.677,"15min":31.677},"last":19.353999999999999},{"interface":"front","average":{"1min":20.417000000000002,"5min":20.757000000000001,"15min":21.004999999999999},"min":{"1min":9.9860000000000007,"5min":9.9860000000000007,"15min":9.9860000000000007},"max":{"1min":26.379000000000001,"5min":31.449999999999999,"15min":31.449999999999999},"last":13.403}]},{"osd":6,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":18.373999999999999,"5min":19.23,"15min":21.132000000000001},"min":{"1min":12.045,"5min":9.7560000000000002,"15min":9.7560000000000002},"max":{"1min":24.495999999999999,"5min":41.316000000000003,"15min":41.316000000000003},"last":20.221},{"interface":"front","average":{"1min":16.523,"5min":17.629999999999999,"15min":19.443999999999999},"min":{"1min":10.407,"5min":9.2699999999999996,"15min":9.2699999999999996},"max":{"1min":22.161999999999999,"5min":53.012,"15min":53.012},"last":21.260999999999999}]},{"osd":7,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":19.762,"5min":24.777999999999999,"15min":37.265999999999998},"min":{"1min":14.295999999999999,"5min":9.5310000000000006,"15min":9.5310000000000006},"max":{"1min":23.609000000000002,"5min":332.072,"15min":332.072},"last":17.216999999999999},{"interface":"front","average":{"1min":17.829000000000001,"5min":20.178999999999998,"15min":24.689},"min":{"1min":11.316000000000001,"5min":8.141,"15min":8.141},"max":{"1min":23.631,"5min":108.00700000000001,"15min":108.00700000000001},"last":23.73}]}]},{"osd":7,"up_from":14,"seq":60129542213,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5272,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":94366568,"statfs":{"total":96636764160,"available":96631365632,"internally_reserved":0,"allocated":212992,"data_stored":75040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:11:21 2023","interfaces":[{"interface":"back","average":{"1min":8.6750000000000007,"5min":8.9369999999999994,"15min":8.9199999999999999},"min":{"1min":6.2140000000000004,"5min":5.3120000000000003,"15min":5.3120000000000003},"max":{"1min":16.916,"5min":19.491,"15min":19.491},"last":7.3099999999999996},{"interface":"front","average":{"1min":10.153,"5min":9.9879999999999995,"15min":9.7560000000000002},"min":{"1min":7.3099999999999996,"5min":6.2850000000000001,"15min":6.2850000000000001},"max":{"1min":19.263000000000002,"5min":20.952000000000002,"15min":20.952000000000002},"last":10.08}]},{"osd":1,"last update":"Sat Sep 30 02:11:21 2023","interfaces":[{"interface":"back","average":{"1min":10.753,"5min":11.709,"15min":14.192},"min":{"1min":6.6630000000000003,"5min":6.6630000000000003,"15min":6.6630000000000003},"max":{"1min":18.591000000000001,"5min":89.454999999999998,"15min":89.454999999999998},"last":11.307},{"interface":"front","average":{"1min":11.548999999999999,"5min":12.834,"15min":14.573},"min":{"1min":8.0909999999999993,"5min":7.0700000000000003,"15min":7.0700000000000003},"max":{"1min":19.093,"5min":83.695999999999998,"15min":83.695999999999998},"last":20.039000000000001}]},{"osd":2,"last update":"Sat Sep 30 02:11:21 2023","interfaces":[{"interface":"back","average":{"1min":13.856,"5min":13.919,"15min":13.823},"min":{"1min":8.1240000000000006,"5min":7.4669999999999996,"15min":7.4669999999999996},"max":{"1min":20.259,"5min":21.370999999999999,"15min":21.370999999999999},"last":14.94},{"interface":"front","average":{"1min":14.368,"5min":14.385,"15min":15.127000000000001},"min":{"1min":10.176,"5min":7.8920000000000003,"15min":7.8920000000000003},"max":{"1min":19.542999999999999,"5min":21.712,"15min":21.712},"last":13.404999999999999}]},{"osd":3,"last update":"Sat Sep 30 02:11:21 2023","interfaces":[{"interface":"back","average":{"1min":15.417999999999999,"5min":15.926,"15min":15.646000000000001},"min":{"1min":11.319000000000001,"5min":7.1710000000000003,"15min":7.1710000000000003},"max":{"1min":21.238,"5min":24.276,"15min":24.276},"last":21.707999999999998},{"interface":"front","average":{"1min":16.335999999999999,"5min":16.087,"15min":16.295000000000002},"min":{"1min":12.477,"5min":7.9630000000000001,"15min":7.9630000000000001},"max":{"1min":22.021999999999998,"5min":24.716000000000001,"15min":24.716000000000001},"last":17.811}]},{"osd":4,"last update":"Sat Sep 30 02:11:12 2023","interfaces":[{"interface":"back","average":{"1min":16.693999999999999,"5min":17.527000000000001,"15min":19.007999999999999},"min":{"1min":9.4380000000000006,"5min":5.7039999999999997,"15min":5.7039999999999997},"max":{"1min":23.076000000000001,"5min":99.298000000000002,"15min":99.298000000000002},"last":15.726000000000001},{"interface":"front","average":{"1min":12.294,"5min":15.105,"15min":17.399000000000001},"min":{"1min":7.1509999999999998,"5min":7.1509999999999998,"15min":7.1509999999999998},"max":{"1min":17.899000000000001,"5min":86.938999999999993,"15min":86.938999999999993},"last":18.771000000000001}]},{"osd":5,"last update":"Sat Sep 30 02:11:12 2023","interfaces":[{"interface":"back","average":{"1min":15.673999999999999,"5min":17.282,"15min":19.91},"min":{"1min":8.6509999999999998,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":21.527000000000001,"5min":118.675,"15min":118.675},"last":16.376999999999999},{"interface":"front","average":{"1min":18.716999999999999,"5min":19.459,"15min":21.693999999999999},"min":{"1min":16.239999999999998,"5min":8.7330000000000005,"15min":8.7330000000000005},"max":{"1min":23.338999999999999,"5min":114.14400000000001,"15min":114.14400000000001},"last":22.946999999999999}]},{"osd":6,"last update":"Sat Sep 30 02:11:12 2023","interfaces":[{"interface":"back","average":{"1min":19.045000000000002,"5min":20.443999999999999,"15min":23.024000000000001},"min":{"1min":8.4359999999999999,"5min":8.4359999999999999,"15min":8.4359999999999999},"max":{"1min":24.417000000000002,"5min":111.048,"15min":111.048},"last":21.039000000000001},{"interface":"front","average":{"1min":18.745000000000001,"5min":19.427,"15min":22.183},"min":{"1min":12.542,"5min":9.8140000000000001,"15min":9.8140000000000001},"max":{"1min":23.036000000000001,"5min":115.241,"15min":115.241},"last":13.801}]}]},{"osd":0,"up_from":18,"seq":77309411396,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5208,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366632,"statfs":{"total":96636764160,"available":96631431168,"internally_reserved":0,"allocated":212992,"data_stored":75040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 30 02:11:19 2023","interfaces":[{"interface":"back","average":{"1min":10.754,"5min":16.196000000000002,"15min":29.155000000000001},"min":{"1min":7.2089999999999996,"5min":7.2089999999999996,"15min":7.2089999999999996},"max":{"1min":19.498999999999999,"5min":425.49599999999998,"15min":425.49599999999998},"last":14.003},{"interface":"front","average":{"1min":13.093,"5min":20.338000000000001,"15min":39.884},"min":{"1min":8.1210000000000004,"5min":7.9269999999999996,"15min":7.9269999999999996},"max":{"1min":18.349,"5min":667.81100000000004,"15min":667.81100000000004},"last":13.289999999999999}]},{"osd":2,"last update":"Sat Sep 30 02:11:19 2023","interfaces":[{"interface":"back","average":{"1min":12.025,"5min":14.532999999999999,"15min":18.928999999999998},"min":{"1min":8.2110000000000003,"5min":7.718,"15min":7.718},"max":{"1min":16.954000000000001,"5min":162.50700000000001,"15min":162.50700000000001},"last":11.449999999999999},{"interface":"front","average":{"1min":17.457000000000001,"5min":17.991,"15min":20.966999999999999},"min":{"1min":10.747,"5min":8.5079999999999991,"15min":8.5079999999999991},"max":{"1min":23.617000000000001,"5min":131.863,"15min":131.863},"last":14.81}]},{"osd":3,"last update":"Sat Sep 30 02:11:19 2023","interfaces":[{"interface":"back","average":{"1min":13.837,"5min":14.763,"15min":15.271000000000001},"min":{"1min":8.1020000000000003,"5min":7.4859999999999998,"15min":7.4859999999999998},"max":{"1min":22.571999999999999,"5min":24.763000000000002,"15min":24.763000000000002},"last":9.7050000000000001},{"interface":"front","average":{"1min":18.533999999999999,"5min":18.309999999999999,"15min":18.783000000000001},"min":{"1min":10.459,"5min":9.8569999999999993,"15min":9.8569999999999993},"max":{"1min":23.57,"5min":27.309000000000001,"15min":27.309000000000001},"last":18.625}]},{"osd":4,"last update":"Sat Sep 30 02:11:19 2023","interfaces":[{"interface":"back","average":{"1min":16.027999999999999,"5min":16.099,"15min":19.271000000000001},"min":{"1min":9.9179999999999993,"5min":9.0700000000000003,"15min":9.0700000000000003},"max":{"1min":22.122,"5min":102.456,"15min":102.456},"last":15.573},{"interface":"front","average":{"1min":13.881,"5min":17.271000000000001,"15min":20.933},"min":{"1min":8.7929999999999993,"5min":8.7929999999999993,"15min":8.7929999999999993},"max":{"1min":19.463999999999999,"5min":98.293999999999997,"15min":98.293999999999997},"last":17.890000000000001}]},{"osd":5,"last update":"Sat Sep 30 02:11:19 2023","interfaces":[{"interface":"back","average":{"1min":20.268999999999998,"5min":20.559999999999999,"15min":22.850000000000001},"min":{"1min":14.006,"5min":10.726000000000001,"15min":10.726000000000001},"max":{"1min":28.995000000000001,"5min":79.566999999999993,"15min":79.566999999999993},"last":17.129999999999999},{"interface":"front","average":{"1min":16.812999999999999,"5min":18.582999999999998,"15min":21.466999999999999},"min":{"1min":9.7599999999999998,"5min":9.7599999999999998,"15min":9.7599999999999998},"max":{"1min":24.213000000000001,"5min":109.80500000000001,"15min":109.80500000000001},"last":8.8889999999999993}]},{"osd":6,"last update":"Sat Sep 30 02:11:19 2023","interfaces":[{"interface":"back","average":{"1min":18.077999999999999,"5min":20.367000000000001,"15min":26.062999999999999},"min":{"1min":11.140000000000001,"5min":9.2829999999999995,"15min":9.2829999999999995},"max":{"1min":24.276,"5min":161.01499999999999,"15min":161.01499999999999},"last":12.545},{"interface":"front","average":{"1min":22.562000000000001,"5min":22.975000000000001,"15min":26.202999999999999},"min":{"1min":13.904,"5min":13.163,"15min":13.163},"max":{"1min":27.786000000000001,"5min":118.71599999999999,"15min":118.71599999999999},"last":22.385000000000002}]},{"osd":7,"last update":"Sat Sep 30 02:11:19 2023","interfaces":[{"interface":"back","average":{"1min":20.117999999999999,"5min":20.393000000000001,"15min":23.483000000000001},"min":{"1min":11.741,"5min":9.8810000000000002,"15min":9.8810000000000002},"max":{"1min":26.491,"5min":111.86,"15min":111.86},"last":16.395},{"interface":"front","average":{"1min":21.664000000000001,"5min":22.388999999999999,"15min":26.010999999999999},"min":{"1min":15.186999999999999,"5min":10.237,"15min":10.237},"max":{"1min":27.012,"5min":126.889,"15min":126.889},"last":21.527999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574916,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5208,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366632,"statfs":{"total":96636764160,"available":96631431168,"internally_reserved":0,"allocated":212992,"data_stored":75040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:11:20 2023","interfaces":[{"interface":"back","average":{"1min":9.6969999999999992,"5min":9.8569999999999993,"15min":9.7870000000000008},"min":{"1min":6.5830000000000002,"5min":5.774,"15min":5.774},"max":{"1min":16.530999999999999,"5min":19.709,"15min":19.709},"last":6.8959999999999999},{"interface":"front","average":{"1min":10.919,"5min":10.390000000000001,"15min":11.138999999999999},"min":{"1min":6.1239999999999997,"5min":5.0419999999999998,"15min":5.0419999999999998},"max":{"1min":19.603000000000002,"5min":23.134,"15min":23.134},"last":10.864000000000001}]},{"osd":1,"last update":"Sat Sep 30 02:11:20 2023","interfaces":[{"interface":"back","average":{"1min":11.829000000000001,"5min":12.981999999999999,"15min":14.561999999999999},"min":{"1min":7.351,"5min":6.3010000000000002,"15min":6.3010000000000002},"max":{"1min":17.946999999999999,"5min":27.242999999999999,"15min":27.242999999999999},"last":14.702},{"interface":"front","average":{"1min":11.532999999999999,"5min":11.537000000000001,"15min":12.59},"min":{"1min":7.9770000000000003,"5min":7.4989999999999997,"15min":7.4989999999999997},"max":{"1min":18.957000000000001,"5min":24.388999999999999,"15min":24.388999999999999},"last":18.056999999999999}]},{"osd":2,"last update":"Sat Sep 30 02:11:20 2023","interfaces":[{"interface":"back","average":{"1min":14.006,"5min":14.630000000000001,"15min":15.132},"min":{"1min":9.6690000000000005,"5min":7.7039999999999997,"15min":7.7039999999999997},"max":{"1min":19.521000000000001,"5min":24.510999999999999,"15min":24.510999999999999},"last":14.407999999999999},{"interface":"front","average":{"1min":12.595000000000001,"5min":13.534000000000001,"15min":14.086},"min":{"1min":7.9699999999999998,"5min":7.9699999999999998,"15min":7.9699999999999998},"max":{"1min":20.085000000000001,"5min":25.033999999999999,"15min":25.033999999999999},"last":13.747999999999999}]},{"osd":3,"last update":"Sat Sep 30 02:11:20 2023","interfaces":[{"interface":"back","average":{"1min":14.676,"5min":19.765999999999998,"15min":31.925000000000001},"min":{"1min":8.3770000000000007,"5min":8.2089999999999996,"15min":8.2089999999999996},"max":{"1min":20.157,"5min":540.52099999999996,"15min":540.52099999999996},"last":17.163},{"interface":"front","average":{"1min":15.218,"5min":21.224,"15min":35.917999999999999},"min":{"1min":9.4489999999999998,"5min":7.9649999999999999,"15min":7.9649999999999999},"max":{"1min":19.452999999999999,"5min":644.31600000000003,"15min":644.31600000000003},"last":21.873999999999999}]},{"osd":4,"last update":"Sat Sep 30 02:11:15 2023","interfaces":[{"interface":"back","average":{"1min":17.922000000000001,"5min":18.664999999999999,"15min":22.556999999999999},"min":{"1min":13.959,"5min":9.0969999999999995,"15min":9.0969999999999995},"max":{"1min":21.425999999999998,"5min":179.749,"15min":179.749},"last":23.646000000000001},{"interface":"front","average":{"1min":15.129,"5min":15.641999999999999,"15min":19.817},"min":{"1min":8.7550000000000008,"5min":7.6070000000000002,"15min":7.6070000000000002},"max":{"1min":19.663,"5min":174.52799999999999,"15min":174.52799999999999},"last":19.271000000000001}]},{"osd":5,"last update":"Sat Sep 30 02:11:15 2023","interfaces":[{"interface":"back","average":{"1min":16.963000000000001,"5min":16.859000000000002,"15min":16.969000000000001},"min":{"1min":9.6210000000000004,"5min":8.2379999999999995,"15min":8.2379999999999995},"max":{"1min":21.975999999999999,"5min":36.709000000000003,"15min":36.709000000000003},"last":19.513999999999999},{"interface":"front","average":{"1min":15.141999999999999,"5min":16.25,"15min":16.556000000000001},"min":{"1min":9.0809999999999995,"5min":5.0279999999999996,"15min":5.0279999999999996},"max":{"1min":21.393999999999998,"5min":27.364999999999998,"15min":27.364999999999998},"last":20.292000000000002}]},{"osd":7,"last update":"Sat Sep 30 02:11:09 2023","interfaces":[{"interface":"back","average":{"1min":19.030000000000001,"5min":19.597000000000001,"15min":22.396999999999998},"min":{"1min":13.644,"5min":10.367000000000001,"15min":10.367000000000001},"max":{"1min":22.771000000000001,"5min":105.801,"15min":105.801},"last":20.815000000000001},{"interface":"front","average":{"1min":17.553000000000001,"5min":19.193999999999999,"15min":20.960000000000001},"min":{"1min":7.3120000000000003,"5min":7.3120000000000003,"15min":7.3120000000000003},"max":{"1min":23.283000000000001,"5min":90.156000000000006,"15min":90.156000000000006},"last":15.582000000000001}]}]},{"osd":4,"up_from":15,"seq":64424509509,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5208,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366632,"statfs":{"total":96636764160,"available":96631431168,"internally_reserved":0,"allocated":212992,"data_stored":75040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:11:17 2023","interfaces":[{"interface":"back","average":{"1min":11.865,"5min":10.567,"15min":9.4399999999999995},"min":{"1min":6.3970000000000002,"5min":5.2699999999999996,"15min":5.2699999999999996},"max":{"1min":20.655000000000001,"5min":24.532,"15min":24.532},"last":11.471},{"interface":"front","average":{"1min":11.137,"5min":10.974,"15min":11.926},"min":{"1min":7.0190000000000001,"5min":5.4530000000000003,"15min":5.4530000000000003},"max":{"1min":21.841000000000001,"5min":25.225000000000001,"15min":25.225000000000001},"last":7.9080000000000004}]},{"osd":1,"last update":"Sat Sep 30 02:11:17 2023","interfaces":[{"interface":"back","average":{"1min":12.827,"5min":13.271000000000001,"15min":13.897},"min":{"1min":8.3960000000000008,"5min":6.2830000000000004,"15min":6.2830000000000004},"max":{"1min":17.864000000000001,"5min":28.917000000000002,"15min":28.917000000000002},"last":15.699999999999999},{"interface":"front","average":{"1min":10.146000000000001,"5min":11.148,"15min":11.789999999999999},"min":{"1min":7.4059999999999997,"5min":7.0250000000000004,"15min":7.0250000000000004},"max":{"1min":19.279,"5min":30.922999999999998,"15min":30.922999999999998},"last":9.0760000000000005}]},{"osd":2,"last update":"Sat Sep 30 02:11:17 2023","interfaces":[{"interface":"back","average":{"1min":14.563000000000001,"5min":13.641999999999999,"15min":14.407999999999999},"min":{"1min":7.8300000000000001,"5min":7.1609999999999996,"15min":7.1609999999999996},"max":{"1min":19.428000000000001,"5min":21.175000000000001,"15min":21.175000000000001},"last":13.936999999999999},{"interface":"front","average":{"1min":12.637,"5min":13.872,"15min":14.563000000000001},"min":{"1min":6.7060000000000004,"5min":6.7060000000000004,"15min":6.7060000000000004},"max":{"1min":18.747,"5min":27.533999999999999,"15min":27.533999999999999},"last":12.906000000000001}]},{"osd":3,"last update":"Sat Sep 30 02:11:23 2023","interfaces":[{"interface":"back","average":{"1min":16.146000000000001,"5min":16.042999999999999,"15min":15.887},"min":{"1min":9.4719999999999995,"5min":9.4719999999999995,"15min":9.4719999999999995},"max":{"1min":23.526,"5min":27.632999999999999,"15min":27.632999999999999},"last":21.678000000000001},{"interface":"front","average":{"1min":15.352,"5min":16.143999999999998,"15min":17.184999999999999},"min":{"1min":8.2599999999999998,"5min":7.9269999999999996,"15min":7.9269999999999996},"max":{"1min":23.113,"5min":30.495000000000001,"15min":30.495000000000001},"last":18.611000000000001}]},{"osd":5,"last update":"Sat Sep 30 02:11:11 2023","interfaces":[{"interface":"back","average":{"1min":16.295999999999999,"5min":18.059000000000001,"15min":20.965},"min":{"1min":12.298,"5min":7.9080000000000004,"15min":7.9080000000000004},"max":{"1min":21.044,"5min":99.784000000000006,"15min":99.784000000000006},"last":24.344000000000001},{"interface":"front","average":{"1min":13.371,"5min":15.058,"15min":17.594999999999999},"min":{"1min":8.3030000000000008,"5min":7.609,"15min":7.609},"max":{"1min":19.103000000000002,"5min":98.634,"15min":98.634},"last":16.399999999999999}]},{"osd":6,"last update":"Sat Sep 30 02:11:11 2023","interfaces":[{"interface":"back","average":{"1min":16.995000000000001,"5min":16.553999999999998,"15min":16.952999999999999},"min":{"1min":7.8949999999999996,"5min":7.8949999999999996,"15min":7.8949999999999996},"max":{"1min":22.498999999999999,"5min":27.277999999999999,"15min":27.277999999999999},"last":19.588999999999999},{"interface":"front","average":{"1min":15.564,"5min":16.422000000000001,"15min":16.088999999999999},"min":{"1min":9.109,"5min":6.5810000000000004,"15min":6.5810000000000004},"max":{"1min":20.640999999999998,"5min":28.640000000000001,"15min":28.640000000000001},"last":14.936}]},{"osd":7,"last update":"Sat Sep 30 02:11:11 2023","interfaces":[{"interface":"back","average":{"1min":17.225000000000001,"5min":18.161000000000001,"15min":18.513000000000002},"min":{"1min":7.79,"5min":7.79,"15min":7.79},"max":{"1min":23.047000000000001,"5min":31.143000000000001,"15min":31.143000000000001},"last":20.795999999999999},{"interface":"front","average":{"1min":17.949000000000002,"5min":18.007000000000001,"15min":18.579000000000001},"min":{"1min":10.542999999999999,"5min":8.6509999999999998,"15min":8.6509999999999998},"max":{"1min":22.966999999999999,"5min":30.529,"15min":30.529},"last":23.006}]}]},{"osd":5,"up_from":15,"seq":64424509509,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":6108,"kb_used_data":600,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":94365732,"statfs":{"total":96636764160,"available":96630509568,"internally_reserved":0,"allocated":614400,"data_stored":472880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:11:18 2023","interfaces":[{"interface":"back","average":{"1min":10.455,"5min":10.214,"15min":11.779999999999999},"min":{"1min":6.1040000000000001,"5min":5.7350000000000003,"15min":5.7350000000000003},"max":{"1min":19.498999999999999,"5min":22.452999999999999,"15min":22.452999999999999},"last":14.721},{"interface":"front","average":{"1min":10.673999999999999,"5min":10.722,"15min":11.548},"min":{"1min":6.6929999999999996,"5min":5.8879999999999999,"15min":5.8879999999999999},"max":{"1min":20.34,"5min":22.821999999999999,"15min":22.821999999999999},"last":8.5670000000000002}]},{"osd":1,"last update":"Sat Sep 30 02:11:18 2023","interfaces":[{"interface":"back","average":{"1min":14.335000000000001,"5min":13.829000000000001,"15min":15.194000000000001},"min":{"1min":8.9589999999999996,"5min":7.29,"15min":7.29},"max":{"1min":22.402999999999999,"5min":47.451999999999998,"15min":47.451999999999998},"last":8.3119999999999994},{"interface":"front","average":{"1min":11.304,"5min":11.545,"15min":12.558},"min":{"1min":7.774,"5min":6.6130000000000004,"15min":6.6130000000000004},"max":{"1min":15.734999999999999,"5min":48.841999999999999,"15min":48.841999999999999},"last":8.0190000000000001}]},{"osd":2,"last update":"Sat Sep 30 02:11:18 2023","interfaces":[{"interface":"back","average":{"1min":15.868,"5min":13.461,"15min":11.835000000000001},"min":{"1min":8.2829999999999995,"5min":6.8250000000000002,"15min":6.8250000000000002},"max":{"1min":27.501000000000001,"5min":27.501000000000001,"15min":27.501000000000001},"last":17.762},{"interface":"front","average":{"1min":12.898999999999999,"5min":14.695,"15min":15.513},"min":{"1min":7.8150000000000004,"5min":7.5369999999999999,"15min":7.5369999999999999},"max":{"1min":19.411000000000001,"5min":26.576000000000001,"15min":26.576000000000001},"last":15.177}]},{"osd":3,"last update":"Sat Sep 30 02:11:30 2023","interfaces":[{"interface":"back","average":{"1min":14.194000000000001,"5min":15.137,"15min":15.063000000000001},"min":{"1min":7.6980000000000004,"5min":7.2619999999999996,"15min":7.2619999999999996},"max":{"1min":21.891999999999999,"5min":26.895,"15min":26.895},"last":9.9550000000000001},{"interface":"front","average":{"1min":15.656000000000001,"5min":17.158999999999999,"15min":17.841999999999999},"min":{"1min":11.603,"5min":11.140000000000001,"15min":11.140000000000001},"max":{"1min":19.952000000000002,"5min":24.702999999999999,"15min":24.702999999999999},"last":13.757}]},{"osd":4,"last update":"Sat Sep 30 02:11:18 2023","interfaces":[{"interface":"back","average":{"1min":15.975,"5min":17.355,"15min":18.113},"min":{"1min":10.063000000000001,"5min":8.8160000000000007,"15min":8.8160000000000007},"max":{"1min":22.620999999999999,"5min":26.710000000000001,"15min":26.710000000000001},"last":18.731000000000002},{"interface":"front","average":{"1min":16.137,"5min":16.138999999999999,"15min":15.467000000000001},"min":{"1min":10.363,"5min":6.0270000000000001,"15min":6.0270000000000001},"max":{"1min":23.376000000000001,"5min":28.344999999999999,"15min":28.344999999999999},"last":13.065}]},{"osd":6,"last update":"Sat Sep 30 02:11:18 2023","interfaces":[{"interface":"back","average":{"1min":17.373000000000001,"5min":16.178999999999998,"15min":14.978999999999999},"min":{"1min":10.907999999999999,"5min":7.0679999999999996,"15min":7.0679999999999996},"max":{"1min":22.890999999999998,"5min":23.573,"15min":23.573},"last":16.010000000000002},{"interface":"front","average":{"1min":14.722,"5min":16.218,"15min":16.617000000000001},"min":{"1min":7.8550000000000004,"5min":7.5540000000000003,"15min":7.5540000000000003},"max":{"1min":20.41,"5min":27.763999999999999,"15min":27.763999999999999},"last":16.571000000000002}]},{"osd":7,"last update":"Sat Sep 30 02:11:18 2023","interfaces":[{"interface":"back","average":{"1min":17.943999999999999,"5min":18.265999999999998,"15min":19.606000000000002},"min":{"1min":12.4,"5min":10.584,"15min":10.584},"max":{"1min":22.768000000000001,"5min":28.951000000000001,"15min":28.951000000000001},"last":20.495999999999999},{"interface":"front","average":{"1min":17.033999999999999,"5min":16.844000000000001,"15min":16.053999999999998},"min":{"1min":11.109,"5min":7.0490000000000004,"15min":7.0490000000000004},"max":{"1min":21.635999999999999,"5min":25.151,"15min":25.151},"last":19.292999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411395,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5208,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366632,"statfs":{"total":96636764160,"available":96631431168,"internally_reserved":0,"allocated":212992,"data_stored":75040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:11:21 2023","interfaces":[{"interface":"back","average":{"1min":12.927,"5min":13.202999999999999,"15min":13.711},"min":{"1min":7.1929999999999996,"5min":6.665,"15min":6.665},"max":{"1min":20.434000000000001,"5min":69.003,"15min":69.003},"last":9.2360000000000007},{"interface":"front","average":{"1min":13.492000000000001,"5min":14.566000000000001,"15min":16.558},"min":{"1min":8.6880000000000006,"5min":8.1329999999999991,"15min":8.1329999999999991},"max":{"1min":25.254000000000001,"5min":91.331999999999994,"15min":91.331999999999994},"last":16.228000000000002}]},{"osd":1,"last update":"Sat Sep 30 02:11:21 2023","interfaces":[{"interface":"back","average":{"1min":14.648999999999999,"5min":16.484999999999999,"15min":20.882000000000001},"min":{"1min":9.4800000000000004,"5min":7.9630000000000001,"15min":7.9630000000000001},"max":{"1min":21.640999999999998,"5min":145.952,"15min":145.952},"last":13.198},{"interface":"front","average":{"1min":15.581,"5min":17.215,"15min":21.452000000000002},"min":{"1min":9.6679999999999993,"5min":8.7469999999999999,"15min":8.7469999999999999},"max":{"1min":25.678000000000001,"5min":155.72900000000001,"15min":155.72900000000001},"last":11.943}]},{"osd":3,"last update":"Sat Sep 30 02:11:21 2023","interfaces":[{"interface":"back","average":{"1min":16.968,"5min":16.724,"15min":17.524000000000001},"min":{"1min":10.601000000000001,"5min":10.124000000000001,"15min":10.124000000000001},"max":{"1min":32.027000000000001,"5min":39.378999999999998,"15min":39.378999999999998},"last":19.701000000000001},{"interface":"front","average":{"1min":17.798999999999999,"5min":18.437000000000001,"15min":18.405000000000001},"min":{"1min":8.9250000000000007,"5min":8.9250000000000007,"15min":8.9250000000000007},"max":{"1min":24.704999999999998,"5min":36.649000000000001,"15min":36.649000000000001},"last":10.896000000000001}]},{"osd":4,"last update":"Sat Sep 30 02:11:21 2023","interfaces":[{"interface":"back","average":{"1min":18.395,"5min":18.151,"15min":18.896999999999998},"min":{"1min":11.792999999999999,"5min":8.9359999999999999,"15min":8.9359999999999999},"max":{"1min":29.699000000000002,"5min":39.186999999999998,"15min":39.186999999999998},"last":20.402000000000001},{"interface":"front","average":{"1min":11.427,"5min":11.253,"15min":12.537000000000001},"min":{"1min":6.7329999999999997,"5min":6.7329999999999997,"15min":6.7329999999999997},"max":{"1min":23.036999999999999,"5min":31.32,"15min":31.32},"last":9.5489999999999995}]},{"osd":5,"last update":"Sat Sep 30 02:11:21 2023","interfaces":[{"interface":"back","average":{"1min":19.495000000000001,"5min":18.518999999999998,"15min":19.271000000000001},"min":{"1min":12.682,"5min":8.3499999999999996,"15min":8.3499999999999996},"max":{"1min":26.695,"5min":43.409999999999997,"15min":43.409999999999997},"last":18.501999999999999},{"interface":"front","average":{"1min":20.788,"5min":21.66,"15min":22.609999999999999},"min":{"1min":11.974,"5min":11.974,"15min":11.974},"max":{"1min":33.695999999999998,"5min":40.646000000000001,"15min":40.646000000000001},"last":17.030000000000001}]},{"osd":6,"last update":"Sat Sep 30 02:11:21 2023","interfaces":[{"interface":"back","average":{"1min":13.474,"5min":14.634,"15min":15.917999999999999},"min":{"1min":9.8520000000000003,"5min":7.4829999999999997,"15min":7.4829999999999997},"max":{"1min":21.18,"5min":34.045000000000002,"15min":34.045000000000002},"last":13.928000000000001},{"interface":"front","average":{"1min":19.893999999999998,"5min":19.949000000000002,"15min":21.067},"min":{"1min":12.353999999999999,"5min":9.1600000000000001,"15min":9.1600000000000001},"max":{"1min":24.425000000000001,"5min":47.997999999999998,"15min":47.997999999999998},"last":12.933}]},{"osd":7,"last update":"Sat Sep 30 02:11:21 2023","interfaces":[{"interface":"back","average":{"1min":21.699999999999999,"5min":22.599,"15min":24.295000000000002},"min":{"1min":15.781000000000001,"5min":13.542999999999999,"15min":13.542999999999999},"max":{"1min":31.061,"5min":44.585000000000001,"15min":44.585000000000001},"last":21.584},{"interface":"front","average":{"1min":15.755000000000001,"5min":16.690999999999999,"15min":17.437000000000001},"min":{"1min":8.8230000000000004,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":26.234000000000002,"5min":38.295000000000002,"15min":38.295000000000002},"last":15.538}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T02:11:32.764 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-30T02:11:32.896 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.896+0000 7f4c31ada700 10 monclient: get_monmap_and_config 2023-09-30T02:11:32.896 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.896+0000 7f4c31ada700 10 monclient: build_initial_monmap 2023-09-30T02:11:32.896 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.896+0000 7f4c31ada700 10 monclient: monmap: 2023-09-30T02:11:32.896 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:11:32.896 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:32.897 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:11:32.897161+0000 2023-09-30T02:11:32.897 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:11:32.897161+0000 2023-09-30T02:11:32.897 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:11:32.897 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:32.897 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:11:32.898 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:11:32.898 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:11:32.898 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:11:32.898 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:11:32.898 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:32.900 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.897+0000 7f4c31ada700 1 Processor -- start 2023-09-30T02:11:32.900 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.897+0000 7f4c31ada700 1 -- start start 2023-09-30T02:11:32.901 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.897+0000 7f4c31ada700 10 monclient: init 2023-09-30T02:11:32.901 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.898+0000 7f4c31ada700 10 monclient: _reopen_session rank -1 2023-09-30T02:11:32.901 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.898+0000 7f4c31ada700 10 monclient: _add_conns ranks=[0,1,3,4,2] 2023-09-30T02:11:32.901 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.898+0000 7f4c31ada700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4c2c0a5b70 addr v1:172.21.15.59:6789/0 2023-09-30T02:11:32.902 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.898+0000 7f4c31ada700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4c2c142290 addr v1:172.21.15.110:6789/0 2023-09-30T02:11:32.902 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.898+0000 7f4c31ada700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f4c2c0b1500 0x7f4c2c0a5f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:32.902 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.898+0000 7f4c31ada700 10 monclient(hunting): picked mon.noname-a con 0x7f4c2c0b1500 addr v2:172.21.15.59:3300/0 2023-09-30T02:11:32.902 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.898+0000 7f4c31ada700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c2c0b1920 con 0x7f4c2c0a5b70 2023-09-30T02:11:32.902 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.898+0000 7f4c31ada700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c2c0a75a0 con 0x7f4c2c142290 2023-09-30T02:11:32.903 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.898+0000 7f4c31ada700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:32.903 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.898+0000 7f4c31ada700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c2c0af3f0 con 0x7f4c2c0b1500 2023-09-30T02:11:32.903 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.898+0000 7f4c31ada700 10 monclient(hunting): _renew_subs 2023-09-30T02:11:32.903 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.898+0000 7f4c31ada700 10 monclient(hunting): authenticate will time out at 2138.200453s 2023-09-30T02:11:32.903 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.899+0000 7f4c2bfff700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f4c2c0b1500 0x7f4c2c0a5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:32.904 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.900+0000 7f4c2b7fe700 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f4c2c0a5b70 0x7f4c2c141b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:58758/0 (socket says 172.21.15.59:58758) 2023-09-30T02:11:32.904 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.900+0000 7f4c2b7fe700 1 -- 172.21.15.59:0/1453898320 learned_addr learned my addr 172.21.15.59:0/1453898320 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:11:32.904 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.901+0000 7f4c2bfff700 10 monclient(hunting): get_auth_request con 0x7f4c2c0b1500 auth_method 0 2023-09-30T02:11:32.904 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.901+0000 7f4c2bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:32.904 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.901+0000 7f4c2bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:32.905 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.901+0000 7f4c2bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:32.905 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.901+0000 7f4c2bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:32.906 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.901+0000 7f4c2bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:32.906 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.901+0000 7f4c2bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:32.906 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.904+0000 7f4c2bfff700 1 -- 172.21.15.59:0/1453898320 >> v2:172.21.15.59:3300/0 conn(0x7f4c2c0b1500 msgr2=0x7f4c2c0a5f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-09-30T02:11:32.906 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.904+0000 7f4c2bfff700 1 -- 172.21.15.59:0/1453898320 >> v2:172.21.15.59:3300/0 conn(0x7f4c2c0b1500 msgr2=0x7f4c2c0a5f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T02:11:32.906 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.904+0000 7f4c2bfff700 1 --2- 172.21.15.59:0/1453898320 >> v2:172.21.15.59:3300/0 conn(0x7f4c2c0b1500 0x7f4c2c0a5f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T02:11:32.907 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.904+0000 7f4c2bfff700 1 --2- 172.21.15.59:0/1453898320 >> v2:172.21.15.59:3300/0 conn(0x7f4c2c0b1500 0x7f4c2c0a5f40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-30T02:11:32.914 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.914+0000 7f4c2a7fc700 1 -- 172.21.15.59:0/1453898320 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2716009344 0 0) 0x7f4c1c001110 con 0x7f4c2c0a5b70 2023-09-30T02:11:32.914 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.914+0000 7f4c2a7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:32.914 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.914+0000 7f4c2a7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:32.914 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.914+0000 7f4c2a7fc700 10 monclient(hunting): my global_id is 4836 2023-09-30T02:11:32.915 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.914+0000 7f4c2a7fc700 1 -- 172.21.15.59:0/1453898320 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c14003570 con 0x7f4c2c0a5b70 2023-09-30T02:11:32.985 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.985+0000 7f4c2a7fc700 1 -- 172.21.15.59:0/1453898320 <== mon.0 v1:172.21.15.59:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1082957753 0 0) 0x7f4c14003570 con 0x7f4c2c0a5b70 2023-09-30T02:11:32.986 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.985+0000 7f4c2a7fc700 1 -- 172.21.15.59:0/1453898320 >> v2:172.21.15.59:3300/0 conn(0x7f4c2c0b1500 msgr2=0x7f4c2c0a5f40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T02:11:32.986 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.985+0000 7f4c2a7fc700 1 --2- 172.21.15.59:0/1453898320 >> v2:172.21.15.59:3300/0 conn(0x7f4c2c0b1500 0x7f4c2c0a5f40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:32.986 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.985+0000 7f4c2a7fc700 1 -- 172.21.15.59:0/1453898320 >> v1:172.21.15.110:6789/0 conn(0x7f4c2c142290 legacy=0x7f4c2c13af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:32.986 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.985+0000 7f4c2a7fc700 10 monclient: _finish_hunting 0 2023-09-30T02:11:32.986 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.985+0000 7f4c2a7fc700 1 monclient: found mon.noname-a-legacy 2023-09-30T02:11:32.986 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.985+0000 7f4c2a7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:11:32.987 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.985+0000 7f4c2a7fc700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.59:6789/0 2023-09-30T02:11:32.987 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.985+0000 7f4c2a7fc700 1 -- 172.21.15.59:0/1453898320 --> v1:172.21.15.59:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c2c13b640 con 0x7f4c2c0a5b70 2023-09-30T02:11:32.987 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.985+0000 7f4c2a7fc700 10 monclient: _finish_auth 0 2023-09-30T02:11:32.987 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.985+0000 7f4c2a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:11:32.987 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.985+0000 7f4c2a7fc700 1 -- 172.21.15.59:0/1453898320 <== mon.0 v1:172.21.15.59:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2174283827 0 0) 0x7f4c1c001600 con 0x7f4c2c0a5b70 2023-09-30T02:11:32.987 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.985+0000 7f4c31ada700 5 monclient: authenticate success, global_id 4836 2023-09-30T02:11:32.987 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.985+0000 7f4c31ada700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:11:32.987 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.985+0000 7f4c2a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:32.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.985+0000 7f4c2a7fc700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-09-30T02:11:32.988 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.985+0000 7f4c2a7fc700 10 monclient: dump: 2023-09-30T02:11:32.988 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:32.988 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:32.988 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:32.988 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:32.988 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:32.988 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:32.989 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:32.989 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:32.989 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:32.989 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:32.989 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.985+0000 7f4c2a7fc700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] but i'm connected to v1:172.21.15.59:6789/0, reconnecting 2023-09-30T02:11:32.989 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.985+0000 7f4c2a7fc700 10 monclient: _reopen_session rank -1 2023-09-30T02:11:32.989 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.985+0000 7f4c2a7fc700 1 -- 172.21.15.59:0/1453898320 >> v1:172.21.15.59:6789/0 conn(0x7f4c2c0a5b70 legacy=0x7f4c2c141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:32.990 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.985+0000 7f4c2a7fc700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T02:11:32.990 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.985+0000 7f4c2a7fc700 1 --2- 172.21.15.59:0/1453898320 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c14003ab0 0x7f4c14003ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:32.990 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.986+0000 7f4c2a7fc700 10 monclient(hunting): picked mon.a con 0x7f4c14003ab0 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:11:32.990 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.986+0000 7f4c2a7fc700 1 --2- 172.21.15.59:0/1453898320 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c14004810 0x7f4c14004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:32.990 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.986+0000 7f4c2a7fc700 10 monclient(hunting): picked mon.c con 0x7f4c14004810 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:11:32.990 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.986+0000 7f4c2a7fc700 1 --2- 172.21.15.59:0/1453898320 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4c14005210 0x7f4c1400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:32.990 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.986+0000 7f4c2a7fc700 10 monclient(hunting): picked mon.b con 0x7f4c14005210 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:11:32.990 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.986+0000 7f4c2a7fc700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:32.991 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.986+0000 7f4c2a7fc700 1 -- 172.21.15.59:0/1453898320 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c2c0af3f0 con 0x7f4c14003ab0 2023-09-30T02:11:32.991 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.986+0000 7f4c2a7fc700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:32.991 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.986+0000 7f4c2a7fc700 1 -- 172.21.15.59:0/1453898320 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c1400c1b0 con 0x7f4c14005210 2023-09-30T02:11:32.991 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.986+0000 7f4c2a7fc700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:32.991 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.986+0000 7f4c2a7fc700 1 -- 172.21.15.59:0/1453898320 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c1400c370 con 0x7f4c14004810 2023-09-30T02:11:32.991 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.986+0000 7f4c2a7fc700 10 monclient(hunting): _renew_subs 2023-09-30T02:11:32.991 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.986+0000 7f4c2a7fc700 10 monclient(hunting): _finish_auth 0 2023-09-30T02:11:32.991 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.986+0000 7f4c31ada700 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2023-09-30T02:11:32.992 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.990+0000 7f4c2b7fe700 1 --2- 172.21.15.59:0/1453898320 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c14003ab0 0x7f4c14003ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:32.992 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.990+0000 7f4c2b7fe700 10 monclient(hunting): get_auth_request con 0x7f4c14003ab0 auth_method 0 2023-09-30T02:11:32.993 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.990+0000 7f4c2b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:32.993 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.990+0000 7f4c2b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:32.993 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.990+0000 7f4c2b7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T02:11:32.993 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.991+0000 7f4c2b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:32.993 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.991+0000 7f4c2b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:32.993 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.991+0000 7f4c2b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T02:11:32.993 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.993+0000 7f4c2b7fe700 10 monclient(hunting): handle_auth_done global_id 4836 payload 293 2023-09-30T02:11:32.993 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.993+0000 7f4c2b7fe700 1 -- 172.21.15.59:0/1453898320 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c14004810 msgr2=0x7f4c14004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:11:32.993 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.993+0000 7f4c2b7fe700 1 --2- 172.21.15.59:0/1453898320 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c14004810 0x7f4c14004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:32.994 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.993+0000 7f4c2b7fe700 1 -- 172.21.15.59:0/1453898320 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4c14005210 msgr2=0x7f4c1400ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:11:32.994 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.993+0000 7f4c2b7fe700 1 --2- 172.21.15.59:0/1453898320 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4c14005210 0x7f4c1400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:32.994 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.993+0000 7f4c2b7fe700 10 monclient: _finish_hunting 0 2023-09-30T02:11:32.994 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.993+0000 7f4c2b7fe700 1 monclient: found mon.a 2023-09-30T02:11:32.994 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.993+0000 7f4c2bfff700 1 -- 172.21.15.59:0/1453898320 reap_dead start 2023-09-30T02:11:32.994 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.993+0000 7f4c2b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:11:32.994 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.993+0000 7f4c2b7fe700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:11:32.994 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.993+0000 7f4c2b7fe700 1 -- 172.21.15.59:0/1453898320 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c14003570 con 0x7f4c14003ab0 2023-09-30T02:11:32.994 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.993+0000 7f4c2b7fe700 10 monclient: _finish_auth 0 2023-09-30T02:11:32.995 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.993+0000 7f4c2b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:11:32.995 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.995+0000 7f4c2b7fe700 1 --2- 172.21.15.59:0/1453898320 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c14003ab0 0x7f4c14003ec0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4c1c00c3c0 tx=0x7f4c1c0079d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c7ed7f7452f89ee server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:32.999 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.999+0000 7f4c2a7fc700 1 -- 172.21.15.59:0/1453898320 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c1c00ccf0 con 0x7f4c14003ab0 2023-09-30T02:11:32.999 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.999+0000 7f4c2a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:32.999 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.999+0000 7f4c2a7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T02:11:32.999 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.999+0000 7f4c2a7fc700 10 monclient: dump: 2023-09-30T02:11:32.999 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:32.999 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:33.000 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:33.000 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:33.000 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:33.000 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:33.000 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:33.000 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:33.000 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:33.001 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:33.001 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.999+0000 7f4c2a7fc700 1 -- 172.21.15.59:0/1453898320 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c1c00ce70 con 0x7f4c14003ab0 2023-09-30T02:11:33.001 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.999+0000 7f4c31ada700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:11:33.001 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.999+0000 7f4c2a7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:11:33.001 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.999+0000 7f4c31ada700 10 monclient: get_monmap_and_config success 2023-09-30T02:11:33.001 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.999+0000 7f4c2a7fc700 1 -- 172.21.15.59:0/1453898320 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c1c005b50 con 0x7f4c14003ab0 2023-09-30T02:11:33.001 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.999+0000 7f4c31ada700 4 set_mon_vals no callback set 2023-09-30T02:11:33.001 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.999+0000 7f4c2a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:33.002 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.999+0000 7f4c2a7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T02:11:33.002 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:32.999+0000 7f4c2a7fc700 10 monclient: dump: 2023-09-30T02:11:33.002 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:33.002 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:33.002 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:33.002 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:33.002 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:33.002 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:33.003 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:33.003 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:33.003 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:33.003 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:33.005 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.000+0000 7f4c31ada700 10 monclient: shutdown 2023-09-30T02:11:33.005 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.000+0000 7f4c31ada700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:11:33.006 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.000+0000 7f4c31ada700 1 -- 172.21.15.59:0/1453898320 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c14003ab0 msgr2=0x7f4c14003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:33.006 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.000+0000 7f4c31ada700 1 --2- 172.21.15.59:0/1453898320 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c14003ab0 0x7f4c14003ec0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4c1c00c3c0 tx=0x7f4c1c0079d0 comp rx=0 tx=0).stop 2023-09-30T02:11:33.006 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.000+0000 7f4c31ada700 1 -- 172.21.15.59:0/1453898320 shutdown_connections 2023-09-30T02:11:33.006 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.000+0000 7f4c31ada700 1 --2- 172.21.15.59:0/1453898320 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c14003ab0 0x7f4c14003ec0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:33.006 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.000+0000 7f4c31ada700 1 -- 172.21.15.59:0/1453898320 >> 172.21.15.59:0/1453898320 conn(0x7f4c2c0631f0 msgr2=0x7f4c2c0635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:11:33.006 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.000+0000 7f4c31ada700 1 -- 172.21.15.59:0/1453898320 shutdown_connections 2023-09-30T02:11:33.006 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.000+0000 7f4c31ada700 1 -- 172.21.15.59:0/1453898320 wait complete. 2023-09-30T02:11:33.006 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.001+0000 7f4c31ada700 10 monclient: build_initial_monmap 2023-09-30T02:11:33.007 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.001+0000 7f4c31ada700 10 monclient: monmap: 2023-09-30T02:11:33.007 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:11:33.007 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:11:33.007 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:11:33.007 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:11:33.007 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:11:33.007 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:11:33.007 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:11:33.007 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:11:33.008 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:11:33.008 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:33.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.001+0000 7f4c31ada700 1 librados: starting msgr at 2023-09-30T02:11:33.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.001+0000 7f4c31ada700 1 librados: starting objecter 2023-09-30T02:11:33.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.002+0000 7f4c31ada700 1 Processor -- start 2023-09-30T02:11:33.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.002+0000 7f4c31ada700 1 -- start start 2023-09-30T02:11:33.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.002+0000 7f4c31ada700 1 librados: setting wanted keys 2023-09-30T02:11:33.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.002+0000 7f4c31ada700 1 librados: calling monclient init 2023-09-30T02:11:33.008 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.002+0000 7f4c31ada700 10 monclient: init 2023-09-30T02:11:33.009 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.002+0000 7f4c31ada700 10 monclient: _reopen_session rank -1 2023-09-30T02:11:33.009 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.002+0000 7f4c31ada700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T02:11:33.009 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.002+0000 7f4c31ada700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4c14003ab0 0x7f4c2c0a83b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:33.009 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.002+0000 7f4c31ada700 10 monclient(hunting): picked mon.noname-b con 0x7f4c14003ab0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:11:33.009 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.002+0000 7f4c31ada700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c2c14e260 0x7f4c2c0a8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:33.009 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.002+0000 7f4c31ada700 10 monclient(hunting): picked mon.noname-a con 0x7f4c2c14e260 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:11:33.009 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.002+0000 7f4c31ada700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c2c0a8f60 0x7f4c2c151580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:33.009 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.002+0000 7f4c31ada700 10 monclient(hunting): picked mon.noname-c con 0x7f4c2c0a8f60 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:11:33.009 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.002+0000 7f4c31ada700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:33.009 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.002+0000 7f4c31ada700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c2c14eeb0 con 0x7f4c2c14e260 2023-09-30T02:11:33.010 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.002+0000 7f4c31ada700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:33.010 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.002+0000 7f4c31ada700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c2c14ea20 con 0x7f4c14003ab0 2023-09-30T02:11:33.010 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.002+0000 7f4c31ada700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:33.010 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.002+0000 7f4c31ada700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c2c0aaa80 con 0x7f4c2c0a8f60 2023-09-30T02:11:33.010 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.002+0000 7f4c31ada700 10 monclient(hunting): _renew_subs 2023-09-30T02:11:33.010 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.002+0000 7f4c31ada700 10 monclient(hunting): authenticate will time out at 2138.304781s 2023-09-30T02:11:33.010 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.003+0000 7f4c2affd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c2c14e260 0x7f4c2c0a8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:33.010 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.003+0000 7f4c2affd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c2c14e260 0x7f4c2c0a8a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50484/0 (socket says 172.21.15.59:50484) 2023-09-30T02:11:33.010 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.003+0000 7f4c2affd700 1 -- 172.21.15.59:0/2171550243 learned_addr learned my addr 172.21.15.59:0/2171550243 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:11:33.011 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.003+0000 7f4c2affd700 10 monclient(hunting): get_auth_request con 0x7f4c2c14e260 auth_method 0 2023-09-30T02:11:33.011 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.003+0000 7f4c2affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:33.011 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.003+0000 7f4c2affd700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:33.011 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.003+0000 7f4c2affd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:33.011 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.004+0000 7f4c2affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:33.011 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.004+0000 7f4c2affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:33.011 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.004+0000 7f4c2affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:33.012 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.007+0000 7f4c2affd700 10 monclient(hunting): handle_auth_done global_id 4839 payload 824 2023-09-30T02:11:33.012 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.007+0000 7f4c2affd700 1 -- 172.21.15.59:0/2171550243 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c2c0a8f60 msgr2=0x7f4c2c151580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:33.012 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.007+0000 7f4c2affd700 1 --2- 172.21.15.59:0/2171550243 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c2c0a8f60 0x7f4c2c151580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:33.012 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.007+0000 7f4c2affd700 1 -- 172.21.15.59:0/2171550243 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4c14003ab0 msgr2=0x7f4c2c0a83b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:33.013 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.007+0000 7f4c2affd700 1 --2- 172.21.15.59:0/2171550243 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4c14003ab0 0x7f4c2c0a83b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:33.013 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.007+0000 7f4c2affd700 10 monclient: _finish_hunting 0 2023-09-30T02:11:33.013 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.007+0000 7f4c2affd700 1 monclient: found mon.noname-a 2023-09-30T02:11:33.013 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.007+0000 7f4c2affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:11:33.013 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.007+0000 7f4c2affd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.59:3300/0 2023-09-30T02:11:33.013 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.007+0000 7f4c2affd700 1 -- 172.21.15.59:0/2171550243 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c2c14e6b0 con 0x7f4c2c14e260 2023-09-30T02:11:33.013 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.011+0000 7f4c2affd700 1 --2- 172.21.15.59:0/2171550243 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c2c14e260 0x7f4c2c0a8a20 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f4c2000c7f0 tx=0x7f4c2000cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:33.013 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.011+0000 7f4c137fe700 10 client.?.objecter ms_handle_connect 0x7f4c2c14e260 2023-09-30T02:11:33.013 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.011+0000 7f4c137fe700 10 client.?.objecter resend_mon_ops 2023-09-30T02:11:33.016 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.016+0000 7f4c137fe700 1 -- 172.21.15.59:0/2171550243 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c20012630 con 0x7f4c2c14e260 2023-09-30T02:11:33.016 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.016+0000 7f4c137fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:33.016 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.016+0000 7f4c137fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T02:11:33.016 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.016+0000 7f4c137fe700 10 monclient: dump: 2023-09-30T02:11:33.017 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:33.017 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:33.017 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:33.017 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:33.017 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:33.017 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:33.017 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:33.017 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:33.017 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:33.018 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:33.018 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.016+0000 7f4c137fe700 10 monclient: _finish_auth 0 2023-09-30T02:11:33.018 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.016+0000 7f4c137fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:11:33.018 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.016+0000 7f4c137fe700 1 -- 172.21.15.59:0/2171550243 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c200127b0 con 0x7f4c2c14e260 2023-09-30T02:11:33.018 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.016+0000 7f4c31ada700 5 monclient: authenticate success, global_id 4839 2023-09-30T02:11:33.018 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.016+0000 7f4c137fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:11:33.018 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.016+0000 7f4c31ada700 10 monclient: _renew_subs 2023-09-30T02:11:33.018 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.016+0000 7f4c137fe700 1 -- 172.21.15.59:0/2171550243 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c20012b70 con 0x7f4c2c14e260 2023-09-30T02:11:33.018 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.016+0000 7f4c31ada700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:11:33.019 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.016+0000 7f4c31ada700 1 -- 172.21.15.59:0/2171550243 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c2c0a78f0 con 0x7f4c2c14e260 2023-09-30T02:11:33.019 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.016+0000 7f4c137fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:33.019 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.016+0000 7f4c137fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T02:11:33.019 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.016+0000 7f4c137fe700 10 monclient: dump: 2023-09-30T02:11:33.019 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:33.019 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:33.019 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:33.019 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:33.019 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:33.019 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:33.020 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:33.020 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:33.020 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:33.020 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:33.020 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.016+0000 7f4c29ffb700 4 set_mon_vals no callback set 2023-09-30T02:11:33.020 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.016+0000 7f4c31ada700 10 client.4839.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:11:33.020 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.016+0000 7f4c31ada700 10 monclient: _renew_subs 2023-09-30T02:11:33.020 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.016+0000 7f4c31ada700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:11:33.020 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.016+0000 7f4c31ada700 1 -- 172.21.15.59:0/2171550243 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c2c151ac0 con 0x7f4c2c14e260 2023-09-30T02:11:33.020 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.016+0000 7f4c31ada700 1 librados: init done 2023-09-30T02:11:33.021 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.017+0000 7f4c31ada700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:11:33.021 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.017+0000 7f4c31ada700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:11:33.021 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.017+0000 7f4c31ada700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:11:33.021 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.017+0000 7f4c31ada700 1 -- 172.21.15.59:0/2171550243 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c2c0710a0 con 0x7f4c2c14e260 2023-09-30T02:11:33.058 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.058+0000 7f4c137fe700 1 -- 172.21.15.59:0/2171550243 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f4c20025430 con 0x7f4c2c14e260 2023-09-30T02:11:33.059 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.059+0000 7f4c137fe700 1 --2- 172.21.15.59:0/2171550243 >> v2:172.21.15.110:6800/115870 conn(0x7f4c18068890 0x7f4c1806ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:33.059 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.059+0000 7f4c137fe700 1 -- 172.21.15.59:0/2171550243 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f4c20084620 con 0x7f4c2c14e260 2023-09-30T02:11:33.059 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.059+0000 7f4c137fe700 10 client.4839.objecter ms_dispatch 0x7f4c2c001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:11:33.059 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.059+0000 7f4c137fe700 3 client.4839.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:11:33.059 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.059+0000 7f4c137fe700 3 client.4839.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:11:33.059 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.059+0000 7f4c137fe700 20 client.4839.objecter dump_active .. 0 homeless 2023-09-30T02:11:33.059 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.059+0000 7f4c2b7fe700 1 --2- 172.21.15.59:0/2171550243 >> v2:172.21.15.110:6800/115870 conn(0x7f4c18068890 0x7f4c1806ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:33.059 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.059+0000 7f4c2b7fe700 10 monclient: get_auth_request con 0x7f4c18068890 auth_method 0 2023-09-30T02:11:33.059 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.060+0000 7f4c137fe700 10 client.4839.objecter ms_handle_connect 0x7f4c18068890 2023-09-30T02:11:33.060 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.060+0000 7f4c2b7fe700 1 --2- 172.21.15.59:0/2171550243 >> v2:172.21.15.110:6800/115870 conn(0x7f4c18068890 0x7f4c1806ad40 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f4c1c00c3c0 tx=0x7f4c1c015000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:33.340 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.340+0000 7f4c137fe700 1 -- 172.21.15.59:0/2171550243 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4c20051ec0 con 0x7f4c2c14e260 2023-09-30T02:11:33.340 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.340+0000 7f4c137fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:11:33.340 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.340+0000 7f4c137fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:11:33.559 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.560+0000 7f4c31ada700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-09-30T02:11:33.559 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.560+0000 7f4c31ada700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T02:11:33.559 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.560+0000 7f4c31ada700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.59:3300/0 2023-09-30T02:11:33.560 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.560+0000 7f4c31ada700 1 -- 172.21.15.59:0/2171550243 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4c2c0aabc0 con 0x7f4c2c14e260 2023-09-30T02:11:33.650 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.650+0000 7f4c137fe700 1 -- 172.21.15.59:0/2171550243 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+11935 (secure 0 0 0) 0x7f4c2c0aabc0 con 0x7f4c2c14e260 2023-09-30T02:11:33.650 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.650+0000 7f4c137fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T02:11:33.651 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.650+0000 7f4c137fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:11:33.651 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:11:33.651 INFO:teuthology.orchestra.run.smithi059.stdout:{"epoch":25,"fsid":"623052cb-4f82-4b2b-b72b-3ed5be614613","created":"2023-09-30T02:05:11.679291+0000","modified":"2023-09-30T02:06:19.209689+0000","last_up_change":"2023-09-30T02:06:10.164005+0000","last_in_change":"2023-09-30T02:05:18.270595+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":8,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-30T02:06:06.187725+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-09-30T02:06:15.695024+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"efdab44b-be57-4b81-8567-ff2deec1d3e3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6800","nonce":116664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6801","nonce":116664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6803","nonce":116664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6802","nonce":116664}]},"public_addr":"172.21.15.59:6800/116664","cluster_addr":"172.21.15.59:6801/116664","heartbeat_back_addr":"172.21.15.59:6803/116664","heartbeat_front_addr":"172.21.15.59:6802/116664","state":["exists","up"]},{"osd":1,"uuid":"eff9786d-8c39-4044-91d2-a31f33e3b16e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6808","nonce":116665}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6809","nonce":116665}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6811","nonce":116665}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6810","nonce":116665}]},"public_addr":"172.21.15.59:6808/116665","cluster_addr":"172.21.15.59:6809/116665","heartbeat_back_addr":"172.21.15.59:6811/116665","heartbeat_front_addr":"172.21.15.59:6810/116665","state":["exists","up"]},{"osd":2,"uuid":"b276ed4b-317b-4ddf-96ab-7ad71b8e3ff7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6804","nonce":116666}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6805","nonce":116666}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6807","nonce":116666}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6806","nonce":116666}]},"public_addr":"172.21.15.59:6804/116666","cluster_addr":"172.21.15.59:6805/116666","heartbeat_back_addr":"172.21.15.59:6807/116666","heartbeat_front_addr":"172.21.15.59:6806/116666","state":["exists","up"]},{"osd":3,"uuid":"48bfaf43-5846-4385-b9c6-6bc424cd5acd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6812","nonce":116667}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6813","nonce":116667}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6815","nonce":116667}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:6814","nonce":116667}]},"public_addr":"172.21.15.59:6812/116667","cluster_addr":"172.21.15.59:6813/116667","heartbeat_back_addr":"172.21.15.59:6815/116667","heartbeat_front_addr":"172.21.15.59:6814/116667","state":["exists","up"]},{"osd":4,"uuid":"76fa5443-1571-4e72-9d22-6ab19e59839a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6801","nonce":116560}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6802","nonce":116560}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6804","nonce":116560}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6803","nonce":116560}]},"public_addr":"172.21.15.110:6801/116560","cluster_addr":"172.21.15.110:6802/116560","heartbeat_back_addr":"172.21.15.110:6804/116560","heartbeat_front_addr":"172.21.15.110:6803/116560","state":["exists","up"]},{"osd":5,"uuid":"e9436478-4140-4fd6-ba2e-04fa6f6cf367","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6809","nonce":116562}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6810","nonce":116562}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6812","nonce":116562}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6811","nonce":116562}]},"public_addr":"172.21.15.110:6809/116562","cluster_addr":"172.21.15.110:6810/116562","heartbeat_back_addr":"172.21.15.110:6812/116562","heartbeat_front_addr":"172.21.15.110:6811/116562","state":["exists","up"]},{"osd":6,"uuid":"76676042-2074-4cf8-9ef9-5c87bf604944","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6813","nonce":116563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6814","nonce":116563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6816","nonce":116563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6815","nonce":116563}]},"public_addr":"172.21.15.110:6813/116563","cluster_addr":"172.21.15.110:6814/116563","heartbeat_back_addr":"172.21.15.110:6816/116563","heartbeat_front_addr":"172.21.15.110:6815/116563","state":["exists","up"]},{"osd":7,"uuid":"045479f4-f7f7-4b95-bc38-a2c52ac31b13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6805","nonce":116561}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6806","nonce":116561}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6808","nonce":116561}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.110:6807","nonce":116561}]},"public_addr":"172.21.15.110:6805/116561","cluster_addr":"172.21.15.110:6806/116561","heartbeat_back_addr":"172.21.15.110:6808/116561","heartbeat_front_addr":"172.21.15.110:6807/116561","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:06:05.267167+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:06:06.183162+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:06:05.646500+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:06:06.546668+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:06:02.158984+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:06:01.902487+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:06:01.310184+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T02:06:01.669641+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T02:11:33.653 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.653+0000 7f4c31ada700 10 librados: watch_flush enter 2023-09-30T02:11:33.653 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.653+0000 7f4c31ada700 10 librados: watch_flush exit 2023-09-30T02:11:33.653 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.653+0000 7f4c31ada700 20 client.4839.objecter shutdown clearing up homeless session... 2023-09-30T02:11:33.653 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.653+0000 7f4c31ada700 10 client.4839.objecter successfully canceled tick 2023-09-30T02:11:33.653 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.653+0000 7f4c31ada700 1 -- 172.21.15.59:0/2171550243 >> v2:172.21.15.110:6800/115870 conn(0x7f4c18068890 msgr2=0x7f4c1806ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:33.653 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.653+0000 7f4c31ada700 1 --2- 172.21.15.59:0/2171550243 >> v2:172.21.15.110:6800/115870 conn(0x7f4c18068890 0x7f4c1806ad40 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f4c1c00c3c0 tx=0x7f4c1c015000 comp rx=0 tx=0).stop 2023-09-30T02:11:33.653 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.653+0000 7f4c31ada700 10 monclient: shutdown 2023-09-30T02:11:33.653 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.653+0000 7f4c31ada700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:11:33.653 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.653+0000 7f4c31ada700 1 -- 172.21.15.59:0/2171550243 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c2c14e260 msgr2=0x7f4c2c0a8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:33.654 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.653+0000 7f4c31ada700 1 --2- 172.21.15.59:0/2171550243 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c2c14e260 0x7f4c2c0a8a20 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f4c2000c7f0 tx=0x7f4c2000cbb0 comp rx=0 tx=0).stop 2023-09-30T02:11:33.654 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.653+0000 7f4c31ada700 1 -- 172.21.15.59:0/2171550243 shutdown_connections 2023-09-30T02:11:33.654 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.653+0000 7f4c31ada700 1 --2- 172.21.15.59:0/2171550243 >> v2:172.21.15.110:6800/115870 conn(0x7f4c18068890 0x7f4c1806ad40 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:33.654 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.653+0000 7f4c31ada700 1 --2- 172.21.15.59:0/2171550243 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f4c14003ab0 0x7f4c2c0a83b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:33.654 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.653+0000 7f4c31ada700 1 --2- 172.21.15.59:0/2171550243 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f4c2c0a8f60 0x7f4c2c151580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:33.654 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.653+0000 7f4c31ada700 1 --2- 172.21.15.59:0/2171550243 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4c2c14e260 0x7f4c2c0a8a20 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:33.654 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.653+0000 7f4c31ada700 1 -- 172.21.15.59:0/2171550243 >> 172.21.15.59:0/2171550243 conn(0x7f4c2c0631f0 msgr2=0x7f4c2c0afdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:11:33.654 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.653+0000 7f4c31ada700 1 -- 172.21.15.59:0/2171550243 shutdown_connections 2023-09-30T02:11:33.654 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.653+0000 7f4c31ada700 1 -- 172.21.15.59:0/2171550243 wait complete. 2023-09-30T02:11:33.654 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:33.653+0000 7f4c31ada700 1 librados: shutdown 2023-09-30T02:11:34.588 INFO:tasks.ceph.mon.a.smithi059.stderr:==00:00:06:44.453 116256== Warning: unimplemented fcntl command: 1036 2023-09-30T02:11:34.641 INFO:tasks.ceph.mon.b.smithi110.stderr:==00:00:06:44.488 115869== Warning: unimplemented fcntl command: 1036 2023-09-30T02:11:34.642 INFO:tasks.ceph.mon.b.smithi110.stderr:==00:00:06:44.490 115869== Warning: unimplemented fcntl command: 1036 2023-09-30T02:11:34.648 INFO:tasks.ceph.mon.a.smithi059.stderr:==00:00:06:44.514 116256== Warning: unimplemented fcntl command: 1036 2023-09-30T02:11:34.666 INFO:tasks.ceph.mon.b.smithi110.stderr:==00:00:06:44.513 115869== Warning: unimplemented fcntl command: 1036 2023-09-30T02:11:34.667 INFO:tasks.ceph:Scrubbing osd.0 2023-09-30T02:11:34.668 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-09-30T02:11:34.679 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:06:44.544 116257== Warning: unimplemented fcntl command: 1036 2023-09-30T02:11:34.689 INFO:tasks.ceph.mon.a.smithi059.stderr:==00:00:06:44.554 116256== Warning: unimplemented fcntl command: 1036 2023-09-30T02:11:34.726 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:06:44.591 116257== Warning: unimplemented fcntl command: 1036 2023-09-30T02:11:34.755 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:06:44.620 116257== Warning: unimplemented fcntl command: 1036 2023-09-30T02:11:34.809 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.810+0000 7f82723b7700 10 monclient: get_monmap_and_config 2023-09-30T02:11:34.810 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.810+0000 7f82723b7700 10 monclient: build_initial_monmap 2023-09-30T02:11:34.810 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.810+0000 7f82723b7700 10 monclient: monmap: 2023-09-30T02:11:34.810 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:11:34.810 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:34.810 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:11:34.810830+0000 2023-09-30T02:11:34.810 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:11:34.810830+0000 2023-09-30T02:11:34.810 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:11:34.810 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:34.810 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:11:34.810 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:11:34.811 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:11:34.811 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:11:34.811 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:11:34.811 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:34.812 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.811+0000 7f82723b7700 1 Processor -- start 2023-09-30T02:11:34.812 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.811+0000 7f82723b7700 1 -- start start 2023-09-30T02:11:34.812 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.811+0000 7f82723b7700 10 monclient: init 2023-09-30T02:11:34.812 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.811+0000 7f82723b7700 10 monclient: _reopen_session rank -1 2023-09-30T02:11:34.812 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.811+0000 7f82723b7700 10 monclient: _add_conns ranks=[1,2,0,3,4] 2023-09-30T02:11:34.812 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.811+0000 7f82723b7700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f826c0a5730 addr v1:172.21.15.110:6789/0 2023-09-30T02:11:34.812 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.812+0000 7f82723b7700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f826c0a7fa0 0x7f826c0af5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:34.812 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.812+0000 7f82723b7700 10 monclient(hunting): picked mon.noname-b con 0x7f826c0a7fa0 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:11:34.813 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.812+0000 7f82723b7700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f826c0afae0 addr v1:172.21.15.59:6789/0 2023-09-30T02:11:34.813 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.812+0000 7f82723b7700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f826c0a7b10 con 0x7f826c0afae0 2023-09-30T02:11:34.813 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.812+0000 7f82723b7700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f826c0a5b00 con 0x7f826c0a5730 2023-09-30T02:11:34.813 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.812+0000 7f82723b7700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:34.813 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.812+0000 7f82723b7700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f826c0a7410 con 0x7f826c0a7fa0 2023-09-30T02:11:34.813 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.812+0000 7f82723b7700 10 monclient(hunting): _renew_subs 2023-09-30T02:11:34.813 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.812+0000 7f82723b7700 10 monclient(hunting): authenticate will time out at 2140.114095s 2023-09-30T02:11:34.815 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.816+0000 7f826b7fe700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f826c0a7fa0 0x7f826c0af5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:34.815 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.816+0000 7f826bfff700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f826c0a5730 0x7f826c1418e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.59:42890/0 (socket says 172.21.15.59:42890) 2023-09-30T02:11:34.815 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.816+0000 7f826bfff700 1 -- 172.21.15.59:0/1372345490 learned_addr learned my addr 172.21.15.59:0/1372345490 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:11:34.819 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.820+0000 7f826b7fe700 10 monclient(hunting): get_auth_request con 0x7f826c0a7fa0 auth_method 0 2023-09-30T02:11:34.819 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.820+0000 7f826b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:34.819 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.820+0000 7f826b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:34.819 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.820+0000 7f826b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:34.820 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.821+0000 7f826affd700 1 -- 172.21.15.59:0/1372345490 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3062607704 0 0) 0x7f826c0a5b00 con 0x7f826c0a5730 2023-09-30T02:11:34.820 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.821+0000 7f826affd700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:34.820 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.821+0000 7f826affd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:34.820 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.821+0000 7f826affd700 10 monclient(hunting): my global_id is 4876 2023-09-30T02:11:34.820 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.821+0000 7f826affd700 1 -- 172.21.15.59:0/1372345490 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8254003570 con 0x7f826c0a5730 2023-09-30T02:11:34.821 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.822+0000 7f826b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:34.821 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.822+0000 7f826b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:34.821 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.822+0000 7f826b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:34.826 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.825+0000 7f826affd700 1 -- 172.21.15.59:0/1372345490 <== mon.1 v1:172.21.15.110:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 777004544 0 0) 0x7f8254003570 con 0x7f826c0a5730 2023-09-30T02:11:34.826 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.825+0000 7f826affd700 1 -- 172.21.15.59:0/1372345490 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f826c0a7fa0 msgr2=0x7f826c0af5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:11:34.826 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.825+0000 7f826affd700 1 --2- 172.21.15.59:0/1372345490 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f826c0a7fa0 0x7f826c0af5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:34.827 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.825+0000 7f826affd700 1 -- 172.21.15.59:0/1372345490 >> v1:172.21.15.59:6789/0 conn(0x7f826c0afae0 legacy=0x7f826c0aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:34.827 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.825+0000 7f826affd700 10 monclient: _finish_hunting 0 2023-09-30T02:11:34.827 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.825+0000 7f826affd700 1 monclient: found mon.noname-c-legacy 2023-09-30T02:11:34.827 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.825+0000 7f826affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:11:34.827 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.825+0000 7f826affd700 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.110:6789/0 2023-09-30T02:11:34.827 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.825+0000 7f826affd700 1 -- 172.21.15.59:0/1372345490 --> v1:172.21.15.110:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f826c0b0670 con 0x7f826c0a5730 2023-09-30T02:11:34.827 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.825+0000 7f826affd700 10 monclient: _finish_auth 0 2023-09-30T02:11:34.827 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.825+0000 7f826affd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:11:34.827 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.825+0000 7f826affd700 1 -- 172.21.15.59:0/1372345490 <== mon.1 v1:172.21.15.110:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2174283827 0 0) 0x7f825c001470 con 0x7f826c0a5730 2023-09-30T02:11:34.827 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.825+0000 7f826affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:34.828 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.825+0000 7f826affd700 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2023-09-30T02:11:34.828 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.825+0000 7f826affd700 10 monclient: dump: 2023-09-30T02:11:34.828 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:34.828 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:34.828 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:34.828 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:34.828 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:34.828 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:34.829 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:34.829 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:34.829 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:34.829 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:34.829 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.825+0000 7f826affd700 1 monclient: mon.b has (v2) addrs [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] but i'm connected to v1:172.21.15.110:6789/0, reconnecting 2023-09-30T02:11:34.829 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.825+0000 7f826affd700 10 monclient: _reopen_session rank -1 2023-09-30T02:11:34.829 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.825+0000 7f826affd700 1 -- 172.21.15.59:0/1372345490 >> v1:172.21.15.110:6789/0 conn(0x7f826c0a5730 legacy=0x7f826c1418e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:34.829 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.825+0000 7f826affd700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T02:11:34.829 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.825+0000 7f826affd700 1 --2- 172.21.15.59:0/1372345490 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8254003a60 0x7f8254003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:34.829 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.825+0000 7f826affd700 10 monclient(hunting): picked mon.b con 0x7f8254003a60 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:11:34.829 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.825+0000 7f826affd700 1 --2- 172.21.15.59:0/1372345490 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8254004810 0x7f8254004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:34.829 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.825+0000 7f826affd700 10 monclient(hunting): picked mon.a con 0x7f8254004810 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:11:34.830 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.826+0000 7f826affd700 1 --2- 172.21.15.59:0/1372345490 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f826c0a7fa0 0x7f8254005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:34.830 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.826+0000 7f826affd700 10 monclient(hunting): picked mon.c con 0x7f826c0a7fa0 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:11:34.830 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.826+0000 7f826affd700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:34.830 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.826+0000 7f826affd700 1 -- 172.21.15.59:0/1372345490 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f826c0a7410 con 0x7f8254004810 2023-09-30T02:11:34.830 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.826+0000 7f826affd700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:34.830 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.826+0000 7f826affd700 1 -- 172.21.15.59:0/1372345490 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f825400bb90 con 0x7f8254003a60 2023-09-30T02:11:34.830 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.826+0000 7f826affd700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:34.830 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.826+0000 7f826affd700 1 -- 172.21.15.59:0/1372345490 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f825400bd50 con 0x7f826c0a7fa0 2023-09-30T02:11:34.830 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.826+0000 7f826affd700 10 monclient(hunting): _renew_subs 2023-09-30T02:11:34.830 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.826+0000 7f826affd700 10 monclient(hunting): _finish_auth 0 2023-09-30T02:11:34.831 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.828+0000 7f826bfff700 1 --2- 172.21.15.59:0/1372345490 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8254003a60 0x7f8254003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:34.831 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.829+0000 7f826b7fe700 1 --2- 172.21.15.59:0/1372345490 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8254004810 0x7f8254004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:34.832 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.829+0000 7f826bfff700 10 monclient(hunting): get_auth_request con 0x7f8254003a60 auth_method 0 2023-09-30T02:11:34.832 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.829+0000 7f826bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:34.832 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.829+0000 7f826bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:34.832 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.829+0000 7f826bfff700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T02:11:34.832 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.830+0000 7f826bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:34.832 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.830+0000 7f826bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:34.832 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.830+0000 7f826bfff700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T02:11:34.832 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.831+0000 7f826b7fe700 10 monclient(hunting): get_auth_request con 0x7f8254004810 auth_method 0 2023-09-30T02:11:34.832 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.831+0000 7f826b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:34.832 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.831+0000 7f826b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:34.832 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.831+0000 7f826b7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T02:11:34.833 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.833+0000 7f826bfff700 10 monclient(hunting): handle_auth_done global_id 4876 payload 293 2023-09-30T02:11:34.833 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.833+0000 7f826bfff700 1 -- 172.21.15.59:0/1372345490 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f826c0a7fa0 msgr2=0x7f8254005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:11:34.833 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.833+0000 7f826bfff700 1 --2- 172.21.15.59:0/1372345490 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f826c0a7fa0 0x7f8254005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:34.833 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.833+0000 7f826bfff700 1 -- 172.21.15.59:0/1372345490 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8254004810 msgr2=0x7f8254004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:11:34.834 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.833+0000 7f826bfff700 1 --2- 172.21.15.59:0/1372345490 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8254004810 0x7f8254004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:34.834 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.833+0000 7f826bfff700 10 monclient: _finish_hunting 0 2023-09-30T02:11:34.834 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.833+0000 7f826bfff700 1 monclient: found mon.b 2023-09-30T02:11:34.834 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.833+0000 7f826bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:11:34.834 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.833+0000 7f826bfff700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:11:34.834 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.833+0000 7f826bfff700 1 -- 172.21.15.59:0/1372345490 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8254003570 con 0x7f8254003a60 2023-09-30T02:11:34.834 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.833+0000 7f826bfff700 10 monclient: _finish_auth 0 2023-09-30T02:11:34.834 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.833+0000 7f826bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:11:34.834 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.833+0000 7f82723b7700 5 monclient: authenticate success, global_id 4876 2023-09-30T02:11:34.835 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.833+0000 7f82723b7700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:11:34.835 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.835+0000 7f826bfff700 1 --2- 172.21.15.59:0/1372345490 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8254003a60 0x7f8254003e70 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f825c00c2a0 tx=0x7f826c0a6240 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b6ad509efe5d7388 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:34.838 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.838+0000 7f826affd700 1 -- 172.21.15.59:0/1372345490 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f825c00cb70 con 0x7f8254003a60 2023-09-30T02:11:34.838 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.838+0000 7f826affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:34.838 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.838+0000 7f826affd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:11:34.838 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.838+0000 7f826affd700 10 monclient: dump: 2023-09-30T02:11:34.838 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:34.838 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:34.838 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:34.838 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:34.838 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:34.838 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:34.838 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:34.839 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:34.839 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:34.839 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:34.839 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.838+0000 7f826affd700 1 -- 172.21.15.59:0/1372345490 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f825c00ccd0 con 0x7f8254003a60 2023-09-30T02:11:34.839 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.838+0000 7f826affd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:11:34.839 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.838+0000 7f826affd700 1 -- 172.21.15.59:0/1372345490 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f825c005b70 con 0x7f8254003a60 2023-09-30T02:11:34.839 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.838+0000 7f82723b7700 10 monclient: get_monmap_and_config success 2023-09-30T02:11:34.839 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.838+0000 7f82723b7700 4 set_mon_vals no callback set 2023-09-30T02:11:34.839 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.838+0000 7f826affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:34.839 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.838+0000 7f826affd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:11:34.839 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.838+0000 7f826affd700 10 monclient: dump: 2023-09-30T02:11:34.839 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:34.840 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:34.840 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:34.840 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:34.840 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:34.840 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:34.840 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:34.840 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:34.840 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:34.840 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:34.843 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.839+0000 7f82723b7700 10 monclient: shutdown 2023-09-30T02:11:34.843 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.839+0000 7f82723b7700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:11:34.843 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.839+0000 7f82723b7700 1 -- 172.21.15.59:0/1372345490 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8254003a60 msgr2=0x7f8254003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:34.843 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.839+0000 7f82723b7700 1 --2- 172.21.15.59:0/1372345490 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8254003a60 0x7f8254003e70 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f825c00c2a0 tx=0x7f826c0a6240 comp rx=0 tx=0).stop 2023-09-30T02:11:34.843 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.839+0000 7f8270954700 1 -- 172.21.15.59:0/1372345490 reap_dead start 2023-09-30T02:11:34.843 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.839+0000 7f82723b7700 1 -- 172.21.15.59:0/1372345490 shutdown_connections 2023-09-30T02:11:34.843 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.839+0000 7f82723b7700 1 -- 172.21.15.59:0/1372345490 >> 172.21.15.59:0/1372345490 conn(0x7f826c0631f0 msgr2=0x7f826c0635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:11:34.843 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.839+0000 7f82723b7700 1 -- 172.21.15.59:0/1372345490 shutdown_connections 2023-09-30T02:11:34.844 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.840+0000 7f82723b7700 1 -- 172.21.15.59:0/1372345490 wait complete. 2023-09-30T02:11:34.844 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.840+0000 7f82723b7700 10 monclient: build_initial_monmap 2023-09-30T02:11:34.844 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.840+0000 7f82723b7700 10 monclient: monmap: 2023-09-30T02:11:34.844 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:11:34.844 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:11:34.844 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:11:34.844 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:11:34.844 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:11:34.844 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:11:34.844 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:11:34.844 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:11:34.844 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:11:34.845 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:34.845 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.841+0000 7f82723b7700 1 librados: starting msgr at 2023-09-30T02:11:34.845 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.841+0000 7f82723b7700 1 librados: starting objecter 2023-09-30T02:11:34.845 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.841+0000 7f82723b7700 1 Processor -- start 2023-09-30T02:11:34.845 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.841+0000 7f82723b7700 1 -- start start 2023-09-30T02:11:34.845 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.841+0000 7f82723b7700 1 librados: setting wanted keys 2023-09-30T02:11:34.845 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.841+0000 7f82723b7700 1 librados: calling monclient init 2023-09-30T02:11:34.845 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.841+0000 7f82723b7700 10 monclient: init 2023-09-30T02:11:34.845 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.842+0000 7f82723b7700 10 monclient: _reopen_session rank -1 2023-09-30T02:11:34.845 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.842+0000 7f82723b7700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T02:11:34.845 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.842+0000 7f82723b7700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f826c151a20 0x7f826c0ac1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:34.845 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.842+0000 7f82723b7700 10 monclient(hunting): picked mon.noname-a con 0x7f826c151a20 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:11:34.846 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.842+0000 7f82723b7700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f826c151df0 0x7f826c0ac6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:34.846 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.842+0000 7f82723b7700 10 monclient(hunting): picked mon.noname-b con 0x7f826c151df0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:11:34.846 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.842+0000 7f82723b7700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f826c138a40 0x7f826c138ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:34.846 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.842+0000 7f82723b7700 10 monclient(hunting): picked mon.noname-c con 0x7f826c138a40 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:11:34.846 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.842+0000 7f82723b7700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:34.846 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.842+0000 7f82723b7700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f826c152300 con 0x7f826c151a20 2023-09-30T02:11:34.846 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.842+0000 7f82723b7700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:34.846 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.842+0000 7f82723b7700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f826c13e790 con 0x7f826c151df0 2023-09-30T02:11:34.846 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.842+0000 7f82723b7700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:34.846 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.842+0000 7f82723b7700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f826c0acc30 con 0x7f826c138a40 2023-09-30T02:11:34.846 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.842+0000 7f82723b7700 10 monclient(hunting): _renew_subs 2023-09-30T02:11:34.846 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.842+0000 7f82723b7700 10 monclient(hunting): authenticate will time out at 2140.144477s 2023-09-30T02:11:34.847 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.843+0000 7f826b7fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f826c151df0 0x7f826c0ac6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:34.847 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.843+0000 7f826b7fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f826c151df0 0x7f826c0ac6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:60526/0 (socket says 172.21.15.59:60526) 2023-09-30T02:11:34.847 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.843+0000 7f826b7fe700 1 -- 172.21.15.59:0/1290033845 learned_addr learned my addr 172.21.15.59:0/1290033845 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:11:34.847 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.843+0000 7f826b7fe700 10 monclient(hunting): get_auth_request con 0x7f826c151df0 auth_method 0 2023-09-30T02:11:34.847 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.843+0000 7f826b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:34.847 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.843+0000 7f826b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:34.847 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.843+0000 7f826b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:34.847 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.844+0000 7f826b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:34.847 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.844+0000 7f826b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:34.847 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.844+0000 7f826b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:34.848 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.844+0000 7f826bfff700 1 --2- 172.21.15.59:0/1290033845 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f826c151a20 0x7f826c0ac1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:34.848 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.847+0000 7f826bfff700 10 monclient(hunting): get_auth_request con 0x7f826c151a20 auth_method 0 2023-09-30T02:11:34.849 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.847+0000 7f826bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:34.849 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.847+0000 7f826bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:34.849 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.847+0000 7f826bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:34.849 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.847+0000 7f826b7fe700 10 monclient(hunting): handle_auth_done global_id 4879 payload 824 2023-09-30T02:11:34.849 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.847+0000 7f826b7fe700 1 -- 172.21.15.59:0/1290033845 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f826c138a40 msgr2=0x7f826c138ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:34.849 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.847+0000 7f826b7fe700 1 --2- 172.21.15.59:0/1290033845 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f826c138a40 0x7f826c138ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:34.849 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.847+0000 7f826b7fe700 1 -- 172.21.15.59:0/1290033845 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f826c151a20 msgr2=0x7f826c0ac1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:34.849 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.847+0000 7f826b7fe700 1 --2- 172.21.15.59:0/1290033845 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f826c151a20 0x7f826c0ac1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:34.849 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.847+0000 7f826b7fe700 10 monclient: _finish_hunting 0 2023-09-30T02:11:34.849 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.847+0000 7f826b7fe700 1 monclient: found mon.noname-b 2023-09-30T02:11:34.849 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.847+0000 7f826b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:11:34.850 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.847+0000 7f826b7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:11:34.850 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.847+0000 7f826b7fe700 1 -- 172.21.15.59:0/1290033845 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f826c0acdf0 con 0x7f826c151df0 2023-09-30T02:11:34.850 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.850+0000 7f826b7fe700 1 --2- 172.21.15.59:0/1290033845 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f826c151df0 0x7f826c0ac6f0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f8260011990 tx=0x7f8260008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:34.850 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.850+0000 7f8253fff700 10 client.?.objecter ms_handle_connect 0x7f826c151df0 2023-09-30T02:11:34.850 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.850+0000 7f8253fff700 10 client.?.objecter resend_mon_ops 2023-09-30T02:11:34.851 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.852+0000 7f8253fff700 1 -- 172.21.15.59:0/1290033845 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8260011ed0 con 0x7f826c151df0 2023-09-30T02:11:34.852 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.852+0000 7f8253fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:34.852 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.852+0000 7f8253fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:11:34.852 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.852+0000 7f8253fff700 10 monclient: dump: 2023-09-30T02:11:34.852 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:34.852 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:34.852 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:34.852 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:34.852 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:34.852 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:34.852 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:34.853 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:34.853 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:34.853 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:34.853 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.852+0000 7f8253fff700 10 monclient: _finish_auth 0 2023-09-30T02:11:34.853 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.852+0000 7f8253fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:11:34.853 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.852+0000 7f82723b7700 5 monclient: authenticate success, global_id 4879 2023-09-30T02:11:34.853 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.852+0000 7f82723b7700 10 monclient: _renew_subs 2023-09-30T02:11:34.853 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.852+0000 7f82723b7700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:11:34.853 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.852+0000 7f82723b7700 1 -- 172.21.15.59:0/1290033845 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f826c139400 con 0x7f826c151df0 2023-09-30T02:11:34.853 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.852+0000 7f82723b7700 10 client.4879.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:11:34.853 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.852+0000 7f82723b7700 10 monclient: _renew_subs 2023-09-30T02:11:34.853 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.852+0000 7f82723b7700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:11:34.853 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.852+0000 7f82723b7700 1 -- 172.21.15.59:0/1290033845 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f826c13c150 con 0x7f826c151df0 2023-09-30T02:11:34.854 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.852+0000 7f82723b7700 1 librados: init done 2023-09-30T02:11:34.854 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.853+0000 7f82723b7700 10 client.4879.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:11:34.855 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.853+0000 7f82723b7700 15 client.4879.objecter _session_command_op_assign -1 1 2023-09-30T02:11:34.855 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.853+0000 7f82723b7700 20 client.4879.objecter _get_session osd=-1 returning homeless 2023-09-30T02:11:34.855 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.853+0000 7f82723b7700 10 client.4879.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:11:34.855 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.853+0000 7f82723b7700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:11:34.855 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.853+0000 7f82723b7700 1 -- 172.21.15.59:0/1290033845 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f826c064a00 con 0x7f826c151df0 2023-09-30T02:11:34.855 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.853+0000 7f8253fff700 1 -- 172.21.15.59:0/1290033845 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f826c064a00 con 0x7f826c151df0 2023-09-30T02:11:34.855 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.853+0000 7f8253fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:11:34.855 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.853+0000 7f8253fff700 1 -- 172.21.15.59:0/1290033845 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f826000faf0 con 0x7f826c151df0 2023-09-30T02:11:34.855 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.853+0000 7f8253fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:34.855 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.853+0000 7f8253fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:11:34.855 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.853+0000 7f8253fff700 10 monclient: dump: 2023-09-30T02:11:34.856 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:34.856 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:34.856 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:34.856 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:34.856 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:34.856 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:34.856 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:34.856 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:34.856 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:34.856 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:34.856 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.853+0000 7f826a7fc700 4 set_mon_vals no callback set 2023-09-30T02:11:34.874 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.875+0000 7f8253fff700 1 -- 172.21.15.59:0/1290033845 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f826000fc50 con 0x7f826c151df0 2023-09-30T02:11:34.875 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.875+0000 7f8253fff700 1 --2- 172.21.15.59:0/1290033845 >> v2:172.21.15.110:6800/115870 conn(0x7f8258068890 0x7f825806ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:34.875 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.875+0000 7f826bfff700 1 --2- 172.21.15.59:0/1290033845 >> v2:172.21.15.110:6800/115870 conn(0x7f8258068890 0x7f825806ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:34.875 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.876+0000 7f826bfff700 10 monclient: get_auth_request con 0x7f8258068890 auth_method 0 2023-09-30T02:11:34.875 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.876+0000 7f826bfff700 1 --2- 172.21.15.59:0/1290033845 >> v2:172.21.15.110:6800/115870 conn(0x7f8258068890 0x7f825806ad40 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f825c000c00 tx=0x7f825c015000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:34.875 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.876+0000 7f8253fff700 10 client.4879.objecter ms_handle_connect 0x7f8258068890 2023-09-30T02:11:34.879 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.879+0000 7f8253fff700 1 -- 172.21.15.59:0/1290033845 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f8260055de0 con 0x7f826c151df0 2023-09-30T02:11:34.879 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.879+0000 7f8253fff700 10 client.4879.objecter ms_dispatch 0x7f826c001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:11:34.879 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.879+0000 7f8253fff700 3 client.4879.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:11:34.879 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.879+0000 7f8253fff700 3 client.4879.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:11:34.879 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.879+0000 7f8253fff700 10 client.4879.objecter checking command 1 2023-09-30T02:11:34.879 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.879+0000 7f8253fff700 1 --2- 172.21.15.59:0/1290033845 >> v2:172.21.15.59:6800/116664 conn(0x7f825806e690 0x7f8258070aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:34.880 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.879+0000 7f8253fff700 20 client.4879.objecter _get_session s=0x7f8260011ed0 osd=0 3 2023-09-30T02:11:34.880 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.879+0000 7f8253fff700 20 client.4879.objecter put_session s=0x7f8260011ed0 osd=0 3 2023-09-30T02:11:34.880 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.879+0000 7f8253fff700 15 client.4879.objecter _session_command_op_remove -1 1 2023-09-30T02:11:34.880 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.879+0000 7f8253fff700 20 client.4879.objecter _get_session s=0x7f8260011ed0 osd=0 3 2023-09-30T02:11:34.880 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.879+0000 7f8253fff700 20 client.4879.objecter get_session s=0x7f8260011ed0 osd=0 3 2023-09-30T02:11:34.880 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.879+0000 7f8253fff700 15 client.4879.objecter _session_command_op_assign 0 1 2023-09-30T02:11:34.880 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.879+0000 7f8253fff700 20 client.4879.objecter put_session s=0x7f8260011ed0 osd=0 4 2023-09-30T02:11:34.880 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.879+0000 7f8253fff700 10 client.4879.objecter _send_command 1 2023-09-30T02:11:34.880 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.879+0000 7f8253fff700 1 -- 172.21.15.59:0/1290033845 --> v2:172.21.15.59:6800/116664 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8258071110 con 0x7f825806e690 2023-09-30T02:11:34.880 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.879+0000 7f8253fff700 20 client.4879.objecter dump_active .. 0 homeless 2023-09-30T02:11:34.880 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.879+0000 7f8253fff700 1 -- 172.21.15.59:0/1290033845 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8260051a20 con 0x7f826c151df0 2023-09-30T02:11:34.880 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.879+0000 7f8253fff700 10 monclient: handle_get_version_reply finishing 1 version 25 2023-09-30T02:11:34.881 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.880+0000 7f8270954700 1 --2- 172.21.15.59:0/1290033845 >> v2:172.21.15.59:6800/116664 conn(0x7f825806e690 0x7f8258070aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:34.881 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.880+0000 7f8270954700 10 monclient: get_auth_request con 0x7f825806e690 auth_method 0 2023-09-30T02:11:34.883 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.884+0000 7f8270954700 1 --2- 172.21.15.59:0/1290033845 >> v2:172.21.15.59:6800/116664 conn(0x7f825806e690 0x7f8258070aa0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:34.883 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.884+0000 7f8253fff700 10 client.4879.objecter ms_handle_connect 0x7f825806e690 2023-09-30T02:11:34.910 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.910+0000 7f8253fff700 1 -- 172.21.15.59:0/1290033845 <== osd.0 v2:172.21.15.59:6800/116664 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f8258071110 con 0x7f825806e690 2023-09-30T02:11:34.910 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.910+0000 7f8253fff700 10 client.4879.objecter ms_dispatch 0x7f826c001550 command_reply(tid 1: 0 ) v1 2023-09-30T02:11:34.910 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.910+0000 7f8253fff700 10 client.4879.objecter _finish_command 1 = system:0 2023-09-30T02:11:34.910 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.910+0000 7f8253fff700 20 client.4879.objecter put_session s=0x7f8260011ed0 osd=0 4 2023-09-30T02:11:34.910 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.910+0000 7f8253fff700 15 client.4879.objecter _session_command_op_remove 0 1 2023-09-30T02:11:34.928 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.929+0000 7f82723b7700 10 client.4879.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-09-30T02:11:34.929 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.929+0000 7f82723b7700 15 client.4879.objecter _session_command_op_assign -1 2 2023-09-30T02:11:34.929 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.929+0000 7f82723b7700 20 client.4879.objecter _get_session s=0x7f8260011ed0 osd=0 3 2023-09-30T02:11:34.929 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.929+0000 7f82723b7700 20 client.4879.objecter put_session s=0x7f8260011ed0 osd=0 3 2023-09-30T02:11:34.929 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.929+0000 7f82723b7700 20 client.4879.objecter _get_session s=0x7f8260011ed0 osd=0 3 2023-09-30T02:11:34.929 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.929+0000 7f82723b7700 15 client.4879.objecter _session_command_op_remove -1 2 2023-09-30T02:11:34.929 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.929+0000 7f82723b7700 20 client.4879.objecter get_session s=0x7f8260011ed0 osd=0 3 2023-09-30T02:11:34.929 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.929+0000 7f82723b7700 15 client.4879.objecter _session_command_op_assign 0 2 2023-09-30T02:11:34.929 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.929+0000 7f82723b7700 20 client.4879.objecter put_session s=0x7f8260011ed0 osd=0 4 2023-09-30T02:11:34.929 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.929+0000 7f82723b7700 10 client.4879.objecter _send_command 2 2023-09-30T02:11:34.929 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.929+0000 7f82723b7700 1 -- 172.21.15.59:0/1290033845 --> v2:172.21.15.59:6800/116664 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f826c0a83d0 con 0x7f825806e690 2023-09-30T02:11:34.964 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.964+0000 7f8253fff700 1 -- 172.21.15.59:0/1290033845 <== osd.0 v2:172.21.15.59:6800/116664 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f826c0a83d0 con 0x7f825806e690 2023-09-30T02:11:34.964 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.964+0000 7f8253fff700 10 client.4879.objecter ms_dispatch 0x7f826c001550 command_reply(tid 2: 0 ) v1 2023-09-30T02:11:34.964 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.964+0000 7f8253fff700 10 client.4879.objecter _finish_command 2 = system:0 2023-09-30T02:11:34.964 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.965+0000 7f8253fff700 20 client.4879.objecter put_session s=0x7f8260011ed0 osd=0 4 2023-09-30T02:11:34.965 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.965+0000 7f8253fff700 15 client.4879.objecter _session_command_op_remove 0 2 2023-09-30T02:11:34.965 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2023-09-30T02:11:34.965 INFO:teuthology.orchestra.run.smithi059.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T02:11:34.965 INFO:teuthology.orchestra.run.smithi059.stdout:} 2023-09-30T02:11:34.966 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.965+0000 7f82723b7700 10 librados: watch_flush enter 2023-09-30T02:11:34.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.965+0000 7f82723b7700 10 librados: watch_flush exit 2023-09-30T02:11:34.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.965+0000 7f82723b7700 10 client.4879.objecter close_session for osd.0 2023-09-30T02:11:34.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.965+0000 7f82723b7700 1 -- 172.21.15.59:0/1290033845 >> v2:172.21.15.59:6800/116664 conn(0x7f825806e690 msgr2=0x7f8258070aa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:34.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.965+0000 7f82723b7700 1 --2- 172.21.15.59:0/1290033845 >> v2:172.21.15.59:6800/116664 conn(0x7f825806e690 0x7f8258070aa0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:34.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.965+0000 7f82723b7700 20 client.4879.objecter put_session s=0x7f8260011ed0 osd=0 1 2023-09-30T02:11:34.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.965+0000 7f82723b7700 20 client.4879.objecter shutdown clearing up homeless session... 2023-09-30T02:11:34.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.965+0000 7f82723b7700 10 client.4879.objecter successfully canceled tick 2023-09-30T02:11:34.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.965+0000 7f82723b7700 1 -- 172.21.15.59:0/1290033845 >> v2:172.21.15.110:6800/115870 conn(0x7f8258068890 msgr2=0x7f825806ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:34.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.965+0000 7f82723b7700 1 --2- 172.21.15.59:0/1290033845 >> v2:172.21.15.110:6800/115870 conn(0x7f8258068890 0x7f825806ad40 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f825c000c00 tx=0x7f825c015000 comp rx=0 tx=0).stop 2023-09-30T02:11:34.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.965+0000 7f82723b7700 10 monclient: shutdown 2023-09-30T02:11:34.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.965+0000 7f82723b7700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:11:34.968 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.965+0000 7f82723b7700 1 -- 172.21.15.59:0/1290033845 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f826c151df0 msgr2=0x7f826c0ac6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:34.968 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.965+0000 7f82723b7700 1 --2- 172.21.15.59:0/1290033845 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f826c151df0 0x7f826c0ac6f0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f8260011990 tx=0x7f8260008040 comp rx=0 tx=0).stop 2023-09-30T02:11:34.968 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.965+0000 7f8270954700 1 -- 172.21.15.59:0/1290033845 reap_dead start 2023-09-30T02:11:34.968 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.965+0000 7f82723b7700 1 -- 172.21.15.59:0/1290033845 shutdown_connections 2023-09-30T02:11:34.968 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.965+0000 7f82723b7700 1 -- 172.21.15.59:0/1290033845 >> 172.21.15.59:0/1290033845 conn(0x7f826c0631f0 msgr2=0x7f826c0b3cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:11:34.968 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.965+0000 7f82723b7700 1 -- 172.21.15.59:0/1290033845 shutdown_connections 2023-09-30T02:11:34.968 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.965+0000 7f82723b7700 1 -- 172.21.15.59:0/1290033845 wait complete. 2023-09-30T02:11:34.968 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:34.965+0000 7f82723b7700 1 librados: shutdown 2023-09-30T02:11:34.978 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-09-30T02:11:35.155 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.155+0000 7fc0bd5fb700 10 monclient: get_monmap_and_config 2023-09-30T02:11:35.155 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.155+0000 7fc0bd5fb700 10 monclient: build_initial_monmap 2023-09-30T02:11:35.155 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.155+0000 7fc0bd5fb700 10 monclient: monmap: 2023-09-30T02:11:35.155 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:11:35.155 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:35.155 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:11:35.155609+0000 2023-09-30T02:11:35.155 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:11:35.155609+0000 2023-09-30T02:11:35.155 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:11:35.155 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:35.155 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:11:35.156 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:11:35.156 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:11:35.156 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:11:35.156 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:11:35.156 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:35.156 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.155+0000 7fc0bd5fb700 1 Processor -- start 2023-09-30T02:11:35.156 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.155+0000 7fc0bd5fb700 1 -- start start 2023-09-30T02:11:35.156 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.155+0000 7fc0bd5fb700 10 monclient: init 2023-09-30T02:11:35.158 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.156+0000 7fc0bd5fb700 10 monclient: _reopen_session rank -1 2023-09-30T02:11:35.158 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.156+0000 7fc0bd5fb700 10 monclient: _add_conns ranks=[1,0,4,2,3] 2023-09-30T02:11:35.158 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.156+0000 7fc0bd5fb700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc0b80a5b70 addr v1:172.21.15.110:6789/0 2023-09-30T02:11:35.158 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.156+0000 7fc0bd5fb700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc0b8142290 addr v1:172.21.15.59:6789/0 2023-09-30T02:11:35.158 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.156+0000 7fc0bd5fb700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fc0b80b1500 0x7fc0b80a5f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:35.158 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.156+0000 7fc0bd5fb700 10 monclient(hunting): picked mon.noname-c con 0x7fc0b80b1500 addr v2:172.21.15.110:3300/0 2023-09-30T02:11:35.158 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.156+0000 7fc0bd5fb700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0b80b1920 con 0x7fc0b8142290 2023-09-30T02:11:35.158 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.156+0000 7fc0bd5fb700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0b80a75a0 con 0x7fc0b80a5b70 2023-09-30T02:11:35.158 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.156+0000 7fc0bd5fb700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:35.159 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.156+0000 7fc0bd5fb700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0b80af3f0 con 0x7fc0b80b1500 2023-09-30T02:11:35.159 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.156+0000 7fc0bd5fb700 10 monclient(hunting): _renew_subs 2023-09-30T02:11:35.159 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.156+0000 7fc0bd5fb700 10 monclient(hunting): authenticate will time out at 2140.458658s 2023-09-30T02:11:35.159 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.157+0000 7fc0b77fe700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fc0b80b1500 0x7fc0b80a5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:35.159 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.157+0000 7fc0b67fc700 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fc0b8142290 0x7fc0b813af30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:58772/0 (socket says 172.21.15.59:58772) 2023-09-30T02:11:35.159 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.157+0000 7fc0b67fc700 1 -- 172.21.15.59:0/3964576972 learned_addr learned my addr 172.21.15.59:0/3964576972 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:11:35.159 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.157+0000 7fc0b77fe700 10 monclient(hunting): get_auth_request con 0x7fc0b80b1500 auth_method 0 2023-09-30T02:11:35.159 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.157+0000 7fc0b77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:35.159 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.157+0000 7fc0b77fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:35.159 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.157+0000 7fc0b77fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:35.159 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.158+0000 7fc0b77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:35.159 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.158+0000 7fc0b77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:35.160 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.159+0000 7fc0b77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:35.160 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.160+0000 7fc0b5ffb700 1 -- 172.21.15.59:0/3964576972 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2820025724 0 0) 0x7fc0a0001110 con 0x7fc0b8142290 2023-09-30T02:11:35.160 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.160+0000 7fc0b5ffb700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:35.160 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.160+0000 7fc0b5ffb700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:35.160 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.160+0000 7fc0b5ffb700 10 monclient(hunting): my global_id is 4848 2023-09-30T02:11:35.160 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.160+0000 7fc0b5ffb700 1 -- 172.21.15.59:0/3964576972 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0a4003570 con 0x7fc0b8142290 2023-09-30T02:11:35.161 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.162+0000 7fc0b77fe700 10 monclient(hunting): handle_auth_done global_id 4882 payload 274 2023-09-30T02:11:35.161 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.162+0000 7fc0b77fe700 1 -- 172.21.15.59:0/3964576972 >> v1:172.21.15.110:6789/0 conn(0x7fc0b80a5b70 legacy=0x7fc0b8141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:35.161 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.162+0000 7fc0b77fe700 1 -- 172.21.15.59:0/3964576972 >> v1:172.21.15.59:6789/0 conn(0x7fc0b8142290 legacy=0x7fc0b813af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:35.161 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.162+0000 7fc0b77fe700 10 monclient: _finish_hunting 0 2023-09-30T02:11:35.161 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.162+0000 7fc0b77fe700 1 monclient: found mon.noname-c 2023-09-30T02:11:35.162 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.162+0000 7fc0b77fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:11:35.162 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.162+0000 7fc0b77fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3300/0 2023-09-30T02:11:35.162 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.162+0000 7fc0b77fe700 1 -- 172.21.15.59:0/3964576972 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0b813b640 con 0x7fc0b80b1500 2023-09-30T02:11:35.165 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.166+0000 7fc0b77fe700 1 --2- 172.21.15.59:0/3964576972 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0b80b1500 0x7fc0b80a5f40 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fc0b0009e60 tx=0x7fc0b0009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a5b9d307feb6cf7f server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:35.171 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.170+0000 7fc0b5ffb700 1 -- 172.21.15.59:0/3964576972 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0b000e070 con 0x7fc0b80b1500 2023-09-30T02:11:35.171 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.170+0000 7fc0b5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:35.171 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.170+0000 7fc0b5ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:11:35.172 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.170+0000 7fc0b5ffb700 10 monclient: dump: 2023-09-30T02:11:35.172 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:35.172 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:35.172 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:35.172 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:35.172 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:35.172 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:35.172 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:35.172 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:35.172 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:35.172 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:35.173 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.170+0000 7fc0b5ffb700 10 monclient: _finish_auth 0 2023-09-30T02:11:35.173 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.170+0000 7fc0b5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:11:35.173 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.170+0000 7fc0b5ffb700 1 -- 172.21.15.59:0/3964576972 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0b000bb60 con 0x7fc0b80b1500 2023-09-30T02:11:35.173 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.170+0000 7fc0bd5fb700 5 monclient: authenticate success, global_id 4882 2023-09-30T02:11:35.173 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.170+0000 7fc0bd5fb700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:11:35.173 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.170+0000 7fc0b5ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:11:35.173 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.170+0000 7fc0bd5fb700 10 monclient: get_monmap_and_config success 2023-09-30T02:11:35.173 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.170+0000 7fc0bd5fb700 4 set_mon_vals no callback set 2023-09-30T02:11:35.173 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.170+0000 7fc0b5ffb700 1 -- 172.21.15.59:0/3964576972 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0b0012430 con 0x7fc0b80b1500 2023-09-30T02:11:35.173 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.170+0000 7fc0b5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:35.173 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.170+0000 7fc0b5ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:11:35.173 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.170+0000 7fc0b5ffb700 10 monclient: dump: 2023-09-30T02:11:35.174 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:35.174 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:35.174 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:35.174 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:35.174 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:35.174 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:35.174 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:35.174 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:35.174 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:35.174 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:35.174 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.170+0000 7fc0bd5fb700 10 monclient: shutdown 2023-09-30T02:11:35.174 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.170+0000 7fc0bd5fb700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:11:35.175 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.170+0000 7fc0bd5fb700 1 -- 172.21.15.59:0/3964576972 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0b80b1500 msgr2=0x7fc0b80a5f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:35.175 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.170+0000 7fc0bd5fb700 1 --2- 172.21.15.59:0/3964576972 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0b80b1500 0x7fc0b80a5f40 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fc0b0009e60 tx=0x7fc0b0009780 comp rx=0 tx=0).stop 2023-09-30T02:11:35.175 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.170+0000 7fc0bd5fb700 1 -- 172.21.15.59:0/3964576972 shutdown_connections 2023-09-30T02:11:35.175 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.170+0000 7fc0bd5fb700 1 --2- 172.21.15.59:0/3964576972 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0b80b1500 0x7fc0b80a5f40 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:35.175 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.170+0000 7fc0bd5fb700 1 -- 172.21.15.59:0/3964576972 >> 172.21.15.59:0/3964576972 conn(0x7fc0b80631f0 msgr2=0x7fc0b80635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:11:35.175 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.171+0000 7fc0bd5fb700 1 -- 172.21.15.59:0/3964576972 shutdown_connections 2023-09-30T02:11:35.175 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.171+0000 7fc0bd5fb700 1 -- 172.21.15.59:0/3964576972 wait complete. 2023-09-30T02:11:35.175 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.171+0000 7fc0bd5fb700 10 monclient: build_initial_monmap 2023-09-30T02:11:35.175 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.171+0000 7fc0bd5fb700 10 monclient: monmap: 2023-09-30T02:11:35.175 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:11:35.175 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:11:35.175 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:11:35.176 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:11:35.176 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:11:35.176 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:11:35.176 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:11:35.176 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:11:35.176 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:11:35.176 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:35.176 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.172+0000 7fc0bd5fb700 1 librados: starting msgr at 2023-09-30T02:11:35.176 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.172+0000 7fc0bd5fb700 1 librados: starting objecter 2023-09-30T02:11:35.176 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.172+0000 7fc0bd5fb700 1 Processor -- start 2023-09-30T02:11:35.176 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.172+0000 7fc0bd5fb700 1 -- start start 2023-09-30T02:11:35.177 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.172+0000 7fc0bd5fb700 1 librados: setting wanted keys 2023-09-30T02:11:35.177 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.172+0000 7fc0bd5fb700 1 librados: calling monclient init 2023-09-30T02:11:35.177 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.172+0000 7fc0bd5fb700 10 monclient: init 2023-09-30T02:11:35.179 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.172+0000 7fc0bd5fb700 10 monclient: _reopen_session rank -1 2023-09-30T02:11:35.179 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.172+0000 7fc0bd5fb700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T02:11:35.179 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.172+0000 7fc0bd5fb700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc0b80a5b70 0x7fc0b8144ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:35.179 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.172+0000 7fc0bd5fb700 10 monclient(hunting): picked mon.noname-c con 0x7fc0b80a5b70 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:11:35.180 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.172+0000 7fc0bd5fb700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0b80b1500 0x7fc0b813c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:35.180 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.172+0000 7fc0bd5fb700 10 monclient(hunting): picked mon.noname-a con 0x7fc0b80b1500 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:11:35.180 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.172+0000 7fc0bd5fb700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0b8142290 0x7fc0b813cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:35.180 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.172+0000 7fc0bd5fb700 10 monclient(hunting): picked mon.noname-b con 0x7fc0b8142290 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:11:35.180 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.172+0000 7fc0bd5fb700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:35.180 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.172+0000 7fc0bd5fb700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b813d070 con 0x7fc0b80b1500 2023-09-30T02:11:35.180 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.172+0000 7fc0bd5fb700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:35.180 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.172+0000 7fc0bd5fb700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b813c160 con 0x7fc0b8142290 2023-09-30T02:11:35.180 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.172+0000 7fc0bd5fb700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:35.180 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.172+0000 7fc0bd5fb700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b813d670 con 0x7fc0b80a5b70 2023-09-30T02:11:35.181 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.172+0000 7fc0bd5fb700 10 monclient(hunting): _renew_subs 2023-09-30T02:11:35.181 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.172+0000 7fc0bd5fb700 10 monclient(hunting): authenticate will time out at 2140.474921s 2023-09-30T02:11:35.181 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.173+0000 7fc0b6ffd700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc0b80a5b70 0x7fc0b8144ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:35.181 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.173+0000 7fc0b67fc700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0b80b1500 0x7fc0b813c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:35.181 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.173+0000 7fc0b77fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0b8142290 0x7fc0b813cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:35.181 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.173+0000 7fc0b77fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0b8142290 0x7fc0b813cb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:60554/0 (socket says 172.21.15.59:60554) 2023-09-30T02:11:35.181 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.173+0000 7fc0b77fe700 1 -- 172.21.15.59:0/2063709016 learned_addr learned my addr 172.21.15.59:0/2063709016 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:11:35.181 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.173+0000 7fc0b77fe700 10 monclient(hunting): get_auth_request con 0x7fc0b8142290 auth_method 0 2023-09-30T02:11:35.181 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.173+0000 7fc0b77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:35.181 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.173+0000 7fc0b77fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:35.181 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.173+0000 7fc0b77fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:35.182 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.173+0000 7fc0b6ffd700 10 monclient(hunting): get_auth_request con 0x7fc0b80a5b70 auth_method 0 2023-09-30T02:11:35.182 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.173+0000 7fc0b6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:35.182 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.173+0000 7fc0b6ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:35.182 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.173+0000 7fc0b6ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:35.182 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.174+0000 7fc0b67fc700 10 monclient(hunting): get_auth_request con 0x7fc0b80b1500 auth_method 0 2023-09-30T02:11:35.182 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.174+0000 7fc0b67fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:35.182 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.174+0000 7fc0b67fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:35.182 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.174+0000 7fc0b67fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:35.182 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.174+0000 7fc0b77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:35.182 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.174+0000 7fc0b77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:35.183 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.174+0000 7fc0b77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:35.183 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.174+0000 7fc0b6ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:35.183 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.174+0000 7fc0b6ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:35.183 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.174+0000 7fc0b6ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:35.183 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.174+0000 7fc0b67fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:35.183 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.174+0000 7fc0b67fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:35.183 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.174+0000 7fc0b67fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:35.183 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.177+0000 7fc0b77fe700 10 monclient(hunting): handle_auth_done global_id 4888 payload 824 2023-09-30T02:11:35.183 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.177+0000 7fc0b77fe700 1 -- 172.21.15.59:0/2063709016 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc0b80a5b70 msgr2=0x7fc0b8144ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:35.183 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.177+0000 7fc0b77fe700 1 --2- 172.21.15.59:0/2063709016 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc0b80a5b70 0x7fc0b8144ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:35.183 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.177+0000 7fc0b77fe700 1 -- 172.21.15.59:0/2063709016 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0b80b1500 msgr2=0x7fc0b813c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:35.183 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.177+0000 7fc0b77fe700 1 --2- 172.21.15.59:0/2063709016 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0b80b1500 0x7fc0b813c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:35.184 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.177+0000 7fc0b77fe700 10 monclient: _finish_hunting 0 2023-09-30T02:11:35.184 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.177+0000 7fc0b77fe700 1 monclient: found mon.noname-b 2023-09-30T02:11:35.184 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.177+0000 7fc0b77fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:11:35.184 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.177+0000 7fc0b77fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:11:35.184 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.177+0000 7fc0b77fe700 1 -- 172.21.15.59:0/2063709016 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0b0009a20 con 0x7fc0b8142290 2023-09-30T02:11:35.184 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.179+0000 7fc0b77fe700 1 --2- 172.21.15.59:0/2063709016 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0b8142290 0x7fc0b813cb30 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fc0b80a6510 tx=0x7fc0b0019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:35.184 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.179+0000 7fc0aeffd700 10 client.?.objecter ms_handle_connect 0x7fc0b8142290 2023-09-30T02:11:35.184 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.179+0000 7fc0aeffd700 10 client.?.objecter resend_mon_ops 2023-09-30T02:11:35.185 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.183+0000 7fc0aeffd700 1 -- 172.21.15.59:0/2063709016 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0b0027470 con 0x7fc0b8142290 2023-09-30T02:11:35.186 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.183+0000 7fc0aeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:35.186 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.183+0000 7fc0aeffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:11:35.186 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.183+0000 7fc0aeffd700 10 monclient: dump: 2023-09-30T02:11:35.186 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:35.186 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:35.186 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:35.186 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:35.186 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:35.186 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:35.186 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:35.186 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:35.187 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:35.187 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:35.187 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.183+0000 7fc0aeffd700 10 monclient: _finish_auth 0 2023-09-30T02:11:35.187 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.183+0000 7fc0aeffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:11:35.187 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.183+0000 7fc0aeffd700 1 -- 172.21.15.59:0/2063709016 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0b00275f0 con 0x7fc0b8142290 2023-09-30T02:11:35.187 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.183+0000 7fc0bd5fb700 5 monclient: authenticate success, global_id 4888 2023-09-30T02:11:35.187 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.183+0000 7fc0bd5fb700 10 monclient: _renew_subs 2023-09-30T02:11:35.187 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.183+0000 7fc0bd5fb700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:11:35.187 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.183+0000 7fc0bd5fb700 1 -- 172.21.15.59:0/2063709016 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0b814de00 con 0x7fc0b8142290 2023-09-30T02:11:35.187 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.183+0000 7fc0aeffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:11:35.187 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.183+0000 7fc0aeffd700 1 -- 172.21.15.59:0/2063709016 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0b00279b0 con 0x7fc0b8142290 2023-09-30T02:11:35.188 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.183+0000 7fc0aeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:35.188 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.183+0000 7fc0aeffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:11:35.188 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.183+0000 7fc0aeffd700 10 monclient: dump: 2023-09-30T02:11:35.188 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:35.188 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:35.188 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:35.188 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:35.188 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:35.188 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:35.188 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:35.188 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:35.188 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:35.189 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:35.189 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.183+0000 7fc0b57fa700 4 set_mon_vals no callback set 2023-09-30T02:11:35.189 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.183+0000 7fc0bd5fb700 10 client.4888.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:11:35.189 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.183+0000 7fc0bd5fb700 10 monclient: _renew_subs 2023-09-30T02:11:35.189 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.183+0000 7fc0bd5fb700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:11:35.189 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.183+0000 7fc0bd5fb700 1 -- 172.21.15.59:0/2063709016 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0b814e3c0 con 0x7fc0b8142290 2023-09-30T02:11:35.189 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.183+0000 7fc0bd5fb700 1 librados: init done 2023-09-30T02:11:35.189 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.184+0000 7fc0bd5fb700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:11:35.189 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.184+0000 7fc0bd5fb700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:11:35.189 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.184+0000 7fc0bd5fb700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:11:35.189 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.184+0000 7fc0bd5fb700 1 -- 172.21.15.59:0/2063709016 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0b813c2f0 con 0x7fc0b8142290 2023-09-30T02:11:35.362 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.362+0000 7fc0aeffd700 1 -- 172.21.15.59:0/2063709016 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fc0b0024070 con 0x7fc0b8142290 2023-09-30T02:11:35.363 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.363+0000 7fc0aeffd700 1 --2- 172.21.15.59:0/2063709016 >> v2:172.21.15.110:6800/115870 conn(0x7fc098068890 0x7fc09806ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:35.363 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.363+0000 7fc0aeffd700 1 -- 172.21.15.59:0/2063709016 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fc0b0086c80 con 0x7fc0b8142290 2023-09-30T02:11:35.363 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.363+0000 7fc0aeffd700 10 client.4888.objecter ms_dispatch 0x7fc0b8001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:11:35.363 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.363+0000 7fc0aeffd700 3 client.4888.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:11:35.363 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.363+0000 7fc0aeffd700 3 client.4888.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:11:35.363 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.363+0000 7fc0b6ffd700 1 --2- 172.21.15.59:0/2063709016 >> v2:172.21.15.110:6800/115870 conn(0x7fc098068890 0x7fc09806ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:35.364 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.363+0000 7fc0b6ffd700 10 monclient: get_auth_request con 0x7fc098068890 auth_method 0 2023-09-30T02:11:35.364 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.363+0000 7fc0aeffd700 20 client.4888.objecter dump_active .. 0 homeless 2023-09-30T02:11:35.364 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.363+0000 7fc0aeffd700 10 client.4888.objecter ms_handle_connect 0x7fc098068890 2023-09-30T02:11:35.364 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.363+0000 7fc0b6ffd700 1 --2- 172.21.15.59:0/2063709016 >> v2:172.21.15.110:6800/115870 conn(0x7fc098068890 0x7fc09806ad40 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc0a8003a80 tx=0x7fc0a8007ec0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:35.464 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.464+0000 7fc0aeffd700 1 -- 172.21.15.59:0/2063709016 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc0b0054520 con 0x7fc0b8142290 2023-09-30T02:11:35.464 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.464+0000 7fc0aeffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:11:35.464 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.464+0000 7fc0aeffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:11:35.674 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.674+0000 7fc0bd5fb700 1 -- 172.21.15.59:0/2063709016 --> v2:172.21.15.110:6800/115870 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7fc0b8064a00 con 0x7fc098068890 2023-09-30T02:11:35.674 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.675+0000 7fc0aeffd700 1 -- 172.21.15.59:0/2063709016 <== mgr.4108 v2:172.21.15.110:6800/115870 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fc0b8064a00 con 0x7fc098068890 2023-09-30T02:11:35.678 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.677+0000 7fc0bd5fb700 10 librados: watch_flush enter 2023-09-30T02:11:35.678 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.677+0000 7fc0bd5fb700 10 librados: watch_flush exit 2023-09-30T02:11:35.678 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.677+0000 7fc0bd5fb700 20 client.4888.objecter shutdown clearing up homeless session... 2023-09-30T02:11:35.678 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.677+0000 7fc0bd5fb700 10 client.4888.objecter successfully canceled tick 2023-09-30T02:11:35.678 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.677+0000 7fc0bd5fb700 1 -- 172.21.15.59:0/2063709016 >> v2:172.21.15.110:6800/115870 conn(0x7fc098068890 msgr2=0x7fc09806ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:35.678 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.677+0000 7fc0bd5fb700 1 --2- 172.21.15.59:0/2063709016 >> v2:172.21.15.110:6800/115870 conn(0x7fc098068890 0x7fc09806ad40 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fc0a8003a80 tx=0x7fc0a8007ec0 comp rx=0 tx=0).stop 2023-09-30T02:11:35.678 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.677+0000 7fc0bd5fb700 10 monclient: shutdown 2023-09-30T02:11:35.678 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.677+0000 7fc0bd5fb700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:11:35.678 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.677+0000 7fc0bd5fb700 1 -- 172.21.15.59:0/2063709016 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0b8142290 msgr2=0x7fc0b813cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:35.678 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.677+0000 7fc0bd5fb700 1 --2- 172.21.15.59:0/2063709016 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0b8142290 0x7fc0b813cb30 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fc0b80a6510 tx=0x7fc0b0019000 comp rx=0 tx=0).stop 2023-09-30T02:11:35.679 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.677+0000 7fc0bd5fb700 1 -- 172.21.15.59:0/2063709016 shutdown_connections 2023-09-30T02:11:35.679 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.677+0000 7fc0bd5fb700 1 --2- 172.21.15.59:0/2063709016 >> v2:172.21.15.110:6800/115870 conn(0x7fc098068890 0x7fc09806ad40 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:35.679 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.677+0000 7fc0bd5fb700 1 --2- 172.21.15.59:0/2063709016 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc0b8142290 0x7fc0b813cb30 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:35.679 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.677+0000 7fc0bd5fb700 1 --2- 172.21.15.59:0/2063709016 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc0b80a5b70 0x7fc0b8144ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:35.679 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.677+0000 7fc0bd5fb700 1 --2- 172.21.15.59:0/2063709016 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0b80b1500 0x7fc0b813c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:35.679 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.677+0000 7fc0bd5fb700 1 -- 172.21.15.59:0/2063709016 >> 172.21.15.59:0/2063709016 conn(0x7fc0b80631f0 msgr2=0x7fc0b80b3970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:11:35.679 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.677+0000 7fc0bd5fb700 1 -- 172.21.15.59:0/2063709016 shutdown_connections 2023-09-30T02:11:35.679 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.677+0000 7fc0bd5fb700 1 -- 172.21.15.59:0/2063709016 wait complete. 2023-09-30T02:11:35.679 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.678+0000 7fc0bd5fb700 1 librados: shutdown 2023-09-30T02:11:35.679 INFO:teuthology.orchestra.run.smithi059.stderr:instructed osd(s) 0 to deep-scrub 2023-09-30T02:11:35.692 INFO:tasks.ceph:Scrubbing osd.1 2023-09-30T02:11:35.693 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-09-30T02:11:35.835 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.835+0000 7f67b6521700 10 monclient: get_monmap_and_config 2023-09-30T02:11:35.835 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.835+0000 7f67b6521700 10 monclient: build_initial_monmap 2023-09-30T02:11:35.835 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.835+0000 7f67b6521700 10 monclient: monmap: 2023-09-30T02:11:35.836 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:11:35.836 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:35.836 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:11:35.836184+0000 2023-09-30T02:11:35.836 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:11:35.836184+0000 2023-09-30T02:11:35.836 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:11:35.836 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:35.836 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:11:35.836 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:11:35.836 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:11:35.836 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:11:35.836 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:11:35.837 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:35.838 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.836+0000 7f67b6521700 1 Processor -- start 2023-09-30T02:11:35.838 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.836+0000 7f67b6521700 1 -- start start 2023-09-30T02:11:35.838 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.836+0000 7f67b6521700 10 monclient: init 2023-09-30T02:11:35.838 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.837+0000 7f67b6521700 10 monclient: _reopen_session rank -1 2023-09-30T02:11:35.839 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.837+0000 7f67b6521700 10 monclient: _add_conns ranks=[3,2,4,1,0] 2023-09-30T02:11:35.839 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.837+0000 7f67b6521700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f67b00a5730 0x7f67b00a7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:35.839 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.837+0000 7f67b6521700 10 monclient(hunting): picked mon.noname-a con 0x7f67b00a5730 addr v2:172.21.15.59:3300/0 2023-09-30T02:11:35.839 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.837+0000 7f67b6521700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f67b01418e0 0x7f67b013ae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:35.839 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.837+0000 7f67b6521700 10 monclient(hunting): picked mon.noname-b con 0x7f67b01418e0 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:11:35.839 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.837+0000 7f67b6521700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f67b013b360 0x7f67b013b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:35.839 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.837+0000 7f67b6521700 10 monclient(hunting): picked mon.noname-c con 0x7f67b013b360 addr v2:172.21.15.110:3300/0 2023-09-30T02:11:35.839 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.837+0000 7f67b6521700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:35.839 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.837+0000 7f67b6521700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67b00a7410 con 0x7f67b00a5730 2023-09-30T02:11:35.839 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.837+0000 7f67b6521700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:35.839 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.837+0000 7f67b6521700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67b00a84e0 con 0x7f67b013b360 2023-09-30T02:11:35.839 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.837+0000 7f67b6521700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:35.840 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.837+0000 7f67b6521700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67b00a7b10 con 0x7f67b01418e0 2023-09-30T02:11:35.840 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.837+0000 7f67b6521700 10 monclient(hunting): _renew_subs 2023-09-30T02:11:35.840 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.837+0000 7f67b6521700 10 monclient(hunting): authenticate will time out at 2141.139934s 2023-09-30T02:11:35.840 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.838+0000 7f67af7fe700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f67b01418e0 0x7f67b013ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:35.840 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.838+0000 7f67affff700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f67b00a5730 0x7f67b00a7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:35.840 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.839+0000 7f67affff700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f67b00a5730 0x7f67b00a7fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50508/0 (socket says 172.21.15.59:50508) 2023-09-30T02:11:35.840 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.839+0000 7f67affff700 1 -- 172.21.15.59:0/3306244415 learned_addr learned my addr 172.21.15.59:0/3306244415 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:11:35.840 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.839+0000 7f67affff700 10 monclient(hunting): get_auth_request con 0x7f67b00a5730 auth_method 0 2023-09-30T02:11:35.840 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.839+0000 7f67affff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:35.840 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.839+0000 7f67affff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:35.840 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.839+0000 7f67affff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:35.840 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.839+0000 7f67af7fe700 10 monclient(hunting): get_auth_request con 0x7f67b01418e0 auth_method 0 2023-09-30T02:11:35.841 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.839+0000 7f67af7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:35.841 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.839+0000 7f67af7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:35.841 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.839+0000 7f67af7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:35.842 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.839+0000 7f67affff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:35.842 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.839+0000 7f67affff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:35.842 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.839+0000 7f67affff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:35.842 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.839+0000 7f67af7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:35.842 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.839+0000 7f67af7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:35.842 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.839+0000 7f67af7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:35.842 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.842+0000 7f67af7fe700 10 monclient(hunting): handle_auth_done global_id 4757 payload 274 2023-09-30T02:11:35.842 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.842+0000 7f67af7fe700 1 -- 172.21.15.59:0/3306244415 >> v2:172.21.15.110:3300/0 conn(0x7f67b013b360 msgr2=0x7f67b013b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:11:35.843 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.842+0000 7f67af7fe700 1 --2- 172.21.15.59:0/3306244415 >> v2:172.21.15.110:3300/0 conn(0x7f67b013b360 0x7f67b013b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:35.843 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.842+0000 7f67af7fe700 1 -- 172.21.15.59:0/3306244415 >> v2:172.21.15.59:3300/0 conn(0x7f67b00a5730 msgr2=0x7f67b00a7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:11:35.843 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.842+0000 7f67af7fe700 1 --2- 172.21.15.59:0/3306244415 >> v2:172.21.15.59:3300/0 conn(0x7f67b00a5730 0x7f67b00a7fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:35.843 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.842+0000 7f67af7fe700 10 monclient: _finish_hunting 0 2023-09-30T02:11:35.843 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.842+0000 7f67af7fe700 1 monclient: found mon.noname-b 2023-09-30T02:11:35.843 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.842+0000 7f67af7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:11:35.843 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.842+0000 7f67af7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.59:3301/0 2023-09-30T02:11:35.843 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.842+0000 7f67af7fe700 1 -- 172.21.15.59:0/3306244415 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67b00a5b00 con 0x7f67b01418e0 2023-09-30T02:11:35.843 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.844+0000 7f67af7fe700 1 --2- 172.21.15.59:0/3306244415 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f67b01418e0 0x7f67b013ae20 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f67b00b2670 tx=0x7f67a40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e29e46de80081268 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:35.848 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.849+0000 7f67aeffd700 1 -- 172.21.15.59:0/3306244415 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67a4011630 con 0x7f67b01418e0 2023-09-30T02:11:35.849 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.849+0000 7f67aeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:35.849 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.849+0000 7f67aeffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:11:35.849 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.849+0000 7f67aeffd700 10 monclient: dump: 2023-09-30T02:11:35.849 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:35.849 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:35.849 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:35.849 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:35.849 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:35.849 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:35.849 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:35.849 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:35.850 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:35.850 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:35.850 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.849+0000 7f67aeffd700 10 monclient: _finish_auth 0 2023-09-30T02:11:35.850 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.849+0000 7f67aeffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:11:35.850 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.849+0000 7f67aeffd700 1 -- 172.21.15.59:0/3306244415 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67a4011790 con 0x7f67b01418e0 2023-09-30T02:11:35.850 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.849+0000 7f67aeffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:11:35.850 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.849+0000 7f67aeffd700 1 -- 172.21.15.59:0/3306244415 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67a4011b30 con 0x7f67b01418e0 2023-09-30T02:11:35.850 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.849+0000 7f67aeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:35.850 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.849+0000 7f67aeffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:11:35.850 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.849+0000 7f67aeffd700 10 monclient: dump: 2023-09-30T02:11:35.850 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:35.851 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:35.851 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:35.851 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:35.851 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:35.851 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:35.851 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:35.851 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:35.851 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:35.851 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:35.851 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.849+0000 7f67b6521700 5 monclient: authenticate success, global_id 4757 2023-09-30T02:11:35.851 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.849+0000 7f67b6521700 10 monclient: get_monmap_and_config success 2023-09-30T02:11:35.851 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.849+0000 7f67b6521700 4 set_mon_vals no callback set 2023-09-30T02:11:35.855 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.850+0000 7f67b6521700 10 monclient: shutdown 2023-09-30T02:11:35.855 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.850+0000 7f67b6521700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:11:35.855 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.850+0000 7f67b6521700 1 -- 172.21.15.59:0/3306244415 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f67b01418e0 msgr2=0x7f67b013ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:35.855 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.850+0000 7f67b6521700 1 --2- 172.21.15.59:0/3306244415 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f67b01418e0 0x7f67b013ae20 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f67b00b2670 tx=0x7f67a40091a0 comp rx=0 tx=0).stop 2023-09-30T02:11:35.855 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.850+0000 7f67b6521700 1 -- 172.21.15.59:0/3306244415 shutdown_connections 2023-09-30T02:11:35.855 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.850+0000 7f67b6521700 1 --2- 172.21.15.59:0/3306244415 >> v2:172.21.15.110:3300/0 conn(0x7f67b013b360 0x7f67b013b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:35.855 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.850+0000 7f67b6521700 1 --2- 172.21.15.59:0/3306244415 >> v2:172.21.15.59:3300/0 conn(0x7f67b00a5730 0x7f67b00a7fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:35.855 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.850+0000 7f67b6521700 1 --2- 172.21.15.59:0/3306244415 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f67b01418e0 0x7f67b013ae20 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:35.855 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.850+0000 7f67b6521700 1 -- 172.21.15.59:0/3306244415 >> 172.21.15.59:0/3306244415 conn(0x7f67b00631f0 msgr2=0x7f67b00635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:11:35.855 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.850+0000 7f67b6521700 1 -- 172.21.15.59:0/3306244415 shutdown_connections 2023-09-30T02:11:35.856 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.850+0000 7f67b6521700 1 -- 172.21.15.59:0/3306244415 wait complete. 2023-09-30T02:11:35.856 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.850+0000 7f67b6521700 10 monclient: build_initial_monmap 2023-09-30T02:11:35.856 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.850+0000 7f67b6521700 10 monclient: monmap: 2023-09-30T02:11:35.856 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:11:35.856 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:11:35.856 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:11:35.856 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:11:35.856 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:11:35.856 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:11:35.856 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:11:35.856 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:11:35.856 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:11:35.857 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:35.857 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.851+0000 7f67b6521700 1 librados: starting msgr at 2023-09-30T02:11:35.857 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.851+0000 7f67b6521700 1 librados: starting objecter 2023-09-30T02:11:35.857 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.851+0000 7f67b6521700 1 Processor -- start 2023-09-30T02:11:35.857 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.851+0000 7f67b6521700 1 -- start start 2023-09-30T02:11:35.857 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.851+0000 7f67b6521700 1 librados: setting wanted keys 2023-09-30T02:11:35.857 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.851+0000 7f67b6521700 1 librados: calling monclient init 2023-09-30T02:11:35.857 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.851+0000 7f67b6521700 10 monclient: init 2023-09-30T02:11:35.857 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.852+0000 7f67b6521700 10 monclient: _reopen_session rank -1 2023-09-30T02:11:35.857 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.852+0000 7f67b6521700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T02:11:35.857 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.852+0000 7f67b6521700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f67b00a5730 0x7f67b0139fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:35.857 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.852+0000 7f67b6521700 10 monclient(hunting): picked mon.noname-c con 0x7f67b00a5730 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:11:35.858 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.852+0000 7f67b6521700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f67b013b360 0x7f67b014ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:35.858 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.852+0000 7f67b6521700 10 monclient(hunting): picked mon.noname-b con 0x7f67b013b360 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:11:35.858 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.852+0000 7f67b6521700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67b01418e0 0x7f67b00abc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:35.858 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.852+0000 7f67b6521700 10 monclient(hunting): picked mon.noname-a con 0x7f67b01418e0 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:11:35.858 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.852+0000 7f67b6521700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:35.858 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.852+0000 7f67b6521700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67b00ae3c0 con 0x7f67b01418e0 2023-09-30T02:11:35.858 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.852+0000 7f67b6521700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:35.858 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.852+0000 7f67b6521700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67b013d520 con 0x7f67b013b360 2023-09-30T02:11:35.858 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.852+0000 7f67b6521700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:35.858 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.852+0000 7f67b6521700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67b00acce0 con 0x7f67b00a5730 2023-09-30T02:11:35.859 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.852+0000 7f67b6521700 10 monclient(hunting): _renew_subs 2023-09-30T02:11:35.859 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.852+0000 7f67b6521700 10 monclient(hunting): authenticate will time out at 2141.154314s 2023-09-30T02:11:35.859 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.853+0000 7f67b4abe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67b01418e0 0x7f67b00abc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:35.859 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.853+0000 7f67affff700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f67b00a5730 0x7f67b0139fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:35.859 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.853+0000 7f67affff700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f67b00a5730 0x7f67b0139fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:33804/0 (socket says 172.21.15.59:33804) 2023-09-30T02:11:35.859 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.853+0000 7f67b4abe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67b01418e0 0x7f67b00abc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:50518/0 (socket says 172.21.15.59:50518) 2023-09-30T02:11:35.859 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.853+0000 7f67affff700 1 -- 172.21.15.59:0/252381559 learned_addr learned my addr 172.21.15.59:0/252381559 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:11:35.859 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.853+0000 7f67affff700 10 monclient(hunting): get_auth_request con 0x7f67b00a5730 auth_method 0 2023-09-30T02:11:35.859 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.853+0000 7f67affff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:35.859 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.853+0000 7f67affff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:35.859 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.853+0000 7f67affff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:35.860 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.853+0000 7f67b4abe700 10 monclient(hunting): get_auth_request con 0x7f67b01418e0 auth_method 0 2023-09-30T02:11:35.860 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.853+0000 7f67b4abe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:35.860 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.853+0000 7f67b4abe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:35.860 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.853+0000 7f67b4abe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:35.860 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.854+0000 7f67b4abe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:35.860 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.854+0000 7f67b4abe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:35.860 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.854+0000 7f67b4abe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:35.860 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.854+0000 7f67affff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:35.860 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.854+0000 7f67affff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:35.860 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.854+0000 7f67affff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:35.861 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.857+0000 7f67affff700 10 monclient(hunting): handle_auth_done global_id 4760 payload 824 2023-09-30T02:11:35.861 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.857+0000 7f67affff700 1 -- 172.21.15.59:0/252381559 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f67b013b360 msgr2=0x7f67b014ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:35.862 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.857+0000 7f67affff700 1 --2- 172.21.15.59:0/252381559 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f67b013b360 0x7f67b014ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:35.862 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.857+0000 7f67affff700 1 -- 172.21.15.59:0/252381559 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67b01418e0 msgr2=0x7f67b00abc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:35.862 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.857+0000 7f67affff700 1 --2- 172.21.15.59:0/252381559 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67b01418e0 0x7f67b00abc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:35.862 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.857+0000 7f67affff700 10 monclient: _finish_hunting 0 2023-09-30T02:11:35.862 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.857+0000 7f67affff700 1 monclient: found mon.noname-c 2023-09-30T02:11:35.862 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.857+0000 7f67affff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:11:35.862 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.857+0000 7f67affff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.59:3301/0 2023-09-30T02:11:35.862 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.857+0000 7f67affff700 1 -- 172.21.15.59:0/252381559 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67a4009420 con 0x7f67b00a5730 2023-09-30T02:11:35.862 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.860+0000 7f67affff700 1 --2- 172.21.15.59:0/252381559 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f67b00a5730 0x7f67b0139fe0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f67a000ea50 tx=0x7f67a0009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:35.862 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.860+0000 7f6797fff700 10 client.?.objecter ms_handle_connect 0x7f67b00a5730 2023-09-30T02:11:35.862 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.860+0000 7f6797fff700 10 client.?.objecter resend_mon_ops 2023-09-30T02:11:35.864 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.862+0000 7f6797fff700 1 -- 172.21.15.59:0/252381559 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67a000eee0 con 0x7f67b00a5730 2023-09-30T02:11:35.864 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.862+0000 7f6797fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:35.864 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.862+0000 7f6797fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:11:35.864 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.862+0000 7f6797fff700 10 monclient: dump: 2023-09-30T02:11:35.864 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:35.864 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:35.864 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:35.864 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:35.864 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:35.864 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:35.865 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:35.865 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:35.865 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:35.865 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:35.865 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.862+0000 7f6797fff700 10 monclient: _finish_auth 0 2023-09-30T02:11:35.865 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.862+0000 7f6797fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:11:35.865 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.862+0000 7f67b6521700 5 monclient: authenticate success, global_id 4760 2023-09-30T02:11:35.865 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.862+0000 7f67b6521700 10 monclient: _renew_subs 2023-09-30T02:11:35.865 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.862+0000 7f67b6521700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:11:35.865 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.862+0000 7f67b6521700 1 -- 172.21.15.59:0/252381559 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67b013f010 con 0x7f67b00a5730 2023-09-30T02:11:35.865 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.862+0000 7f67b6521700 10 client.4760.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:11:35.865 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.863+0000 7f67b6521700 10 monclient: _renew_subs 2023-09-30T02:11:35.866 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.863+0000 7f67b6521700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:11:35.866 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.863+0000 7f67b6521700 1 -- 172.21.15.59:0/252381559 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67b00ae850 con 0x7f67b00a5730 2023-09-30T02:11:35.866 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.863+0000 7f67b6521700 1 librados: init done 2023-09-30T02:11:35.866 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.863+0000 7f67b6521700 10 client.4760.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:11:35.866 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.863+0000 7f67b6521700 15 client.4760.objecter _session_command_op_assign -1 1 2023-09-30T02:11:35.866 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.863+0000 7f67b6521700 20 client.4760.objecter _get_session osd=-1 returning homeless 2023-09-30T02:11:35.866 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.863+0000 7f67b6521700 10 client.4760.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:11:35.866 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.863+0000 7f67b6521700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:11:35.866 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.863+0000 7f67b6521700 1 -- 172.21.15.59:0/252381559 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f67b00aeb50 con 0x7f67b00a5730 2023-09-30T02:11:35.866 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.864+0000 7f6797fff700 1 -- 172.21.15.59:0/252381559 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67b00aeb50 con 0x7f67b00a5730 2023-09-30T02:11:35.866 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.864+0000 7f6797fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:11:35.866 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.864+0000 7f6797fff700 1 -- 172.21.15.59:0/252381559 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67a000ca60 con 0x7f67b00a5730 2023-09-30T02:11:35.867 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.864+0000 7f6797fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:35.867 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.864+0000 7f6797fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:11:35.867 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.864+0000 7f6797fff700 10 monclient: dump: 2023-09-30T02:11:35.867 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:35.867 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:35.867 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:35.867 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:35.867 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:35.867 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:35.867 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:35.867 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:35.867 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:35.867 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:35.868 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.864+0000 7f67ae7fc700 4 set_mon_vals no callback set 2023-09-30T02:11:35.893 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.893+0000 7f6797fff700 1 -- 172.21.15.59:0/252381559 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f67a000cbc0 con 0x7f67b00a5730 2023-09-30T02:11:35.893 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.893+0000 7f6797fff700 1 --2- 172.21.15.59:0/252381559 >> v2:172.21.15.110:6800/115870 conn(0x7f679c068890 0x7f679c06ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:35.893 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.893+0000 7f6797fff700 1 -- 172.21.15.59:0/252381559 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f67a0084270 con 0x7f67b00a5730 2023-09-30T02:11:35.893 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.893+0000 7f6797fff700 10 client.4760.objecter ms_dispatch 0x7f67b0001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:11:35.894 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.893+0000 7f6797fff700 3 client.4760.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:11:35.894 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.893+0000 7f6797fff700 3 client.4760.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:11:35.894 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.893+0000 7f67af7fe700 1 --2- 172.21.15.59:0/252381559 >> v2:172.21.15.110:6800/115870 conn(0x7f679c068890 0x7f679c06ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:35.894 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.893+0000 7f6797fff700 10 client.4760.objecter checking command 1 2023-09-30T02:11:35.894 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.893+0000 7f6797fff700 1 --2- 172.21.15.59:0/252381559 >> v2:172.21.15.59:6808/116665 conn(0x7f679c06e800 0x7f679c070c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:35.894 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.893+0000 7f67af7fe700 10 monclient: get_auth_request con 0x7f679c068890 auth_method 0 2023-09-30T02:11:35.894 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.893+0000 7f6797fff700 20 client.4760.objecter _get_session s=0x7f679c06e150 osd=1 3 2023-09-30T02:11:35.894 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.893+0000 7f6797fff700 20 client.4760.objecter put_session s=0x7f679c06e150 osd=1 3 2023-09-30T02:11:35.894 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.893+0000 7f6797fff700 15 client.4760.objecter _session_command_op_remove -1 1 2023-09-30T02:11:35.894 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.893+0000 7f6797fff700 20 client.4760.objecter _get_session s=0x7f679c06e150 osd=1 3 2023-09-30T02:11:35.894 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.893+0000 7f6797fff700 20 client.4760.objecter get_session s=0x7f679c06e150 osd=1 3 2023-09-30T02:11:35.895 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.893+0000 7f6797fff700 15 client.4760.objecter _session_command_op_assign 1 1 2023-09-30T02:11:35.895 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.893+0000 7f6797fff700 20 client.4760.objecter put_session s=0x7f679c06e150 osd=1 4 2023-09-30T02:11:35.895 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.893+0000 7f6797fff700 10 client.4760.objecter _send_command 1 2023-09-30T02:11:35.895 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.893+0000 7f6797fff700 1 -- 172.21.15.59:0/252381559 --> v2:172.21.15.59:6808/116665 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f679c071280 con 0x7f679c06e800 2023-09-30T02:11:35.895 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.893+0000 7f6797fff700 20 client.4760.objecter dump_active .. 0 homeless 2023-09-30T02:11:35.895 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.894+0000 7f6797fff700 1 -- 172.21.15.59:0/252381559 <== mon.2 v2:172.21.15.59:3301/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f67a00845a0 con 0x7f67b00a5730 2023-09-30T02:11:35.895 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.894+0000 7f6797fff700 10 monclient: handle_get_version_reply finishing 1 version 25 2023-09-30T02:11:35.895 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.894+0000 7f67af7fe700 1 --2- 172.21.15.59:0/252381559 >> v2:172.21.15.110:6800/115870 conn(0x7f679c068890 0x7f679c06ad40 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f67b013a860 tx=0x7f67a4017000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:35.895 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.894+0000 7f6797fff700 10 client.4760.objecter ms_handle_connect 0x7f679c068890 2023-09-30T02:11:35.896 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.894+0000 7f67b4abe700 1 --2- 172.21.15.59:0/252381559 >> v2:172.21.15.59:6808/116665 conn(0x7f679c06e800 0x7f679c070c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:35.896 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.895+0000 7f67b4abe700 10 monclient: get_auth_request con 0x7f679c06e800 auth_method 0 2023-09-30T02:11:35.897 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.898+0000 7f6797fff700 10 client.4760.objecter ms_handle_connect 0x7f679c06e800 2023-09-30T02:11:35.897 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.898+0000 7f67b4abe700 1 --2- 172.21.15.59:0/252381559 >> v2:172.21.15.59:6808/116665 conn(0x7f679c06e800 0x7f679c070c10 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:35.923 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.923+0000 7f6797fff700 1 -- 172.21.15.59:0/252381559 <== osd.1 v2:172.21.15.59:6808/116665 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f679c071280 con 0x7f679c06e800 2023-09-30T02:11:35.923 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.923+0000 7f6797fff700 10 client.4760.objecter ms_dispatch 0x7f67b0001550 command_reply(tid 1: 0 ) v1 2023-09-30T02:11:35.923 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.923+0000 7f6797fff700 10 client.4760.objecter _finish_command 1 = system:0 2023-09-30T02:11:35.923 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.923+0000 7f6797fff700 20 client.4760.objecter put_session s=0x7f679c06e150 osd=1 4 2023-09-30T02:11:35.923 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.923+0000 7f6797fff700 15 client.4760.objecter _session_command_op_remove 1 1 2023-09-30T02:11:35.938 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.938+0000 7f67b6521700 10 client.4760.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-09-30T02:11:35.938 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.938+0000 7f67b6521700 15 client.4760.objecter _session_command_op_assign -1 2 2023-09-30T02:11:35.938 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.938+0000 7f67b6521700 20 client.4760.objecter _get_session s=0x7f679c06e150 osd=1 3 2023-09-30T02:11:35.938 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.938+0000 7f67b6521700 20 client.4760.objecter put_session s=0x7f679c06e150 osd=1 3 2023-09-30T02:11:35.938 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.938+0000 7f67b6521700 20 client.4760.objecter _get_session s=0x7f679c06e150 osd=1 3 2023-09-30T02:11:35.938 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.938+0000 7f67b6521700 15 client.4760.objecter _session_command_op_remove -1 2 2023-09-30T02:11:35.938 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.938+0000 7f67b6521700 20 client.4760.objecter get_session s=0x7f679c06e150 osd=1 3 2023-09-30T02:11:35.938 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.938+0000 7f67b6521700 15 client.4760.objecter _session_command_op_assign 1 2 2023-09-30T02:11:35.938 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.938+0000 7f67b6521700 20 client.4760.objecter put_session s=0x7f679c06e150 osd=1 4 2023-09-30T02:11:35.939 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.938+0000 7f67b6521700 10 client.4760.objecter _send_command 2 2023-09-30T02:11:35.939 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.938+0000 7f67b6521700 1 -- 172.21.15.59:0/252381559 --> v2:172.21.15.59:6808/116665 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f67b00b0850 con 0x7f679c06e800 2023-09-30T02:11:35.963 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.963+0000 7f6797fff700 1 -- 172.21.15.59:0/252381559 <== osd.1 v2:172.21.15.59:6808/116665 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f67b00b0850 con 0x7f679c06e800 2023-09-30T02:11:35.963 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.963+0000 7f6797fff700 10 client.4760.objecter ms_dispatch 0x7f67b0001550 command_reply(tid 2: 0 ) v1 2023-09-30T02:11:35.963 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.963+0000 7f6797fff700 10 client.4760.objecter _finish_command 2 = system:0 2023-09-30T02:11:35.963 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.963+0000 7f6797fff700 20 client.4760.objecter put_session s=0x7f679c06e150 osd=1 4 2023-09-30T02:11:35.963 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.963+0000 7f6797fff700 15 client.4760.objecter _session_command_op_remove 1 2 2023-09-30T02:11:35.963 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2023-09-30T02:11:35.963 INFO:teuthology.orchestra.run.smithi059.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T02:11:35.964 INFO:teuthology.orchestra.run.smithi059.stdout:} 2023-09-30T02:11:35.965 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.964+0000 7f67b6521700 10 librados: watch_flush enter 2023-09-30T02:11:35.965 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.964+0000 7f67b6521700 10 librados: watch_flush exit 2023-09-30T02:11:35.965 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.964+0000 7f67b6521700 10 client.4760.objecter close_session for osd.1 2023-09-30T02:11:35.965 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.964+0000 7f67b6521700 1 -- 172.21.15.59:0/252381559 >> v2:172.21.15.59:6808/116665 conn(0x7f679c06e800 msgr2=0x7f679c070c10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:35.965 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.964+0000 7f67b6521700 1 --2- 172.21.15.59:0/252381559 >> v2:172.21.15.59:6808/116665 conn(0x7f679c06e800 0x7f679c070c10 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:35.965 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.964+0000 7f67b6521700 20 client.4760.objecter put_session s=0x7f679c06e150 osd=1 1 2023-09-30T02:11:35.965 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.964+0000 7f67b6521700 20 client.4760.objecter shutdown clearing up homeless session... 2023-09-30T02:11:35.965 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.964+0000 7f67b6521700 10 client.4760.objecter successfully canceled tick 2023-09-30T02:11:35.966 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.964+0000 7f67b6521700 1 -- 172.21.15.59:0/252381559 >> v2:172.21.15.110:6800/115870 conn(0x7f679c068890 msgr2=0x7f679c06ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:35.966 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.964+0000 7f67b6521700 1 --2- 172.21.15.59:0/252381559 >> v2:172.21.15.110:6800/115870 conn(0x7f679c068890 0x7f679c06ad40 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f67b013a860 tx=0x7f67a4017000 comp rx=0 tx=0).stop 2023-09-30T02:11:35.966 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.964+0000 7f67b6521700 10 monclient: shutdown 2023-09-30T02:11:35.966 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.964+0000 7f67b6521700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:11:35.966 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.964+0000 7f67b6521700 1 -- 172.21.15.59:0/252381559 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f67b00a5730 msgr2=0x7f67b0139fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:35.966 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.964+0000 7f67b6521700 1 --2- 172.21.15.59:0/252381559 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f67b00a5730 0x7f67b0139fe0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f67a000ea50 tx=0x7f67a0009250 comp rx=0 tx=0).stop 2023-09-30T02:11:35.966 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.964+0000 7f67b4abe700 1 -- 172.21.15.59:0/252381559 reap_dead start 2023-09-30T02:11:35.966 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.964+0000 7f67b6521700 1 -- 172.21.15.59:0/252381559 shutdown_connections 2023-09-30T02:11:35.966 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.964+0000 7f67b6521700 1 -- 172.21.15.59:0/252381559 >> 172.21.15.59:0/252381559 conn(0x7f67b00631f0 msgr2=0x7f67b0141d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:11:35.966 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.964+0000 7f67b6521700 1 -- 172.21.15.59:0/252381559 shutdown_connections 2023-09-30T02:11:35.966 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.964+0000 7f67b6521700 1 -- 172.21.15.59:0/252381559 wait complete. 2023-09-30T02:11:35.966 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:35.964+0000 7f67b6521700 1 librados: shutdown 2023-09-30T02:11:35.976 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-09-30T02:11:36.109 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.109+0000 7f3eb015c700 10 monclient: get_monmap_and_config 2023-09-30T02:11:36.109 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.109+0000 7f3eb015c700 10 monclient: build_initial_monmap 2023-09-30T02:11:36.109 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.109+0000 7f3eb015c700 10 monclient: monmap: 2023-09-30T02:11:36.109 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:11:36.109 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:36.109 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:11:36.110261+0000 2023-09-30T02:11:36.110 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:11:36.110261+0000 2023-09-30T02:11:36.110 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:11:36.110 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:36.110 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:11:36.110 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:11:36.110 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:11:36.110 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:11:36.110 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:11:36.110 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:36.112 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.110+0000 7f3eb015c700 1 Processor -- start 2023-09-30T02:11:36.112 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.110+0000 7f3eb015c700 1 -- start start 2023-09-30T02:11:36.112 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.110+0000 7f3eb015c700 10 monclient: init 2023-09-30T02:11:36.112 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.111+0000 7f3eb015c700 10 monclient: _reopen_session rank -1 2023-09-30T02:11:36.112 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.111+0000 7f3eb015c700 10 monclient: _add_conns ranks=[0,1,2,4,3] 2023-09-30T02:11:36.112 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.111+0000 7f3eb015c700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3ea80a5b70 addr v1:172.21.15.59:6789/0 2023-09-30T02:11:36.112 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.111+0000 7f3eb015c700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3ea8142290 addr v1:172.21.15.110:6789/0 2023-09-30T02:11:36.112 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.111+0000 7f3eb015c700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3ea80b1500 0x7f3ea80a5f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:36.113 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.111+0000 7f3eb015c700 10 monclient(hunting): picked mon.noname-b con 0x7f3ea80b1500 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:11:36.113 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.111+0000 7f3eb015c700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ea80a75a0 con 0x7f3ea80a5b70 2023-09-30T02:11:36.113 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.111+0000 7f3eb015c700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ea80af3f0 con 0x7f3ea8142290 2023-09-30T02:11:36.113 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.111+0000 7f3eb015c700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:36.113 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.111+0000 7f3eb015c700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea80b1940 con 0x7f3ea80b1500 2023-09-30T02:11:36.113 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.111+0000 7f3eb015c700 10 monclient(hunting): _renew_subs 2023-09-30T02:11:36.113 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.111+0000 7f3eb015c700 10 monclient(hunting): authenticate will time out at 2141.413562s 2023-09-30T02:11:36.113 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.112+0000 7f3ead6f7700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f3ea8142290 0x7f3ea813af30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.59:42898/0 (socket says 172.21.15.59:42898) 2023-09-30T02:11:36.113 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.112+0000 7f3ead6f7700 1 -- 172.21.15.59:0/330347250 learned_addr learned my addr 172.21.15.59:0/330347250 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:11:36.113 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.112+0000 7f3eae6f9700 1 --2- 172.21.15.59:0/330347250 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3ea80b1500 0x7f3ea80a5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:36.113 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.112+0000 7f3eae6f9700 10 monclient(hunting): get_auth_request con 0x7f3ea80b1500 auth_method 0 2023-09-30T02:11:36.114 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.112+0000 7f3eae6f9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:36.114 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.112+0000 7f3eae6f9700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:36.114 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.112+0000 7f3eae6f9700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:36.115 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.113+0000 7f3eae6f9700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:36.115 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.113+0000 7f3eae6f9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:36.115 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.113+0000 7f3eae6f9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:36.115 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.115+0000 7f3eacef6700 1 -- 172.21.15.59:0/330347250 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3368894343 0 0) 0x7f3ea80af3f0 con 0x7f3ea8142290 2023-09-30T02:11:36.115 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.115+0000 7f3eacef6700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:36.115 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.115+0000 7f3eacef6700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:36.115 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.115+0000 7f3eacef6700 10 monclient(hunting): my global_id is 4891 2023-09-30T02:11:36.115 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.115+0000 7f3eacef6700 1 -- 172.21.15.59:0/330347250 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e94003570 con 0x7f3ea8142290 2023-09-30T02:11:36.115 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.115+0000 7f3eae6f9700 10 monclient(hunting): handle_auth_done global_id 4763 payload 274 2023-09-30T02:11:36.116 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.115+0000 7f3eae6f9700 1 -- 172.21.15.59:0/330347250 >> v1:172.21.15.110:6789/0 conn(0x7f3ea8142290 legacy=0x7f3ea813af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:36.116 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.115+0000 7f3eae6f9700 1 -- 172.21.15.59:0/330347250 >> v1:172.21.15.59:6789/0 conn(0x7f3ea80a5b70 legacy=0x7f3ea8141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:36.116 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.115+0000 7f3eae6f9700 10 monclient: _finish_hunting 0 2023-09-30T02:11:36.116 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.115+0000 7f3eae6f9700 1 monclient: found mon.noname-b 2023-09-30T02:11:36.116 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.115+0000 7f3eae6f9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:11:36.116 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.115+0000 7f3eae6f9700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.59:3301/0 2023-09-30T02:11:36.116 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.115+0000 7f3eae6f9700 1 -- 172.21.15.59:0/330347250 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ea813b640 con 0x7f3ea80b1500 2023-09-30T02:11:36.116 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.117+0000 7f3eae6f9700 1 --2- 172.21.15.59:0/330347250 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3ea80b1500 0x7f3ea80a5f40 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f3ea0009e90 tx=0x7f3ea0009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8e41f0445d04dec4 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:36.121 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.121+0000 7f3eacef6700 1 -- 172.21.15.59:0/330347250 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea000e070 con 0x7f3ea80b1500 2023-09-30T02:11:36.121 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.121+0000 7f3eacef6700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:36.121 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.121+0000 7f3eacef6700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:11:36.121 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.121+0000 7f3eacef6700 10 monclient: dump: 2023-09-30T02:11:36.121 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:36.121 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:36.121 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:36.122 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:36.122 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:36.122 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:36.122 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:36.122 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:36.122 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:36.122 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:36.122 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.121+0000 7f3eacef6700 10 monclient: _finish_auth 0 2023-09-30T02:11:36.122 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.121+0000 7f3eacef6700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:11:36.123 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.121+0000 7f3eacef6700 1 -- 172.21.15.59:0/330347250 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ea000bc30 con 0x7f3ea80b1500 2023-09-30T02:11:36.123 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.121+0000 7f3eacef6700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:11:36.123 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.121+0000 7f3eacef6700 1 -- 172.21.15.59:0/330347250 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea0012430 con 0x7f3ea80b1500 2023-09-30T02:11:36.123 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.121+0000 7f3eacef6700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:36.123 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.121+0000 7f3eacef6700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:11:36.123 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.121+0000 7f3eacef6700 10 monclient: dump: 2023-09-30T02:11:36.123 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:36.123 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:36.123 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:36.124 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:36.124 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:36.124 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:36.124 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:36.124 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:36.124 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:36.124 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:36.124 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.121+0000 7f3eb015c700 5 monclient: authenticate success, global_id 4763 2023-09-30T02:11:36.124 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.121+0000 7f3eb015c700 10 monclient: get_monmap_and_config success 2023-09-30T02:11:36.125 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.121+0000 7f3eb015c700 4 set_mon_vals no callback set 2023-09-30T02:11:36.128 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.122+0000 7f3eb015c700 10 monclient: shutdown 2023-09-30T02:11:36.128 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.122+0000 7f3eb015c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:11:36.128 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.122+0000 7f3eb015c700 1 -- 172.21.15.59:0/330347250 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3ea80b1500 msgr2=0x7f3ea80a5f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:36.128 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.122+0000 7f3eb015c700 1 --2- 172.21.15.59:0/330347250 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3ea80b1500 0x7f3ea80a5f40 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f3ea0009e90 tx=0x7f3ea0009780 comp rx=0 tx=0).stop 2023-09-30T02:11:36.128 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.122+0000 7f3eb015c700 1 -- 172.21.15.59:0/330347250 shutdown_connections 2023-09-30T02:11:36.128 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.122+0000 7f3eb015c700 1 --2- 172.21.15.59:0/330347250 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3ea80b1500 0x7f3ea80a5f40 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:36.128 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.122+0000 7f3eb015c700 1 -- 172.21.15.59:0/330347250 >> 172.21.15.59:0/330347250 conn(0x7f3ea80631f0 msgr2=0x7f3ea80635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:11:36.129 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.122+0000 7f3eb015c700 1 -- 172.21.15.59:0/330347250 shutdown_connections 2023-09-30T02:11:36.129 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.122+0000 7f3eb015c700 1 -- 172.21.15.59:0/330347250 wait complete. 2023-09-30T02:11:36.129 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.123+0000 7f3eb015c700 10 monclient: build_initial_monmap 2023-09-30T02:11:36.129 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.123+0000 7f3eb015c700 10 monclient: monmap: 2023-09-30T02:11:36.129 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:11:36.129 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:11:36.129 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:11:36.129 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:11:36.129 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:11:36.129 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:11:36.130 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:11:36.130 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:11:36.130 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:11:36.130 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:36.130 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.123+0000 7f3eb015c700 1 librados: starting msgr at 2023-09-30T02:11:36.130 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.123+0000 7f3eb015c700 1 librados: starting objecter 2023-09-30T02:11:36.130 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.123+0000 7f3eb015c700 1 Processor -- start 2023-09-30T02:11:36.130 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.124+0000 7f3eb015c700 1 -- start start 2023-09-30T02:11:36.130 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.124+0000 7f3eb015c700 1 librados: setting wanted keys 2023-09-30T02:11:36.130 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.124+0000 7f3eb015c700 1 librados: calling monclient init 2023-09-30T02:11:36.131 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.124+0000 7f3eb015c700 10 monclient: init 2023-09-30T02:11:36.131 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.124+0000 7f3eb015c700 10 monclient: _reopen_session rank -1 2023-09-30T02:11:36.131 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.124+0000 7f3eb015c700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T02:11:36.131 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.124+0000 7f3eb015c700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3ea80a5b70 0x7f3ea8144b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:36.131 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.124+0000 7f3eb015c700 10 monclient(hunting): picked mon.noname-c con 0x7f3ea80a5b70 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:11:36.131 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.124+0000 7f3eb015c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3ea80b1500 0x7f3ea813e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:36.131 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.124+0000 7f3eb015c700 10 monclient(hunting): picked mon.noname-a con 0x7f3ea80b1500 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:11:36.132 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.124+0000 7f3eb015c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3ea8142290 0x7f3ea814d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:36.132 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.124+0000 7f3eb015c700 10 monclient(hunting): picked mon.noname-b con 0x7f3ea8142290 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:11:36.132 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.124+0000 7f3eb015c700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:36.132 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.124+0000 7f3eb015c700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea813e670 con 0x7f3ea80b1500 2023-09-30T02:11:36.132 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.124+0000 7f3eb015c700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:36.132 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.124+0000 7f3eb015c700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea813e890 con 0x7f3ea8142290 2023-09-30T02:11:36.132 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.124+0000 7f3eb015c700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:36.132 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.124+0000 7f3eb015c700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ea813de60 con 0x7f3ea80a5b70 2023-09-30T02:11:36.133 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.124+0000 7f3eb015c700 10 monclient(hunting): _renew_subs 2023-09-30T02:11:36.133 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.124+0000 7f3eb015c700 10 monclient(hunting): authenticate will time out at 2141.426653s 2023-09-30T02:11:36.133 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.125+0000 7f3eae6f9700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3ea8142290 0x7f3ea814d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:36.133 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.125+0000 7f3ead6f7700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3ea80b1500 0x7f3ea813e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:36.133 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.125+0000 7f3eae6f9700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3ea8142290 0x7f3ea814d850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:60576/0 (socket says 172.21.15.59:60576) 2023-09-30T02:11:36.133 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.125+0000 7f3eae6f9700 1 -- 172.21.15.59:0/3686026779 learned_addr learned my addr 172.21.15.59:0/3686026779 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:11:36.133 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.125+0000 7f3eae6f9700 10 monclient(hunting): get_auth_request con 0x7f3ea8142290 auth_method 0 2023-09-30T02:11:36.133 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.125+0000 7f3eae6f9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:36.133 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.125+0000 7f3eae6f9700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:36.133 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.125+0000 7f3eae6f9700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:36.133 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.125+0000 7f3ead6f7700 10 monclient(hunting): get_auth_request con 0x7f3ea80b1500 auth_method 0 2023-09-30T02:11:36.134 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.125+0000 7f3ead6f7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:36.134 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.125+0000 7f3ead6f7700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:36.134 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.125+0000 7f3ead6f7700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:36.134 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.125+0000 7f3eadef8700 1 --2- 172.21.15.59:0/3686026779 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3ea80a5b70 0x7f3ea8144b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:36.134 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.126+0000 7f3eae6f9700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:36.134 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.126+0000 7f3eae6f9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:36.134 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.126+0000 7f3eae6f9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:36.134 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.126+0000 7f3eadef8700 10 monclient(hunting): get_auth_request con 0x7f3ea80a5b70 auth_method 0 2023-09-30T02:11:36.134 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.126+0000 7f3eadef8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:36.134 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.126+0000 7f3eadef8700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:36.135 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.126+0000 7f3eadef8700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:36.135 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.126+0000 7f3ead6f7700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:36.135 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.126+0000 7f3ead6f7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:36.135 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.126+0000 7f3ead6f7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:36.135 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.126+0000 7f3eadef8700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:36.135 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.126+0000 7f3eadef8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:36.135 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.126+0000 7f3eadef8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:36.137 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.129+0000 7f3eae6f9700 10 monclient(hunting): handle_auth_done global_id 4894 payload 824 2023-09-30T02:11:36.137 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.129+0000 7f3eae6f9700 1 -- 172.21.15.59:0/3686026779 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3ea80a5b70 msgr2=0x7f3ea8144b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:36.137 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.129+0000 7f3eae6f9700 1 --2- 172.21.15.59:0/3686026779 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3ea80a5b70 0x7f3ea8144b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:36.138 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.129+0000 7f3eae6f9700 1 -- 172.21.15.59:0/3686026779 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3ea80b1500 msgr2=0x7f3ea813e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:36.138 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.129+0000 7f3eae6f9700 1 --2- 172.21.15.59:0/3686026779 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3ea80b1500 0x7f3ea813e100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:36.138 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.129+0000 7f3eae6f9700 10 monclient: _finish_hunting 0 2023-09-30T02:11:36.138 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.129+0000 7f3eae6f9700 1 monclient: found mon.noname-b 2023-09-30T02:11:36.138 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.129+0000 7f3eae6f9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:11:36.138 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.129+0000 7f3eae6f9700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:11:36.138 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.129+0000 7f3eae6f9700 1 -- 172.21.15.59:0/3686026779 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ea0009a20 con 0x7f3ea8142290 2023-09-30T02:11:36.138 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.129+0000 7f3ead6f7700 1 --2- 172.21.15.59:0/3686026779 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3ea80b1500 0x7f3ea813e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T02:11:36.138 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.132+0000 7f3eae6f9700 1 --2- 172.21.15.59:0/3686026779 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3ea8142290 0x7f3ea814d850 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f3ea0013000 tx=0x7f3ea0019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:36.138 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.132+0000 7f3e99ffb700 10 client.?.objecter ms_handle_connect 0x7f3ea8142290 2023-09-30T02:11:36.138 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.132+0000 7f3e99ffb700 10 client.?.objecter resend_mon_ops 2023-09-30T02:11:36.139 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.136+0000 7f3e99ffb700 1 -- 172.21.15.59:0/3686026779 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea0027470 con 0x7f3ea8142290 2023-09-30T02:11:36.139 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.136+0000 7f3e99ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:36.139 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.136+0000 7f3e99ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:11:36.139 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.136+0000 7f3e99ffb700 10 monclient: dump: 2023-09-30T02:11:36.139 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:36.139 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:36.139 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:36.139 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:36.140 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:36.140 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:36.140 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:36.140 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:36.140 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:36.140 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:36.140 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.136+0000 7f3e99ffb700 10 monclient: _finish_auth 0 2023-09-30T02:11:36.140 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.136+0000 7f3e99ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:11:36.140 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.136+0000 7f3e99ffb700 1 -- 172.21.15.59:0/3686026779 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ea00275d0 con 0x7f3ea8142290 2023-09-30T02:11:36.140 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.136+0000 7f3e99ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:11:36.140 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.136+0000 7f3e99ffb700 1 -- 172.21.15.59:0/3686026779 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea0027990 con 0x7f3ea8142290 2023-09-30T02:11:36.141 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.136+0000 7f3eb015c700 5 monclient: authenticate success, global_id 4894 2023-09-30T02:11:36.141 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.136+0000 7f3eb015c700 10 monclient: _renew_subs 2023-09-30T02:11:36.141 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.136+0000 7f3eb015c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:11:36.141 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.136+0000 7f3eb015c700 1 -- 172.21.15.59:0/3686026779 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ea81457d0 con 0x7f3ea8142290 2023-09-30T02:11:36.141 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.136+0000 7f3e9bfff700 4 set_mon_vals no callback set 2023-09-30T02:11:36.141 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.136+0000 7f3e99ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:36.141 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.136+0000 7f3e99ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:11:36.141 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.136+0000 7f3e99ffb700 10 monclient: dump: 2023-09-30T02:11:36.141 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:36.141 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:36.142 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:36.142 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:36.142 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:36.142 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:36.142 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:36.142 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:36.142 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:36.142 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:36.142 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.136+0000 7f3eb015c700 10 client.4894.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:11:36.142 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.136+0000 7f3eb015c700 10 monclient: _renew_subs 2023-09-30T02:11:36.143 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.136+0000 7f3eb015c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:11:36.143 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.136+0000 7f3eb015c700 1 -- 172.21.15.59:0/3686026779 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ea814dd90 con 0x7f3ea8142290 2023-09-30T02:11:36.143 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.136+0000 7f3eb015c700 1 librados: init done 2023-09-30T02:11:36.143 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.137+0000 7f3eb015c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:11:36.143 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.137+0000 7f3eb015c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:11:36.143 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.137+0000 7f3eb015c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:11:36.143 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.137+0000 7f3eb015c700 1 -- 172.21.15.59:0/3686026779 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ea8064a00 con 0x7f3ea8142290 2023-09-30T02:11:36.312 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.311+0000 7f3e99ffb700 1 -- 172.21.15.59:0/3686026779 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f3ea0024070 con 0x7f3ea8142290 2023-09-30T02:11:36.312 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.312+0000 7f3e99ffb700 1 --2- 172.21.15.59:0/3686026779 >> v2:172.21.15.110:6800/115870 conn(0x7f3e8c068910 0x7f3e8c06adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:36.313 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.312+0000 7f3e99ffb700 1 -- 172.21.15.59:0/3686026779 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f3ea0086af0 con 0x7f3ea8142290 2023-09-30T02:11:36.313 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.312+0000 7f3e99ffb700 10 client.4894.objecter ms_dispatch 0x7f3ea8001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:11:36.313 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.312+0000 7f3e99ffb700 3 client.4894.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:11:36.313 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.312+0000 7f3e99ffb700 3 client.4894.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:11:36.313 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.312+0000 7f3eadef8700 1 --2- 172.21.15.59:0/3686026779 >> v2:172.21.15.110:6800/115870 conn(0x7f3e8c068910 0x7f3e8c06adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:36.313 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.312+0000 7f3e99ffb700 20 client.4894.objecter dump_active .. 0 homeless 2023-09-30T02:11:36.313 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.312+0000 7f3eadef8700 10 monclient: get_auth_request con 0x7f3e8c068910 auth_method 0 2023-09-30T02:11:36.313 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.313+0000 7f3e99ffb700 10 client.4894.objecter ms_handle_connect 0x7f3e8c068910 2023-09-30T02:11:36.313 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.313+0000 7f3eadef8700 1 --2- 172.21.15.59:0/3686026779 >> v2:172.21.15.110:6800/115870 conn(0x7f3e8c068910 0x7f3e8c06adc0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f3ea4003a80 tx=0x7f3ea4007ec0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:36.397 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.397+0000 7f3e99ffb700 1 -- 172.21.15.59:0/3686026779 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3ea0054440 con 0x7f3ea8142290 2023-09-30T02:11:36.397 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.397+0000 7f3e99ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:11:36.397 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.397+0000 7f3e99ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:11:36.606 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.607+0000 7f3eb015c700 1 -- 172.21.15.59:0/3686026779 --> v2:172.21.15.110:6800/115870 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f3ea806a720 con 0x7f3e8c068910 2023-09-30T02:11:36.607 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.607+0000 7f3e99ffb700 1 -- 172.21.15.59:0/3686026779 <== mgr.4108 v2:172.21.15.110:6800/115870 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f3ea806a720 con 0x7f3e8c068910 2023-09-30T02:11:36.610 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.610+0000 7f3eb015c700 10 librados: watch_flush enter 2023-09-30T02:11:36.610 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.610+0000 7f3eb015c700 10 librados: watch_flush exit 2023-09-30T02:11:36.610 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.610+0000 7f3eb015c700 20 client.4894.objecter shutdown clearing up homeless session... 2023-09-30T02:11:36.610 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.610+0000 7f3eb015c700 10 client.4894.objecter successfully canceled tick 2023-09-30T02:11:36.610 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.610+0000 7f3eb015c700 1 -- 172.21.15.59:0/3686026779 >> v2:172.21.15.110:6800/115870 conn(0x7f3e8c068910 msgr2=0x7f3e8c06adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:36.610 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.610+0000 7f3eb015c700 1 --2- 172.21.15.59:0/3686026779 >> v2:172.21.15.110:6800/115870 conn(0x7f3e8c068910 0x7f3e8c06adc0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f3ea4003a80 tx=0x7f3ea4007ec0 comp rx=0 tx=0).stop 2023-09-30T02:11:36.610 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.610+0000 7f3eb015c700 10 monclient: shutdown 2023-09-30T02:11:36.610 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.610+0000 7f3eb015c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:11:36.610 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.610+0000 7f3eb015c700 1 -- 172.21.15.59:0/3686026779 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3ea8142290 msgr2=0x7f3ea814d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:36.610 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.610+0000 7f3eb015c700 1 --2- 172.21.15.59:0/3686026779 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3ea8142290 0x7f3ea814d850 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f3ea0013000 tx=0x7f3ea0019000 comp rx=0 tx=0).stop 2023-09-30T02:11:36.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.610+0000 7f3eb015c700 1 -- 172.21.15.59:0/3686026779 shutdown_connections 2023-09-30T02:11:36.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.610+0000 7f3eb015c700 1 --2- 172.21.15.59:0/3686026779 >> v2:172.21.15.110:6800/115870 conn(0x7f3e8c068910 0x7f3e8c06adc0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:36.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.610+0000 7f3eb015c700 1 --2- 172.21.15.59:0/3686026779 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3ea8142290 0x7f3ea814d850 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:36.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.610+0000 7f3eb015c700 1 --2- 172.21.15.59:0/3686026779 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f3ea80a5b70 0x7f3ea8144b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:36.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.610+0000 7f3eb015c700 1 --2- 172.21.15.59:0/3686026779 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3ea80b1500 0x7f3ea813e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:36.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.610+0000 7f3eb015c700 1 -- 172.21.15.59:0/3686026779 >> 172.21.15.59:0/3686026779 conn(0x7f3ea80631f0 msgr2=0x7f3ea80635c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:11:36.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.610+0000 7f3eb015c700 1 -- 172.21.15.59:0/3686026779 shutdown_connections 2023-09-30T02:11:36.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.610+0000 7f3eb015c700 1 -- 172.21.15.59:0/3686026779 wait complete. 2023-09-30T02:11:36.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.610+0000 7f3eb015c700 1 librados: shutdown 2023-09-30T02:11:36.612 INFO:teuthology.orchestra.run.smithi059.stderr:instructed osd(s) 1 to deep-scrub 2023-09-30T02:11:36.624 INFO:tasks.ceph:Scrubbing osd.2 2023-09-30T02:11:36.625 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-09-30T02:11:36.765 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.766+0000 7f2d044a8700 10 monclient: get_monmap_and_config 2023-09-30T02:11:36.765 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.766+0000 7f2d044a8700 10 monclient: build_initial_monmap 2023-09-30T02:11:36.765 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.766+0000 7f2d044a8700 10 monclient: monmap: 2023-09-30T02:11:36.766 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:11:36.766 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:36.766 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:11:36.766642+0000 2023-09-30T02:11:36.766 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:11:36.766642+0000 2023-09-30T02:11:36.766 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:11:36.766 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:36.766 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:11:36.766 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:11:36.766 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:11:36.766 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:11:36.766 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:11:36.767 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:36.768 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.766+0000 7f2d044a8700 1 Processor -- start 2023-09-30T02:11:36.768 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.767+0000 7f2d044a8700 1 -- start start 2023-09-30T02:11:36.768 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.767+0000 7f2d044a8700 10 monclient: init 2023-09-30T02:11:36.768 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.767+0000 7f2d044a8700 10 monclient: _reopen_session rank -1 2023-09-30T02:11:36.768 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.767+0000 7f2d044a8700 10 monclient: _add_conns ranks=[4,1,2,3,0] 2023-09-30T02:11:36.768 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.767+0000 7f2d044a8700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f2cfc0a5730 0x7f2cfc0a7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:36.768 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.767+0000 7f2d044a8700 10 monclient(hunting): picked mon.noname-c con 0x7f2cfc0a5730 addr v2:172.21.15.110:3300/0 2023-09-30T02:11:36.768 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.767+0000 7f2d044a8700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2cfc1418e0 addr v1:172.21.15.110:6789/0 2023-09-30T02:11:36.768 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.767+0000 7f2d044a8700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2cfc13b530 0x7f2cfc148f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:36.768 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.767+0000 7f2d044a8700 10 monclient(hunting): picked mon.noname-b con 0x7f2cfc13b530 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:11:36.769 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.767+0000 7f2d044a8700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cfc0a7b10 con 0x7f2cfc1418e0 2023-09-30T02:11:36.769 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.767+0000 7f2d044a8700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:36.769 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.767+0000 7f2d044a8700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cfc0a84e0 con 0x7f2cfc0a5730 2023-09-30T02:11:36.769 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.767+0000 7f2d044a8700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:36.769 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.767+0000 7f2d044a8700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cfc0a5b00 con 0x7f2cfc13b530 2023-09-30T02:11:36.769 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.767+0000 7f2d044a8700 10 monclient(hunting): _renew_subs 2023-09-30T02:11:36.769 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.767+0000 7f2d044a8700 10 monclient(hunting): authenticate will time out at 2142.069998s 2023-09-30T02:11:36.806 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.806+0000 7f2d02244700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f2cfc0a5730 0x7f2cfc0a7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:36.806 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.807+0000 7f2d02244700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f2cfc0a5730 0x7f2cfc0a7fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:60578/0 (socket says 172.21.15.59:60578) 2023-09-30T02:11:36.807 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.807+0000 7f2d02244700 1 -- 172.21.15.59:0/3465618731 learned_addr learned my addr 172.21.15.59:0/3465618731 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:11:36.807 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.807+0000 7f2d02244700 10 monclient(hunting): get_auth_request con 0x7f2cfc0a5730 auth_method 0 2023-09-30T02:11:36.807 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.807+0000 7f2d02244700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:36.807 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.807+0000 7f2d02244700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:36.807 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.807+0000 7f2d02244700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:36.807 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.808+0000 7f2d02244700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:36.807 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.808+0000 7f2d02244700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:36.807 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.808+0000 7f2d02244700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:36.809 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.810+0000 7f2d02244700 10 monclient(hunting): handle_auth_done global_id 4897 payload 274 2023-09-30T02:11:36.809 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.810+0000 7f2d02244700 1 -- 172.21.15.59:0/3465618731 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2cfc13b530 msgr2=0x7f2cfc148f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:11:36.810 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.810+0000 7f2d02244700 1 --2- 172.21.15.59:0/3465618731 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2cfc13b530 0x7f2cfc148f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:36.810 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.810+0000 7f2d02244700 1 -- 172.21.15.59:0/3465618731 >> v1:172.21.15.110:6789/0 conn(0x7f2cfc1418e0 legacy=0x7f2cfc13ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:36.810 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.810+0000 7f2d02244700 10 monclient: _finish_hunting 0 2023-09-30T02:11:36.810 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.810+0000 7f2d02244700 1 monclient: found mon.noname-c 2023-09-30T02:11:36.810 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.810+0000 7f2d02244700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:11:36.810 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.810+0000 7f2d02244700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3300/0 2023-09-30T02:11:36.810 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.810+0000 7f2d02244700 1 -- 172.21.15.59:0/3465618731 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cfc0a7750 con 0x7f2cfc0a5730 2023-09-30T02:11:36.814 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.815+0000 7f2d02244700 1 --2- 172.21.15.59:0/3465618731 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2cfc0a5730 0x7f2cfc0a7fa0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f2cf80099b0 tx=0x7f2cf8009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=69e6329ae871a6ad server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:36.823 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.824+0000 7f2d01242700 1 -- 172.21.15.59:0/3465618731 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cf800b860 con 0x7f2cfc0a5730 2023-09-30T02:11:36.823 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.824+0000 7f2d01242700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:36.824 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.824+0000 7f2d01242700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:11:36.824 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.824+0000 7f2d01242700 10 monclient: dump: 2023-09-30T02:11:36.824 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:36.824 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:36.824 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:36.824 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:36.824 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:36.824 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:36.824 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:36.824 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:36.824 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:36.825 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:36.825 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.824+0000 7f2d01242700 10 monclient: _finish_auth 0 2023-09-30T02:11:36.825 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.824+0000 7f2d01242700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:11:36.825 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.824+0000 7f2d01242700 1 -- 172.21.15.59:0/3465618731 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cf800b9c0 con 0x7f2cfc0a5730 2023-09-30T02:11:36.825 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.824+0000 7f2d01242700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:11:36.825 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.824+0000 7f2d01242700 1 -- 172.21.15.59:0/3465618731 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cf800bd60 con 0x7f2cfc0a5730 2023-09-30T02:11:36.825 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.824+0000 7f2d044a8700 5 monclient: authenticate success, global_id 4897 2023-09-30T02:11:36.825 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.824+0000 7f2d044a8700 10 monclient: get_monmap_and_config success 2023-09-30T02:11:36.825 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.824+0000 7f2d01242700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:36.825 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.824+0000 7f2d044a8700 4 set_mon_vals no callback set 2023-09-30T02:11:36.825 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.824+0000 7f2d01242700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:11:36.825 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.824+0000 7f2d01242700 10 monclient: dump: 2023-09-30T02:11:36.826 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:36.826 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:36.826 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:36.826 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:36.826 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:36.826 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:36.826 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:36.826 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:36.826 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:36.826 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:36.830 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.825+0000 7f2d044a8700 10 monclient: shutdown 2023-09-30T02:11:36.830 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.825+0000 7f2d044a8700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:11:36.830 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.825+0000 7f2d044a8700 1 -- 172.21.15.59:0/3465618731 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2cfc0a5730 msgr2=0x7f2cfc0a7fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:36.830 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.825+0000 7f2d044a8700 1 --2- 172.21.15.59:0/3465618731 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2cfc0a5730 0x7f2cfc0a7fa0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f2cf80099b0 tx=0x7f2cf8009250 comp rx=0 tx=0).stop 2023-09-30T02:11:36.830 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.825+0000 7f2d044a8700 1 -- 172.21.15.59:0/3465618731 shutdown_connections 2023-09-30T02:11:36.830 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.825+0000 7f2d044a8700 1 --2- 172.21.15.59:0/3465618731 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2cfc13b530 0x7f2cfc148f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:36.830 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.825+0000 7f2d044a8700 1 --2- 172.21.15.59:0/3465618731 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2cfc0a5730 0x7f2cfc0a7fa0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:36.830 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.825+0000 7f2d044a8700 1 -- 172.21.15.59:0/3465618731 >> 172.21.15.59:0/3465618731 conn(0x7f2cfc0631f0 msgr2=0x7f2cfc0635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:11:36.830 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.825+0000 7f2d044a8700 1 -- 172.21.15.59:0/3465618731 shutdown_connections 2023-09-30T02:11:36.831 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.825+0000 7f2d044a8700 1 -- 172.21.15.59:0/3465618731 wait complete. 2023-09-30T02:11:36.831 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.825+0000 7f2d044a8700 10 monclient: build_initial_monmap 2023-09-30T02:11:36.831 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.825+0000 7f2d044a8700 10 monclient: monmap: 2023-09-30T02:11:36.831 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:11:36.831 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:11:36.831 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:11:36.831 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:11:36.831 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:11:36.831 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:11:36.831 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:11:36.831 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:11:36.831 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:11:36.832 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:36.832 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.826+0000 7f2d044a8700 1 librados: starting msgr at 2023-09-30T02:11:36.832 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.826+0000 7f2d044a8700 1 librados: starting objecter 2023-09-30T02:11:36.832 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.826+0000 7f2d044a8700 1 Processor -- start 2023-09-30T02:11:36.832 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.826+0000 7f2d044a8700 1 -- start start 2023-09-30T02:11:36.832 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.826+0000 7f2d044a8700 1 librados: setting wanted keys 2023-09-30T02:11:36.832 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.826+0000 7f2d044a8700 1 librados: calling monclient init 2023-09-30T02:11:36.832 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.826+0000 7f2d044a8700 10 monclient: init 2023-09-30T02:11:36.832 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.827+0000 7f2d044a8700 10 monclient: _reopen_session rank -1 2023-09-30T02:11:36.832 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.827+0000 7f2d044a8700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T02:11:36.832 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.827+0000 7f2d044a8700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2cfc0a5730 0x7f2cfc13cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:36.832 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.827+0000 7f2d044a8700 10 monclient(hunting): picked mon.noname-c con 0x7f2cfc0a5730 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:11:36.832 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.827+0000 7f2d044a8700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2cfc13b530 0x7f2cfc13e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:36.833 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.827+0000 7f2d044a8700 10 monclient(hunting): picked mon.noname-a con 0x7f2cfc13b530 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:11:36.833 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.827+0000 7f2d044a8700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2cfc1418e0 0x7f2cfc13edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:36.833 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.827+0000 7f2d044a8700 10 monclient(hunting): picked mon.noname-b con 0x7f2cfc1418e0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:11:36.833 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.827+0000 7f2d044a8700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:36.833 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.827+0000 7f2d044a8700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cfc0abf00 con 0x7f2cfc13b530 2023-09-30T02:11:36.833 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.827+0000 7f2d044a8700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:36.833 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.827+0000 7f2d044a8700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cfc13c710 con 0x7f2cfc1418e0 2023-09-30T02:11:36.833 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.827+0000 7f2d044a8700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:36.833 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.827+0000 7f2d044a8700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cfc13e470 con 0x7f2cfc0a5730 2023-09-30T02:11:36.833 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.827+0000 7f2d044a8700 10 monclient(hunting): _renew_subs 2023-09-30T02:11:36.833 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.827+0000 7f2d044a8700 10 monclient(hunting): authenticate will time out at 2142.129127s 2023-09-30T02:11:36.834 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.827+0000 7f2d02244700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2cfc0a5730 0x7f2cfc13cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:36.834 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.827+0000 7f2d01a43700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2cfc13b530 0x7f2cfc13e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:36.834 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.827+0000 7f2d02244700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2cfc0a5730 0x7f2cfc13cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:33830/0 (socket says 172.21.15.59:33830) 2023-09-30T02:11:36.834 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.827+0000 7f2d02244700 1 -- 172.21.15.59:0/2762402645 learned_addr learned my addr 172.21.15.59:0/2762402645 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:11:36.834 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.827+0000 7f2d02244700 10 monclient(hunting): get_auth_request con 0x7f2cfc0a5730 auth_method 0 2023-09-30T02:11:36.834 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.827+0000 7f2d02244700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:36.834 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.827+0000 7f2d02244700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:36.834 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.827+0000 7f2d02244700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:36.834 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.828+0000 7f2d01a43700 10 monclient(hunting): get_auth_request con 0x7f2cfc13b530 auth_method 0 2023-09-30T02:11:36.834 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.828+0000 7f2d01a43700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:36.834 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.828+0000 7f2d01a43700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:36.834 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.828+0000 7f2d01a43700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:36.835 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.828+0000 7f2d02244700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:36.835 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.828+0000 7f2d02244700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:36.835 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.828+0000 7f2d02244700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:36.835 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.828+0000 7f2d01a43700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:36.835 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.828+0000 7f2d01a43700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:36.835 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.828+0000 7f2d01a43700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:36.835 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.829+0000 7f2d02a45700 1 --2- 172.21.15.59:0/2762402645 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2cfc1418e0 0x7f2cfc13edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:36.835 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.829+0000 7f2d02a45700 10 monclient(hunting): get_auth_request con 0x7f2cfc1418e0 auth_method 0 2023-09-30T02:11:36.835 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.829+0000 7f2d02a45700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:36.835 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.829+0000 7f2d02a45700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:36.835 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.829+0000 7f2d02a45700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:36.835 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.829+0000 7f2d02a45700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:36.836 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.829+0000 7f2d02a45700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:36.836 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.829+0000 7f2d02a45700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:36.837 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.832+0000 7f2d02244700 10 monclient(hunting): handle_auth_done global_id 4769 payload 824 2023-09-30T02:11:36.837 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.832+0000 7f2d02244700 1 -- 172.21.15.59:0/2762402645 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2cfc1418e0 msgr2=0x7f2cfc13edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:36.837 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.832+0000 7f2d02244700 1 --2- 172.21.15.59:0/2762402645 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2cfc1418e0 0x7f2cfc13edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:36.837 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.832+0000 7f2d02244700 1 -- 172.21.15.59:0/2762402645 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2cfc13b530 msgr2=0x7f2cfc13e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:36.837 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.832+0000 7f2d02244700 1 --2- 172.21.15.59:0/2762402645 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2cfc13b530 0x7f2cfc13e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:36.837 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.832+0000 7f2d02244700 10 monclient: _finish_hunting 0 2023-09-30T02:11:36.837 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.832+0000 7f2d02244700 1 monclient: found mon.noname-c 2023-09-30T02:11:36.837 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.832+0000 7f2d02244700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:11:36.837 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.832+0000 7f2d02244700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.59:3301/0 2023-09-30T02:11:36.837 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.832+0000 7f2d02244700 1 -- 172.21.15.59:0/2762402645 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cf80094f0 con 0x7f2cfc0a5730 2023-09-30T02:11:36.838 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.835+0000 7f2d02244700 1 --2- 172.21.15.59:0/2762402645 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2cfc0a5730 0x7f2cfc13cb90 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f2cf80090b0 tx=0x7f2cf8017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:36.838 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.835+0000 7f2cf27fc700 10 client.?.objecter ms_handle_connect 0x7f2cfc0a5730 2023-09-30T02:11:36.838 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.835+0000 7f2cf27fc700 10 client.?.objecter resend_mon_ops 2023-09-30T02:11:36.840 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.840+0000 7f2cf27fc700 1 -- 172.21.15.59:0/2762402645 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cf801a450 con 0x7f2cfc0a5730 2023-09-30T02:11:36.840 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.840+0000 7f2cf27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:36.840 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.840+0000 7f2cf27fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:11:36.840 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.840+0000 7f2cf27fc700 10 monclient: dump: 2023-09-30T02:11:36.840 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:36.840 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:36.840 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:36.840 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:36.840 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:36.840 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:36.841 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:36.841 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:36.841 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:36.841 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:36.841 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.840+0000 7f2cf27fc700 10 monclient: _finish_auth 0 2023-09-30T02:11:36.841 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.840+0000 7f2cf27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:11:36.841 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.840+0000 7f2cf27fc700 1 -- 172.21.15.59:0/2762402645 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cf801a5b0 con 0x7f2cfc0a5730 2023-09-30T02:11:36.841 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.840+0000 7f2cf27fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:11:36.841 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.840+0000 7f2cf27fc700 1 -- 172.21.15.59:0/2762402645 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cf801a970 con 0x7f2cfc0a5730 2023-09-30T02:11:36.841 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.840+0000 7f2d044a8700 5 monclient: authenticate success, global_id 4769 2023-09-30T02:11:36.841 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.840+0000 7f2d044a8700 10 monclient: _renew_subs 2023-09-30T02:11:36.841 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.840+0000 7f2d044a8700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:11:36.842 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.840+0000 7f2d044a8700 1 -- 172.21.15.59:0/2762402645 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cfc13c1f0 con 0x7f2cfc0a5730 2023-09-30T02:11:36.842 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.840+0000 7f2cf27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:36.842 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.840+0000 7f2d00a41700 4 set_mon_vals no callback set 2023-09-30T02:11:36.842 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.840+0000 7f2cf27fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:11:36.842 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.840+0000 7f2cf27fc700 10 monclient: dump: 2023-09-30T02:11:36.842 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:36.842 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:36.842 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:36.842 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:36.842 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:36.842 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:36.842 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:36.843 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:36.843 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:36.843 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:36.843 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.840+0000 7f2d044a8700 10 client.4769.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:11:36.843 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.840+0000 7f2d044a8700 10 monclient: _renew_subs 2023-09-30T02:11:36.843 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.840+0000 7f2d044a8700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:11:36.843 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.840+0000 7f2d044a8700 1 -- 172.21.15.59:0/2762402645 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cfc13d310 con 0x7f2cfc0a5730 2023-09-30T02:11:36.843 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.840+0000 7f2d044a8700 1 librados: init done 2023-09-30T02:11:36.843 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.841+0000 7f2d044a8700 10 client.4769.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:11:36.843 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.841+0000 7f2d044a8700 15 client.4769.objecter _session_command_op_assign -1 1 2023-09-30T02:11:36.843 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.841+0000 7f2d044a8700 20 client.4769.objecter _get_session osd=-1 returning homeless 2023-09-30T02:11:36.844 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.841+0000 7f2d044a8700 10 client.4769.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:11:36.844 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.841+0000 7f2d044a8700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:11:36.844 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.841+0000 7f2d044a8700 1 -- 172.21.15.59:0/2762402645 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2cfc064a00 con 0x7f2cfc0a5730 2023-09-30T02:11:36.868 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.867+0000 7f2cf27fc700 1 -- 172.21.15.59:0/2762402645 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f2cf801ad20 con 0x7f2cfc0a5730 2023-09-30T02:11:36.868 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.867+0000 7f2cf27fc700 1 --2- 172.21.15.59:0/2762402645 >> v2:172.21.15.110:6800/115870 conn(0x7f2ce0068910 0x7f2ce006adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:36.868 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.868+0000 7f2cf27fc700 1 -- 172.21.15.59:0/2762402645 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f2cf8085a50 con 0x7f2cfc0a5730 2023-09-30T02:11:36.868 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.868+0000 7f2cf27fc700 10 client.4769.objecter ms_dispatch 0x7f2cfc001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:11:36.868 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.868+0000 7f2cf27fc700 3 client.4769.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:11:36.869 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.868+0000 7f2cf27fc700 3 client.4769.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:11:36.869 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.868+0000 7f2cf27fc700 10 client.4769.objecter checking command 1 2023-09-30T02:11:36.869 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.868+0000 7f2cf27fc700 1 --2- 172.21.15.59:0/2762402645 >> v2:172.21.15.59:6804/116666 conn(0x7f2ce006e880 0x7f2ce0070c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:36.869 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.868+0000 7f2cf27fc700 20 client.4769.objecter _get_session s=0x7f2ce006e1d0 osd=2 3 2023-09-30T02:11:36.869 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.868+0000 7f2cf27fc700 20 client.4769.objecter put_session s=0x7f2ce006e1d0 osd=2 3 2023-09-30T02:11:36.869 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.868+0000 7f2cf27fc700 15 client.4769.objecter _session_command_op_remove -1 1 2023-09-30T02:11:36.869 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.868+0000 7f2cf27fc700 20 client.4769.objecter _get_session s=0x7f2ce006e1d0 osd=2 3 2023-09-30T02:11:36.869 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.868+0000 7f2cf27fc700 20 client.4769.objecter get_session s=0x7f2ce006e1d0 osd=2 3 2023-09-30T02:11:36.870 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.868+0000 7f2cf27fc700 15 client.4769.objecter _session_command_op_assign 2 1 2023-09-30T02:11:36.870 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.868+0000 7f2cf27fc700 20 client.4769.objecter put_session s=0x7f2ce006e1d0 osd=2 4 2023-09-30T02:11:36.870 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.868+0000 7f2cf27fc700 10 client.4769.objecter _send_command 1 2023-09-30T02:11:36.870 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.868+0000 7f2cf27fc700 1 -- 172.21.15.59:0/2762402645 --> v2:172.21.15.59:6804/116666 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2ce0071300 con 0x7f2ce006e880 2023-09-30T02:11:36.870 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.868+0000 7f2cf27fc700 20 client.4769.objecter dump_active .. 0 homeless 2023-09-30T02:11:36.870 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.868+0000 7f2cf27fc700 1 -- 172.21.15.59:0/2762402645 <== mon.2 v2:172.21.15.59:3301/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2cf8085d80 con 0x7f2cfc0a5730 2023-09-30T02:11:36.870 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.868+0000 7f2cf27fc700 10 monclient: handle_get_version_reply finishing 1 version 25 2023-09-30T02:11:36.870 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.868+0000 7f2d01a43700 1 --2- 172.21.15.59:0/2762402645 >> v2:172.21.15.110:6800/115870 conn(0x7f2ce0068910 0x7f2ce006adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:36.870 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.868+0000 7f2d01a43700 10 monclient: get_auth_request con 0x7f2ce0068910 auth_method 0 2023-09-30T02:11:36.870 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.868+0000 7f2cf27fc700 10 client.4769.objecter ms_handle_connect 0x7f2ce0068910 2023-09-30T02:11:36.870 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.868+0000 7f2d01a43700 1 --2- 172.21.15.59:0/2762402645 >> v2:172.21.15.110:6800/115870 conn(0x7f2ce0068910 0x7f2ce006adc0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc0ae800 tx=0x7f2cec007ec0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:36.871 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.869+0000 7f2d02a45700 1 --2- 172.21.15.59:0/2762402645 >> v2:172.21.15.59:6804/116666 conn(0x7f2ce006e880 0x7f2ce0070c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:36.871 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.869+0000 7f2d02a45700 10 monclient: get_auth_request con 0x7f2ce006e880 auth_method 0 2023-09-30T02:11:36.872 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.872+0000 7f2d02a45700 1 --2- 172.21.15.59:0/2762402645 >> v2:172.21.15.59:6804/116666 conn(0x7f2ce006e880 0x7f2ce0070c90 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:36.872 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.872+0000 7f2cf27fc700 10 client.4769.objecter ms_handle_connect 0x7f2ce006e880 2023-09-30T02:11:36.897 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.897+0000 7f2cf27fc700 1 -- 172.21.15.59:0/2762402645 <== osd.2 v2:172.21.15.59:6804/116666 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f2ce0071300 con 0x7f2ce006e880 2023-09-30T02:11:36.897 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.897+0000 7f2cf27fc700 10 client.4769.objecter ms_dispatch 0x7f2cfc001550 command_reply(tid 1: 0 ) v1 2023-09-30T02:11:36.897 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.897+0000 7f2cf27fc700 10 client.4769.objecter _finish_command 1 = system:0 2023-09-30T02:11:36.897 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.897+0000 7f2cf27fc700 20 client.4769.objecter put_session s=0x7f2ce006e1d0 osd=2 4 2023-09-30T02:11:36.897 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.897+0000 7f2cf27fc700 15 client.4769.objecter _session_command_op_remove 2 1 2023-09-30T02:11:36.911 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.912+0000 7f2d044a8700 10 client.4769.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-09-30T02:11:36.912 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.912+0000 7f2d044a8700 15 client.4769.objecter _session_command_op_assign -1 2 2023-09-30T02:11:36.912 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.912+0000 7f2d044a8700 20 client.4769.objecter _get_session s=0x7f2ce006e1d0 osd=2 3 2023-09-30T02:11:36.912 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.912+0000 7f2d044a8700 20 client.4769.objecter put_session s=0x7f2ce006e1d0 osd=2 3 2023-09-30T02:11:36.912 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.912+0000 7f2d044a8700 20 client.4769.objecter _get_session s=0x7f2ce006e1d0 osd=2 3 2023-09-30T02:11:36.912 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.912+0000 7f2d044a8700 15 client.4769.objecter _session_command_op_remove -1 2 2023-09-30T02:11:36.912 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.912+0000 7f2d044a8700 20 client.4769.objecter get_session s=0x7f2ce006e1d0 osd=2 3 2023-09-30T02:11:36.912 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.912+0000 7f2d044a8700 15 client.4769.objecter _session_command_op_assign 2 2 2023-09-30T02:11:36.912 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.912+0000 7f2d044a8700 20 client.4769.objecter put_session s=0x7f2ce006e1d0 osd=2 4 2023-09-30T02:11:36.912 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.912+0000 7f2d044a8700 10 client.4769.objecter _send_command 2 2023-09-30T02:11:36.912 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.912+0000 7f2d044a8700 1 -- 172.21.15.59:0/2762402645 --> v2:172.21.15.59:6804/116666 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f2cfc0a7050 con 0x7f2ce006e880 2023-09-30T02:11:36.935 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.936+0000 7f2cf27fc700 1 -- 172.21.15.59:0/2762402645 <== osd.2 v2:172.21.15.59:6804/116666 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f2cfc0a7050 con 0x7f2ce006e880 2023-09-30T02:11:36.935 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.936+0000 7f2cf27fc700 10 client.4769.objecter ms_dispatch 0x7f2cfc001550 command_reply(tid 2: 0 ) v1 2023-09-30T02:11:36.936 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.936+0000 7f2cf27fc700 10 client.4769.objecter _finish_command 2 = system:0 2023-09-30T02:11:36.936 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.936+0000 7f2cf27fc700 20 client.4769.objecter put_session s=0x7f2ce006e1d0 osd=2 4 2023-09-30T02:11:36.936 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.936+0000 7f2cf27fc700 15 client.4769.objecter _session_command_op_remove 2 2 2023-09-30T02:11:36.936 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2023-09-30T02:11:36.936 INFO:teuthology.orchestra.run.smithi059.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T02:11:36.937 INFO:teuthology.orchestra.run.smithi059.stdout:} 2023-09-30T02:11:36.938 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.936+0000 7f2d044a8700 10 librados: watch_flush enter 2023-09-30T02:11:36.938 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.936+0000 7f2d044a8700 10 librados: watch_flush exit 2023-09-30T02:11:36.938 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.936+0000 7f2d044a8700 10 client.4769.objecter close_session for osd.2 2023-09-30T02:11:36.939 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.936+0000 7f2d044a8700 1 -- 172.21.15.59:0/2762402645 >> v2:172.21.15.59:6804/116666 conn(0x7f2ce006e880 msgr2=0x7f2ce0070c90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:36.939 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.936+0000 7f2d044a8700 1 --2- 172.21.15.59:0/2762402645 >> v2:172.21.15.59:6804/116666 conn(0x7f2ce006e880 0x7f2ce0070c90 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:36.939 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.936+0000 7f2d044a8700 20 client.4769.objecter put_session s=0x7f2ce006e1d0 osd=2 1 2023-09-30T02:11:36.939 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.936+0000 7f2d044a8700 20 client.4769.objecter shutdown clearing up homeless session... 2023-09-30T02:11:36.939 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.936+0000 7f2d044a8700 10 client.4769.objecter successfully canceled tick 2023-09-30T02:11:36.939 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.936+0000 7f2d044a8700 1 -- 172.21.15.59:0/2762402645 >> v2:172.21.15.110:6800/115870 conn(0x7f2ce0068910 msgr2=0x7f2ce006adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:36.939 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.936+0000 7f2d044a8700 1 --2- 172.21.15.59:0/2762402645 >> v2:172.21.15.110:6800/115870 conn(0x7f2ce0068910 0x7f2ce006adc0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc0ae800 tx=0x7f2cec007ec0 comp rx=0 tx=0).stop 2023-09-30T02:11:36.939 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.936+0000 7f2d044a8700 10 monclient: shutdown 2023-09-30T02:11:36.939 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.936+0000 7f2d044a8700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:11:36.940 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.936+0000 7f2d044a8700 1 -- 172.21.15.59:0/2762402645 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2cfc0a5730 msgr2=0x7f2cfc13cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:36.940 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.936+0000 7f2d044a8700 1 --2- 172.21.15.59:0/2762402645 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2cfc0a5730 0x7f2cfc13cb90 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f2cf80090b0 tx=0x7f2cf8017000 comp rx=0 tx=0).stop 2023-09-30T02:11:36.940 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.936+0000 7f2d02a45700 1 -- 172.21.15.59:0/2762402645 reap_dead start 2023-09-30T02:11:36.940 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.936+0000 7f2d044a8700 1 -- 172.21.15.59:0/2762402645 shutdown_connections 2023-09-30T02:11:36.940 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.936+0000 7f2d044a8700 1 -- 172.21.15.59:0/2762402645 >> 172.21.15.59:0/2762402645 conn(0x7f2cfc0631f0 msgr2=0x7f2cfc070f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:11:36.940 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.936+0000 7f2d044a8700 1 -- 172.21.15.59:0/2762402645 shutdown_connections 2023-09-30T02:11:36.940 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.936+0000 7f2d044a8700 1 -- 172.21.15.59:0/2762402645 wait complete. 2023-09-30T02:11:36.940 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:36.937+0000 7f2d044a8700 1 librados: shutdown 2023-09-30T02:11:36.949 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-09-30T02:11:37.083 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.083+0000 7fa9cbad9700 10 monclient: get_monmap_and_config 2023-09-30T02:11:37.083 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.083+0000 7fa9cbad9700 10 monclient: build_initial_monmap 2023-09-30T02:11:37.083 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.083+0000 7fa9cbad9700 10 monclient: monmap: 2023-09-30T02:11:37.083 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:11:37.083 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:37.083 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:11:37.084037+0000 2023-09-30T02:11:37.083 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:11:37.084037+0000 2023-09-30T02:11:37.083 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:11:37.084 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:37.084 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:11:37.084 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:11:37.084 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:11:37.084 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:11:37.084 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:11:37.084 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:37.086 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.084+0000 7fa9cbad9700 1 Processor -- start 2023-09-30T02:11:37.086 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.084+0000 7fa9cbad9700 1 -- start start 2023-09-30T02:11:37.086 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.084+0000 7fa9cbad9700 10 monclient: init 2023-09-30T02:11:37.086 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.085+0000 7fa9cbad9700 10 monclient: _reopen_session rank -1 2023-09-30T02:11:37.087 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.085+0000 7fa9cbad9700 10 monclient: _add_conns ranks=[4,3,2,1,0] 2023-09-30T02:11:37.087 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.085+0000 7fa9cbad9700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fa9c40a5b70 0x7fa9c40b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:37.087 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.085+0000 7fa9cbad9700 10 monclient(hunting): picked mon.noname-c con 0x7fa9c40a5b70 addr v2:172.21.15.110:3300/0 2023-09-30T02:11:37.087 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.085+0000 7fa9cbad9700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa9c40a5f40 0x7fa9c4141b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:37.087 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.085+0000 7fa9cbad9700 10 monclient(hunting): picked mon.noname-a con 0x7fa9c40a5f40 addr v2:172.21.15.59:3300/0 2023-09-30T02:11:37.087 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.085+0000 7fa9cbad9700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa9c41420c0 0x7fa9c413e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:37.087 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.085+0000 7fa9cbad9700 10 monclient(hunting): picked mon.noname-b con 0x7fa9c41420c0 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:11:37.087 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.085+0000 7fa9cbad9700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:37.087 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.085+0000 7fa9cbad9700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9c40a63e0 con 0x7fa9c40a5f40 2023-09-30T02:11:37.087 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.085+0000 7fa9cbad9700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:37.087 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.085+0000 7fa9cbad9700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9c40a75a0 con 0x7fa9c40a5b70 2023-09-30T02:11:37.088 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.085+0000 7fa9cbad9700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:37.088 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.085+0000 7fa9cbad9700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c40af3f0 con 0x7fa9c41420c0 2023-09-30T02:11:37.088 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.085+0000 7fa9cbad9700 10 monclient(hunting): _renew_subs 2023-09-30T02:11:37.088 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.085+0000 7fa9cbad9700 10 monclient(hunting): authenticate will time out at 2142.387209s 2023-09-30T02:11:37.088 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.085+0000 7fa9ca076700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa9c41420c0 0x7fa9c413e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:37.088 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.086+0000 7fa9c9875700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fa9c40a5b70 0x7fa9c40b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:37.088 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.086+0000 7fa9c9074700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fa9c40a5f40 0x7fa9c4141b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:37.088 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.086+0000 7fa9c9875700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fa9c40a5b70 0x7fa9c40b1500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:60596/0 (socket says 172.21.15.59:60596) 2023-09-30T02:11:37.088 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.086+0000 7fa9c9875700 1 -- 172.21.15.59:0/2897130492 learned_addr learned my addr 172.21.15.59:0/2897130492 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:11:37.088 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.086+0000 7fa9c9875700 10 monclient(hunting): get_auth_request con 0x7fa9c40a5b70 auth_method 0 2023-09-30T02:11:37.088 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.086+0000 7fa9c9875700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:37.088 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.086+0000 7fa9c9875700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:37.089 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.086+0000 7fa9c9875700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:37.089 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.086+0000 7fa9ca076700 10 monclient(hunting): get_auth_request con 0x7fa9c41420c0 auth_method 0 2023-09-30T02:11:37.089 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.086+0000 7fa9ca076700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:37.089 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.086+0000 7fa9ca076700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:37.089 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.086+0000 7fa9ca076700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:37.089 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.086+0000 7fa9c9074700 10 monclient(hunting): get_auth_request con 0x7fa9c40a5f40 auth_method 0 2023-09-30T02:11:37.089 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.086+0000 7fa9c9074700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:37.089 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.086+0000 7fa9c9074700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:37.089 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.086+0000 7fa9c9074700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:37.089 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.086+0000 7fa9c9875700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:37.089 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.086+0000 7fa9c9875700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:37.089 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.086+0000 7fa9c9875700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:37.090 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.087+0000 7fa9ca076700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:37.090 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.087+0000 7fa9ca076700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:37.090 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.087+0000 7fa9ca076700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:37.090 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.087+0000 7fa9c9074700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:37.090 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.087+0000 7fa9c9074700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:37.090 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.087+0000 7fa9c9074700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:37.091 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.088+0000 7fa9c9875700 10 monclient(hunting): handle_auth_done global_id 4903 payload 274 2023-09-30T02:11:37.091 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.088+0000 7fa9c9875700 1 -- 172.21.15.59:0/2897130492 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa9c41420c0 msgr2=0x7fa9c413e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:11:37.091 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.088+0000 7fa9c9875700 1 --2- 172.21.15.59:0/2897130492 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa9c41420c0 0x7fa9c413e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:37.091 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.088+0000 7fa9c9875700 1 -- 172.21.15.59:0/2897130492 >> v2:172.21.15.59:3300/0 conn(0x7fa9c40a5f40 msgr2=0x7fa9c4141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:11:37.091 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.088+0000 7fa9c9875700 1 --2- 172.21.15.59:0/2897130492 >> v2:172.21.15.59:3300/0 conn(0x7fa9c40a5f40 0x7fa9c4141b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:37.091 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.088+0000 7fa9c9875700 10 monclient: _finish_hunting 0 2023-09-30T02:11:37.091 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.088+0000 7fa9c9875700 1 monclient: found mon.noname-c 2023-09-30T02:11:37.092 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.088+0000 7fa9c9875700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:11:37.092 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.088+0000 7fa9c9875700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3300/0 2023-09-30T02:11:37.092 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.088+0000 7fa9c9875700 1 -- 172.21.15.59:0/2897130492 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9c4142500 con 0x7fa9c40a5b70 2023-09-30T02:11:37.092 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.090+0000 7fa9c9875700 1 --2- 172.21.15.59:0/2897130492 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa9c40a5b70 0x7fa9c40b1500 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fa9c40b49e0 tx=0x7fa9c0009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3b411fde7422d1e5 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:37.093 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.094+0000 7fa9c8873700 1 -- 172.21.15.59:0/2897130492 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9c000b640 con 0x7fa9c40a5b70 2023-09-30T02:11:37.094 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.094+0000 7fa9c8873700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:37.094 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.094+0000 7fa9c8873700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:11:37.094 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.094+0000 7fa9c8873700 10 monclient: dump: 2023-09-30T02:11:37.094 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:37.094 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:37.094 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:37.094 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:37.094 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:37.094 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:37.094 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:37.095 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:37.095 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:37.095 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:37.095 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.094+0000 7fa9c8873700 10 monclient: _finish_auth 0 2023-09-30T02:11:37.095 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.094+0000 7fa9c8873700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:11:37.095 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.094+0000 7fa9c8873700 1 -- 172.21.15.59:0/2897130492 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9c000b7a0 con 0x7fa9c40a5b70 2023-09-30T02:11:37.095 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.094+0000 7fa9cbad9700 5 monclient: authenticate success, global_id 4903 2023-09-30T02:11:37.095 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.094+0000 7fa9c8873700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:11:37.095 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.094+0000 7fa9cbad9700 10 monclient: get_monmap_and_config success 2023-09-30T02:11:37.095 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.094+0000 7fa9c8873700 1 -- 172.21.15.59:0/2897130492 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9c000bb60 con 0x7fa9c40a5b70 2023-09-30T02:11:37.096 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.094+0000 7fa9cbad9700 4 set_mon_vals no callback set 2023-09-30T02:11:37.096 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.094+0000 7fa9c8873700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:37.096 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.094+0000 7fa9c8873700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:11:37.096 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.094+0000 7fa9c8873700 10 monclient: dump: 2023-09-30T02:11:37.096 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:37.096 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:37.096 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:37.096 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:37.096 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:37.096 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:37.096 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:37.097 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:37.097 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:37.097 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:37.100 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.095+0000 7fa9cbad9700 10 monclient: shutdown 2023-09-30T02:11:37.100 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.095+0000 7fa9cbad9700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:11:37.100 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.095+0000 7fa9cbad9700 1 -- 172.21.15.59:0/2897130492 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa9c40a5b70 msgr2=0x7fa9c40b1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:37.100 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.095+0000 7fa9cbad9700 1 --2- 172.21.15.59:0/2897130492 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa9c40a5b70 0x7fa9c40b1500 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fa9c40b49e0 tx=0x7fa9c0009250 comp rx=0 tx=0).stop 2023-09-30T02:11:37.100 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.095+0000 7fa9cbad9700 1 -- 172.21.15.59:0/2897130492 shutdown_connections 2023-09-30T02:11:37.100 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.095+0000 7fa9cbad9700 1 --2- 172.21.15.59:0/2897130492 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa9c41420c0 0x7fa9c413e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:37.100 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.095+0000 7fa9cbad9700 1 --2- 172.21.15.59:0/2897130492 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa9c40a5b70 0x7fa9c40b1500 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:37.101 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.095+0000 7fa9cbad9700 1 --2- 172.21.15.59:0/2897130492 >> v2:172.21.15.59:3300/0 conn(0x7fa9c40a5f40 0x7fa9c4141b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:37.101 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.095+0000 7fa9cbad9700 1 -- 172.21.15.59:0/2897130492 >> 172.21.15.59:0/2897130492 conn(0x7fa9c40631f0 msgr2=0x7fa9c40635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:11:37.101 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.095+0000 7fa9cbad9700 1 -- 172.21.15.59:0/2897130492 shutdown_connections 2023-09-30T02:11:37.101 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.095+0000 7fa9cbad9700 1 -- 172.21.15.59:0/2897130492 wait complete. 2023-09-30T02:11:37.101 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.096+0000 7fa9cbad9700 10 monclient: build_initial_monmap 2023-09-30T02:11:37.101 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.096+0000 7fa9cbad9700 10 monclient: monmap: 2023-09-30T02:11:37.101 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:11:37.101 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:11:37.101 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:11:37.101 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:11:37.101 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:11:37.101 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:11:37.102 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:11:37.102 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:11:37.102 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:11:37.102 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:37.102 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.096+0000 7fa9cbad9700 1 librados: starting msgr at 2023-09-30T02:11:37.102 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.096+0000 7fa9cbad9700 1 librados: starting objecter 2023-09-30T02:11:37.102 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.096+0000 7fa9cbad9700 1 Processor -- start 2023-09-30T02:11:37.102 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.096+0000 7fa9cbad9700 1 -- start start 2023-09-30T02:11:37.102 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.096+0000 7fa9cbad9700 1 librados: setting wanted keys 2023-09-30T02:11:37.102 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.096+0000 7fa9cbad9700 1 librados: calling monclient init 2023-09-30T02:11:37.102 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.096+0000 7fa9cbad9700 10 monclient: init 2023-09-30T02:11:37.103 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.097+0000 7fa9cbad9700 10 monclient: _reopen_session rank -1 2023-09-30T02:11:37.103 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.097+0000 7fa9cbad9700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T02:11:37.103 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.097+0000 7fa9cbad9700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa9c40a5b70 0x7fa9c40a8800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:37.103 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.097+0000 7fa9cbad9700 10 monclient(hunting): picked mon.noname-b con 0x7fa9c40a5b70 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:11:37.103 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.097+0000 7fa9cbad9700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa9c40a5f40 0x7fa9c40a8d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:37.103 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.097+0000 7fa9cbad9700 10 monclient(hunting): picked mon.noname-c con 0x7fa9c40a5f40 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:11:37.103 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.097+0000 7fa9cbad9700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9c41420c0 0x7fa9c41399b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:37.103 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.097+0000 7fa9cbad9700 10 monclient(hunting): picked mon.noname-a con 0x7fa9c41420c0 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:11:37.103 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.097+0000 7fa9cbad9700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:37.103 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.097+0000 7fa9cbad9700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c40a7e00 con 0x7fa9c41420c0 2023-09-30T02:11:37.103 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.097+0000 7fa9cbad9700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:37.104 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.097+0000 7fa9cbad9700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c40a7ad0 con 0x7fa9c40a5b70 2023-09-30T02:11:37.104 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.097+0000 7fa9cbad9700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:37.104 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.097+0000 7fa9cbad9700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c40aa130 con 0x7fa9c40a5f40 2023-09-30T02:11:37.104 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.097+0000 7fa9cbad9700 10 monclient(hunting): _renew_subs 2023-09-30T02:11:37.104 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.097+0000 7fa9cbad9700 10 monclient(hunting): authenticate will time out at 2142.399301s 2023-09-30T02:11:37.104 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.097+0000 7fa9c9875700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa9c40a5b70 0x7fa9c40a8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:37.104 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.097+0000 7fa9c9074700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa9c40a5f40 0x7fa9c40a8d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:37.104 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.098+0000 7fa9ca076700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9c41420c0 0x7fa9c41399b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:37.104 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.098+0000 7fa9c9875700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa9c40a5b70 0x7fa9c40a8800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:60604/0 (socket says 172.21.15.59:60604) 2023-09-30T02:11:37.104 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.098+0000 7fa9c9875700 1 -- 172.21.15.59:0/3777488120 learned_addr learned my addr 172.21.15.59:0/3777488120 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:11:37.104 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.098+0000 7fa9c9074700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa9c40a5f40 0x7fa9c40a8d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:33838/0 (socket says 172.21.15.59:33838) 2023-09-30T02:11:37.105 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.098+0000 7fa9c9875700 10 monclient(hunting): get_auth_request con 0x7fa9c40a5b70 auth_method 0 2023-09-30T02:11:37.105 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.098+0000 7fa9c9875700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:37.105 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.098+0000 7fa9c9875700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:37.105 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.098+0000 7fa9c9875700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:37.105 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.098+0000 7fa9c9074700 10 monclient(hunting): get_auth_request con 0x7fa9c40a5f40 auth_method 0 2023-09-30T02:11:37.105 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.098+0000 7fa9c9074700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:37.105 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.098+0000 7fa9c9074700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:37.105 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.098+0000 7fa9c9074700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:37.105 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.098+0000 7fa9ca076700 10 monclient(hunting): get_auth_request con 0x7fa9c41420c0 auth_method 0 2023-09-30T02:11:37.105 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.098+0000 7fa9ca076700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:37.105 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.098+0000 7fa9ca076700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:37.105 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.098+0000 7fa9ca076700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:37.106 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.098+0000 7fa9c9875700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:37.106 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.098+0000 7fa9c9875700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:37.106 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.098+0000 7fa9c9875700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:37.106 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.098+0000 7fa9c9074700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:37.106 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.098+0000 7fa9c9074700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:37.106 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.099+0000 7fa9c9074700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:37.106 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.099+0000 7fa9ca076700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:37.106 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.099+0000 7fa9ca076700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:37.106 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.099+0000 7fa9ca076700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:37.108 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.101+0000 7fa9c9875700 10 monclient(hunting): handle_auth_done global_id 4906 payload 824 2023-09-30T02:11:37.108 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.101+0000 7fa9c9875700 1 -- 172.21.15.59:0/3777488120 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa9c40a5f40 msgr2=0x7fa9c40a8d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:37.108 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.101+0000 7fa9c9875700 1 --2- 172.21.15.59:0/3777488120 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa9c40a5f40 0x7fa9c40a8d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:37.109 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.101+0000 7fa9c9875700 1 -- 172.21.15.59:0/3777488120 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9c41420c0 msgr2=0x7fa9c41399b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:37.109 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.101+0000 7fa9c9875700 1 --2- 172.21.15.59:0/3777488120 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9c41420c0 0x7fa9c41399b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:37.109 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.101+0000 7fa9c9875700 10 monclient: _finish_hunting 0 2023-09-30T02:11:37.109 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.101+0000 7fa9c9875700 1 monclient: found mon.noname-b 2023-09-30T02:11:37.109 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.101+0000 7fa9c9875700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:11:37.109 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.101+0000 7fa9c9875700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:11:37.109 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.101+0000 7fa9c9875700 1 -- 172.21.15.59:0/3777488120 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9c00094f0 con 0x7fa9c40a5b70 2023-09-30T02:11:37.109 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.104+0000 7fa9c9875700 1 --2- 172.21.15.59:0/3777488120 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa9c40a5b70 0x7fa9c40a8800 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fa9c40a76e0 tx=0x7fa9c0017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:37.109 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.104+0000 7fa9b9ffb700 10 client.?.objecter ms_handle_connect 0x7fa9c40a5b70 2023-09-30T02:11:37.109 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.104+0000 7fa9b9ffb700 10 client.?.objecter resend_mon_ops 2023-09-30T02:11:37.109 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.107+0000 7fa9b9ffb700 1 -- 172.21.15.59:0/3777488120 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9c000b910 con 0x7fa9c40a5b70 2023-09-30T02:11:37.109 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.107+0000 7fa9b9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:37.110 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.107+0000 7fa9b9ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:11:37.110 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.107+0000 7fa9b9ffb700 10 monclient: dump: 2023-09-30T02:11:37.110 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:37.110 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:37.110 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:37.110 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:37.110 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:37.110 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:37.110 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:37.110 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:37.110 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:37.110 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:37.111 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.107+0000 7fa9b9ffb700 10 monclient: _finish_auth 0 2023-09-30T02:11:37.111 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.107+0000 7fa9b9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:11:37.111 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.107+0000 7fa9b9ffb700 1 -- 172.21.15.59:0/3777488120 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9c000ba70 con 0x7fa9c40a5b70 2023-09-30T02:11:37.111 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.107+0000 7fa9cbad9700 5 monclient: authenticate success, global_id 4906 2023-09-30T02:11:37.111 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.107+0000 7fa9cbad9700 10 monclient: _renew_subs 2023-09-30T02:11:37.111 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.107+0000 7fa9cbad9700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:11:37.111 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.107+0000 7fa9cbad9700 1 -- 172.21.15.59:0/3777488120 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9c413b570 con 0x7fa9c40a5b70 2023-09-30T02:11:37.111 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.107+0000 7fa9b9ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:11:37.111 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.107+0000 7fa9b9ffb700 1 -- 172.21.15.59:0/3777488120 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9c0011630 con 0x7fa9c40a5b70 2023-09-30T02:11:37.111 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.107+0000 7fa9b9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:37.111 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.107+0000 7fa9b9ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:11:37.111 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.107+0000 7fa9b9ffb700 10 monclient: dump: 2023-09-30T02:11:37.112 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:37.112 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:37.112 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:37.112 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:37.112 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:37.112 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:37.112 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:37.112 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:37.112 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:37.112 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:37.112 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.107+0000 7fa9bbfff700 4 set_mon_vals no callback set 2023-09-30T02:11:37.112 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.108+0000 7fa9cbad9700 10 client.4906.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:11:37.113 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.108+0000 7fa9cbad9700 10 monclient: _renew_subs 2023-09-30T02:11:37.113 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.108+0000 7fa9cbad9700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:11:37.113 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.108+0000 7fa9cbad9700 1 -- 172.21.15.59:0/3777488120 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9c413a180 con 0x7fa9c40a5b70 2023-09-30T02:11:37.113 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.108+0000 7fa9cbad9700 1 librados: init done 2023-09-30T02:11:37.113 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.109+0000 7fa9cbad9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:11:37.113 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.109+0000 7fa9cbad9700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:11:37.113 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.109+0000 7fa9cbad9700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:11:37.113 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.109+0000 7fa9cbad9700 1 -- 172.21.15.59:0/3777488120 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9c4064a00 con 0x7fa9c40a5b70 2023-09-30T02:11:37.356 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.357+0000 7fa9b9ffb700 1 -- 172.21.15.59:0/3777488120 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fa9c00119a0 con 0x7fa9c40a5b70 2023-09-30T02:11:37.357 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.357+0000 7fa9b9ffb700 1 --2- 172.21.15.59:0/3777488120 >> v2:172.21.15.110:6800/115870 conn(0x7fa9a8068910 0x7fa9a806adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:37.357 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.357+0000 7fa9b9ffb700 1 -- 172.21.15.59:0/3777488120 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fa9c0085650 con 0x7fa9c40a5b70 2023-09-30T02:11:37.357 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.357+0000 7fa9b9ffb700 10 client.4906.objecter ms_dispatch 0x7fa9c4001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:11:37.357 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.357+0000 7fa9b9ffb700 3 client.4906.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:11:37.358 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.357+0000 7fa9b9ffb700 3 client.4906.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:11:37.358 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.357+0000 7fa9b9ffb700 20 client.4906.objecter dump_active .. 0 homeless 2023-09-30T02:11:37.358 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.357+0000 7fa9c9074700 1 --2- 172.21.15.59:0/3777488120 >> v2:172.21.15.110:6800/115870 conn(0x7fa9a8068910 0x7fa9a806adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:37.358 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.357+0000 7fa9c9074700 10 monclient: get_auth_request con 0x7fa9a8068910 auth_method 0 2023-09-30T02:11:37.358 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.358+0000 7fa9c9074700 1 --2- 172.21.15.59:0/3777488120 >> v2:172.21.15.110:6800/115870 conn(0x7fa9a8068910 0x7fa9a806adc0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fa9c413be80 tx=0x7fa9b40091a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:37.358 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.358+0000 7fa9b9ffb700 10 client.4906.objecter ms_handle_connect 0x7fa9a8068910 2023-09-30T02:11:37.366 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.366+0000 7fa9b9ffb700 1 -- 172.21.15.59:0/3777488120 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa9c0053020 con 0x7fa9c40a5b70 2023-09-30T02:11:37.366 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.366+0000 7fa9b9ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:11:37.366 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.366+0000 7fa9b9ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:11:37.594 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.595+0000 7fa9cbad9700 1 -- 172.21.15.59:0/3777488120 --> v2:172.21.15.110:6800/115870 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7fa9c406a720 con 0x7fa9a8068910 2023-09-30T02:11:37.595 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.596+0000 7fa9b9ffb700 1 -- 172.21.15.59:0/3777488120 <== mgr.4108 v2:172.21.15.110:6800/115870 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fa9c406a720 con 0x7fa9a8068910 2023-09-30T02:11:37.598 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.598+0000 7fa9cbad9700 10 librados: watch_flush enter 2023-09-30T02:11:37.598 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.598+0000 7fa9cbad9700 10 librados: watch_flush exit 2023-09-30T02:11:37.598 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.598+0000 7fa9cbad9700 20 client.4906.objecter shutdown clearing up homeless session... 2023-09-30T02:11:37.598 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.598+0000 7fa9cbad9700 10 client.4906.objecter successfully canceled tick 2023-09-30T02:11:37.598 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.598+0000 7fa9cbad9700 1 -- 172.21.15.59:0/3777488120 >> v2:172.21.15.110:6800/115870 conn(0x7fa9a8068910 msgr2=0x7fa9a806adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:37.598 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.598+0000 7fa9cbad9700 1 --2- 172.21.15.59:0/3777488120 >> v2:172.21.15.110:6800/115870 conn(0x7fa9a8068910 0x7fa9a806adc0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fa9c413be80 tx=0x7fa9b40091a0 comp rx=0 tx=0).stop 2023-09-30T02:11:37.599 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.598+0000 7fa9cbad9700 10 monclient: shutdown 2023-09-30T02:11:37.599 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.598+0000 7fa9cbad9700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:11:37.599 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.598+0000 7fa9cbad9700 1 -- 172.21.15.59:0/3777488120 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa9c40a5b70 msgr2=0x7fa9c40a8800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:37.599 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.598+0000 7fa9cbad9700 1 --2- 172.21.15.59:0/3777488120 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa9c40a5b70 0x7fa9c40a8800 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fa9c40a76e0 tx=0x7fa9c0017000 comp rx=0 tx=0).stop 2023-09-30T02:11:37.599 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.598+0000 7fa9cbad9700 1 -- 172.21.15.59:0/3777488120 shutdown_connections 2023-09-30T02:11:37.599 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.598+0000 7fa9cbad9700 1 --2- 172.21.15.59:0/3777488120 >> v2:172.21.15.110:6800/115870 conn(0x7fa9a8068910 0x7fa9a806adc0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:37.599 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.598+0000 7fa9cbad9700 1 --2- 172.21.15.59:0/3777488120 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fa9c40a5b70 0x7fa9c40a8800 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:37.599 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.598+0000 7fa9cbad9700 1 --2- 172.21.15.59:0/3777488120 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9c41420c0 0x7fa9c41399b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:37.599 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.598+0000 7fa9cbad9700 1 --2- 172.21.15.59:0/3777488120 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fa9c40a5f40 0x7fa9c40a8d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:37.599 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.598+0000 7fa9cbad9700 1 -- 172.21.15.59:0/3777488120 >> 172.21.15.59:0/3777488120 conn(0x7fa9c40631f0 msgr2=0x7fa9c40b0be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:11:37.599 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.598+0000 7fa9cbad9700 1 -- 172.21.15.59:0/3777488120 shutdown_connections 2023-09-30T02:11:37.600 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.598+0000 7fa9cbad9700 1 -- 172.21.15.59:0/3777488120 wait complete. 2023-09-30T02:11:37.600 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.598+0000 7fa9cbad9700 1 librados: shutdown 2023-09-30T02:11:37.600 INFO:teuthology.orchestra.run.smithi059.stderr:instructed osd(s) 2 to deep-scrub 2023-09-30T02:11:37.613 INFO:tasks.ceph:Scrubbing osd.3 2023-09-30T02:11:37.613 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2023-09-30T02:11:37.751 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.751+0000 7fe9a2aad700 10 monclient: get_monmap_and_config 2023-09-30T02:11:37.751 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.751+0000 7fe9a2aad700 10 monclient: build_initial_monmap 2023-09-30T02:11:37.751 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.751+0000 7fe9a2aad700 10 monclient: monmap: 2023-09-30T02:11:37.751 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:11:37.751 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:37.751 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:11:37.752096+0000 2023-09-30T02:11:37.752 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:11:37.752096+0000 2023-09-30T02:11:37.752 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:11:37.752 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:37.752 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:11:37.752 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:11:37.752 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:11:37.752 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:11:37.752 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:11:37.753 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:37.754 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.752+0000 7fe9a2aad700 1 Processor -- start 2023-09-30T02:11:37.755 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.752+0000 7fe9a2aad700 1 -- start start 2023-09-30T02:11:37.755 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.752+0000 7fe9a2aad700 10 monclient: init 2023-09-30T02:11:37.755 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.753+0000 7fe9a2aad700 10 monclient: _reopen_session rank -1 2023-09-30T02:11:37.755 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.753+0000 7fe9a2aad700 10 monclient: _add_conns ranks=[0,2,4,1,3] 2023-09-30T02:11:37.755 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.753+0000 7fe9a2aad700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe99c0a5730 addr v1:172.21.15.59:6789/0 2023-09-30T02:11:37.755 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.753+0000 7fe9a2aad700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe99c0a7fa0 0x7fe99c0af5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:37.755 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.753+0000 7fe9a2aad700 10 monclient(hunting): picked mon.noname-b con 0x7fe99c0a7fa0 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:11:37.755 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.753+0000 7fe9a2aad700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe99c0afae0 0x7fe99c0aff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:37.755 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.753+0000 7fe9a2aad700 10 monclient(hunting): picked mon.noname-c con 0x7fe99c0afae0 addr v2:172.21.15.110:3300/0 2023-09-30T02:11:37.756 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.753+0000 7fe9a2aad700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe99c0a7b10 con 0x7fe99c0a5730 2023-09-30T02:11:37.756 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.753+0000 7fe9a2aad700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:37.756 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.753+0000 7fe9a2aad700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe99c0a7410 con 0x7fe99c0afae0 2023-09-30T02:11:37.756 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.753+0000 7fe9a2aad700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:37.756 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.753+0000 7fe9a2aad700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe99c0a5b00 con 0x7fe99c0a7fa0 2023-09-30T02:11:37.756 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.753+0000 7fe9a2aad700 10 monclient(hunting): _renew_subs 2023-09-30T02:11:37.756 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.753+0000 7fe9a2aad700 10 monclient(hunting): authenticate will time out at 2143.055461s 2023-09-30T02:11:37.756 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.754+0000 7fe99bfff700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe99c0a7fa0 0x7fe99c0af5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:37.756 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.754+0000 7fe9a104a700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe99c0afae0 0x7fe99c0aff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:37.756 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.754+0000 7fe99bfff700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe99c0a7fa0 0x7fe99c0af5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:46136/0 (socket says 172.21.15.59:46136) 2023-09-30T02:11:37.757 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.754+0000 7fe99bfff700 1 -- 172.21.15.59:0/3153757177 learned_addr learned my addr 172.21.15.59:0/3153757177 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:11:37.757 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.754+0000 7fe99bfff700 10 monclient(hunting): get_auth_request con 0x7fe99c0a7fa0 auth_method 0 2023-09-30T02:11:37.757 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.754+0000 7fe99bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:37.757 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.754+0000 7fe99bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:37.757 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.754+0000 7fe99bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:37.757 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.754+0000 7fe9a104a700 10 monclient(hunting): get_auth_request con 0x7fe99c0afae0 auth_method 0 2023-09-30T02:11:37.757 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.754+0000 7fe9a104a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:37.757 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.754+0000 7fe9a104a700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:37.757 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.754+0000 7fe9a104a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:37.757 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.755+0000 7fe99bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:37.758 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.755+0000 7fe99bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:37.758 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.755+0000 7fe99bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:37.758 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.755+0000 7fe9a104a700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:37.758 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.755+0000 7fe9a104a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:37.758 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.755+0000 7fe9a104a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:37.759 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.757+0000 7fe99bfff700 10 monclient(hunting): handle_auth_done global_id 4778 payload 274 2023-09-30T02:11:37.759 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.757+0000 7fe99bfff700 1 -- 172.21.15.59:0/3153757177 >> v2:172.21.15.110:3300/0 conn(0x7fe99c0afae0 msgr2=0x7fe99c0aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:11:37.759 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.757+0000 7fe99bfff700 1 --2- 172.21.15.59:0/3153757177 >> v2:172.21.15.110:3300/0 conn(0x7fe99c0afae0 0x7fe99c0aff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:37.759 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.757+0000 7fe99bfff700 1 -- 172.21.15.59:0/3153757177 >> v1:172.21.15.59:6789/0 conn(0x7fe99c0a5730 legacy=0x7fe99c1418e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:37.759 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.757+0000 7fe99bfff700 10 monclient: _finish_hunting 0 2023-09-30T02:11:37.759 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.757+0000 7fe99bfff700 1 monclient: found mon.noname-b 2023-09-30T02:11:37.759 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.757+0000 7fe99bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:11:37.760 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.757+0000 7fe99bfff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.59:3301/0 2023-09-30T02:11:37.760 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.757+0000 7fe99bfff700 1 -- 172.21.15.59:0/3153757177 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe99c0a7750 con 0x7fe99c0a7fa0 2023-09-30T02:11:37.760 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.759+0000 7fe99bfff700 1 --2- 172.21.15.59:0/3153757177 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe99c0a7fa0 0x7fe99c0af5a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fe9880099b0 tx=0x7fe988009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ee9bd240b5abb6a server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:37.762 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.763+0000 7fe99b7fe700 1 -- 172.21.15.59:0/3153757177 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe988009ea0 con 0x7fe99c0a7fa0 2023-09-30T02:11:37.762 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.763+0000 7fe99b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:37.762 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.763+0000 7fe99b7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:11:37.763 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.763+0000 7fe99b7fe700 10 monclient: dump: 2023-09-30T02:11:37.763 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:37.763 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:37.763 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:37.763 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:37.763 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:37.763 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:37.763 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:37.763 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:37.763 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:37.763 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:37.764 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.763+0000 7fe99b7fe700 10 monclient: _finish_auth 0 2023-09-30T02:11:37.764 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.763+0000 7fe99b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:11:37.764 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.763+0000 7fe9a2aad700 5 monclient: authenticate success, global_id 4778 2023-09-30T02:11:37.764 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.763+0000 7fe9a2aad700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:11:37.766 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.767+0000 7fe99b7fe700 1 -- 172.21.15.59:0/3153757177 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe98800b860 con 0x7fe99c0a7fa0 2023-09-30T02:11:37.767 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.767+0000 7fe99b7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:11:37.767 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.767+0000 7fe99b7fe700 1 -- 172.21.15.59:0/3153757177 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe98800bc00 con 0x7fe99c0a7fa0 2023-09-30T02:11:37.767 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.767+0000 7fe99b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:37.767 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.767+0000 7fe99b7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:11:37.767 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.767+0000 7fe99b7fe700 10 monclient: dump: 2023-09-30T02:11:37.767 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:37.767 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:37.767 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:37.767 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:37.767 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:37.768 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:37.768 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:37.768 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:37.768 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:37.768 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:37.768 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.767+0000 7fe9a2aad700 10 monclient: get_monmap_and_config success 2023-09-30T02:11:37.768 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.767+0000 7fe9a2aad700 4 set_mon_vals no callback set 2023-09-30T02:11:37.769 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.768+0000 7fe9a2aad700 10 monclient: shutdown 2023-09-30T02:11:37.769 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.768+0000 7fe9a2aad700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:11:37.770 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.768+0000 7fe9a2aad700 1 -- 172.21.15.59:0/3153757177 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe99c0a7fa0 msgr2=0x7fe99c0af5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:37.770 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.768+0000 7fe9a2aad700 1 --2- 172.21.15.59:0/3153757177 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe99c0a7fa0 0x7fe99c0af5a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fe9880099b0 tx=0x7fe988009250 comp rx=0 tx=0).stop 2023-09-30T02:11:37.770 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.768+0000 7fe9a2aad700 1 -- 172.21.15.59:0/3153757177 shutdown_connections 2023-09-30T02:11:37.770 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.768+0000 7fe9a2aad700 1 --2- 172.21.15.59:0/3153757177 >> v2:172.21.15.110:3300/0 conn(0x7fe99c0afae0 0x7fe99c0aff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:37.770 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.768+0000 7fe9a2aad700 1 --2- 172.21.15.59:0/3153757177 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe99c0a7fa0 0x7fe99c0af5a0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:37.770 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.768+0000 7fe9a2aad700 1 -- 172.21.15.59:0/3153757177 >> 172.21.15.59:0/3153757177 conn(0x7fe99c0631f0 msgr2=0x7fe99c0635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:11:37.770 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.768+0000 7fe9a2aad700 1 -- 172.21.15.59:0/3153757177 shutdown_connections 2023-09-30T02:11:37.770 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.768+0000 7fe9a2aad700 1 -- 172.21.15.59:0/3153757177 wait complete. 2023-09-30T02:11:37.770 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.769+0000 7fe9a2aad700 10 monclient: build_initial_monmap 2023-09-30T02:11:37.771 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.769+0000 7fe9a2aad700 10 monclient: monmap: 2023-09-30T02:11:37.771 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:11:37.771 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:11:37.771 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:11:37.771 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:11:37.771 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:11:37.771 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:11:37.771 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:11:37.771 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:11:37.771 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:11:37.772 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:37.772 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.769+0000 7fe9a2aad700 1 librados: starting msgr at 2023-09-30T02:11:37.772 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.769+0000 7fe9a2aad700 1 librados: starting objecter 2023-09-30T02:11:37.772 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.770+0000 7fe9a2aad700 1 Processor -- start 2023-09-30T02:11:37.772 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.770+0000 7fe9a2aad700 1 -- start start 2023-09-30T02:11:37.772 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.770+0000 7fe9a2aad700 1 librados: setting wanted keys 2023-09-30T02:11:37.772 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.770+0000 7fe9a2aad700 1 librados: calling monclient init 2023-09-30T02:11:37.772 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.770+0000 7fe9a2aad700 10 monclient: init 2023-09-30T02:11:37.774 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.771+0000 7fe9a2aad700 10 monclient: _reopen_session rank -1 2023-09-30T02:11:37.775 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.771+0000 7fe9a2aad700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T02:11:37.775 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.771+0000 7fe9a2aad700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe99c0a5730 0x7fe99c14e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:37.775 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.771+0000 7fe9a2aad700 10 monclient(hunting): picked mon.noname-b con 0x7fe99c0a5730 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:11:37.775 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.771+0000 7fe9a2aad700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe99c0a7fa0 0x7fe99c13cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:37.775 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.771+0000 7fe9a2aad700 10 monclient(hunting): picked mon.noname-a con 0x7fe99c0a7fa0 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:11:37.775 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.771+0000 7fe9a2aad700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe99c0afae0 0x7fe99c13dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:37.775 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.771+0000 7fe9a2aad700 10 monclient(hunting): picked mon.noname-c con 0x7fe99c0afae0 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:11:37.775 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.771+0000 7fe9a2aad700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:37.775 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.771+0000 7fe9a2aad700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe99c0a86d0 con 0x7fe99c0a7fa0 2023-09-30T02:11:37.775 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.771+0000 7fe9a2aad700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:37.776 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.771+0000 7fe9a2aad700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe99c13d510 con 0x7fe99c0a5730 2023-09-30T02:11:37.776 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.771+0000 7fe9a2aad700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:37.776 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.771+0000 7fe9a2aad700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe99c13d7c0 con 0x7fe99c0afae0 2023-09-30T02:11:37.776 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.771+0000 7fe9a2aad700 10 monclient(hunting): _renew_subs 2023-09-30T02:11:37.776 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.771+0000 7fe9a2aad700 10 monclient(hunting): authenticate will time out at 2143.073279s 2023-09-30T02:11:37.776 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.771+0000 7fe9a104a700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe99c0afae0 0x7fe99c13dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:37.776 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.772+0000 7fe9a104a700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe99c0afae0 0x7fe99c13dce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:46146/0 (socket says 172.21.15.59:46146) 2023-09-30T02:11:37.776 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.772+0000 7fe9a104a700 1 -- 172.21.15.59:0/4046959445 learned_addr learned my addr 172.21.15.59:0/4046959445 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:11:37.776 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.772+0000 7fe9a104a700 10 monclient(hunting): get_auth_request con 0x7fe99c0afae0 auth_method 0 2023-09-30T02:11:37.776 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.772+0000 7fe9a104a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:37.776 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.772+0000 7fe9a104a700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:37.777 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.772+0000 7fe9a104a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:37.777 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.772+0000 7fe9a104a700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:37.777 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.772+0000 7fe9a104a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:37.777 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.772+0000 7fe9a104a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:37.777 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.773+0000 7fe9a0849700 1 --2- 172.21.15.59:0/4046959445 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe99c0a5730 0x7fe99c14e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:37.778 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.776+0000 7fe9a104a700 10 monclient(hunting): handle_auth_done global_id 4781 payload 824 2023-09-30T02:11:37.778 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.776+0000 7fe9a104a700 1 -- 172.21.15.59:0/4046959445 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe99c0a5730 msgr2=0x7fe99c14e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:37.778 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.776+0000 7fe9a104a700 1 --2- 172.21.15.59:0/4046959445 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe99c0a5730 0x7fe99c14e1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:37.778 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.776+0000 7fe9a104a700 1 -- 172.21.15.59:0/4046959445 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe99c0a7fa0 msgr2=0x7fe99c13cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:37.779 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.776+0000 7fe9a104a700 1 --2- 172.21.15.59:0/4046959445 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe99c0a7fa0 0x7fe99c13cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:37.779 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.776+0000 7fe9a104a700 10 monclient: _finish_hunting 0 2023-09-30T02:11:37.779 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.776+0000 7fe9a104a700 1 monclient: found mon.noname-c 2023-09-30T02:11:37.779 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.776+0000 7fe9a104a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:11:37.779 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.776+0000 7fe9a104a700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.59:3301/0 2023-09-30T02:11:37.779 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.776+0000 7fe9a104a700 1 -- 172.21.15.59:0/4046959445 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9880094f0 con 0x7fe99c0afae0 2023-09-30T02:11:37.779 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.779+0000 7fe9a104a700 1 --2- 172.21.15.59:0/4046959445 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe99c0afae0 0x7fe99c13dce0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fe994009df0 tx=0x7fe99400b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:37.779 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.779+0000 7fe998ff9700 10 client.?.objecter ms_handle_connect 0x7fe99c0afae0 2023-09-30T02:11:37.779 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.779+0000 7fe998ff9700 10 client.?.objecter resend_mon_ops 2023-09-30T02:11:37.783 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.783+0000 7fe998ff9700 1 -- 172.21.15.59:0/4046959445 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe99400cb50 con 0x7fe99c0afae0 2023-09-30T02:11:37.783 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.783+0000 7fe998ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:37.783 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.783+0000 7fe998ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:11:37.784 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.783+0000 7fe998ff9700 10 monclient: dump: 2023-09-30T02:11:37.784 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:37.784 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:37.784 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:37.784 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:37.784 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:37.784 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:37.784 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:37.784 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:37.784 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:37.784 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:37.784 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.783+0000 7fe998ff9700 10 monclient: _finish_auth 0 2023-09-30T02:11:37.785 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.783+0000 7fe998ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:11:37.785 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.783+0000 7fe998ff9700 1 -- 172.21.15.59:0/4046959445 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe99400f040 con 0x7fe99c0afae0 2023-09-30T02:11:37.785 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.783+0000 7fe9a2aad700 5 monclient: authenticate success, global_id 4781 2023-09-30T02:11:37.785 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.783+0000 7fe9a2aad700 10 monclient: _renew_subs 2023-09-30T02:11:37.785 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.783+0000 7fe9a2aad700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:11:37.785 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.783+0000 7fe9a2aad700 1 -- 172.21.15.59:0/4046959445 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe99c14ed10 con 0x7fe99c0afae0 2023-09-30T02:11:37.785 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.783+0000 7fe998ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:11:37.785 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.783+0000 7fe998ff9700 1 -- 172.21.15.59:0/4046959445 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe994014480 con 0x7fe99c0afae0 2023-09-30T02:11:37.785 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.783+0000 7fe998ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:37.785 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.783+0000 7fe998ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:11:37.785 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.783+0000 7fe998ff9700 10 monclient: dump: 2023-09-30T02:11:37.786 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:37.786 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:37.786 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:37.786 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:37.786 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:37.786 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:37.786 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:37.786 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:37.786 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:37.786 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:37.786 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.783+0000 7fe99affd700 4 set_mon_vals no callback set 2023-09-30T02:11:37.787 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.783+0000 7fe9a2aad700 10 client.4781.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:11:37.787 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.783+0000 7fe9a2aad700 10 monclient: _renew_subs 2023-09-30T02:11:37.787 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.783+0000 7fe9a2aad700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:11:37.787 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.783+0000 7fe9a2aad700 1 -- 172.21.15.59:0/4046959445 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe99c0ad250 con 0x7fe99c0afae0 2023-09-30T02:11:37.787 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.783+0000 7fe9a2aad700 1 librados: init done 2023-09-30T02:11:37.787 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.784+0000 7fe9a2aad700 10 client.4781.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:11:37.787 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.784+0000 7fe9a2aad700 15 client.4781.objecter _session_command_op_assign -1 1 2023-09-30T02:11:37.787 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.784+0000 7fe9a2aad700 20 client.4781.objecter _get_session osd=-1 returning homeless 2023-09-30T02:11:37.787 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.784+0000 7fe9a2aad700 10 client.4781.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:11:37.787 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.784+0000 7fe9a2aad700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:11:37.787 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.784+0000 7fe9a2aad700 1 -- 172.21.15.59:0/4046959445 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe99c064a00 con 0x7fe99c0afae0 2023-09-30T02:11:37.808 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.808+0000 7fe998ff9700 1 -- 172.21.15.59:0/4046959445 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fe994014830 con 0x7fe99c0afae0 2023-09-30T02:11:37.808 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.809+0000 7fe998ff9700 1 --2- 172.21.15.59:0/4046959445 >> v2:172.21.15.110:6800/115870 conn(0x7fe980068960 0x7fe98006ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:37.809 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.809+0000 7fe9a0849700 1 --2- 172.21.15.59:0/4046959445 >> v2:172.21.15.110:6800/115870 conn(0x7fe980068960 0x7fe98006ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:37.809 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.810+0000 7fe9a0849700 10 monclient: get_auth_request con 0x7fe980068960 auth_method 0 2023-09-30T02:11:37.809 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.810+0000 7fe998ff9700 10 client.4781.objecter ms_handle_connect 0x7fe980068960 2023-09-30T02:11:37.809 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.810+0000 7fe9a0849700 1 --2- 172.21.15.59:0/4046959445 >> v2:172.21.15.110:6800/115870 conn(0x7fe980068960 0x7fe98006ae10 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fe9900016d0 tx=0x7fe990009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:37.818 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.818+0000 7fe998ff9700 1 -- 172.21.15.59:0/4046959445 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fe994057030 con 0x7fe99c0afae0 2023-09-30T02:11:37.819 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.818+0000 7fe998ff9700 10 client.4781.objecter ms_dispatch 0x7fe99c001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:11:37.819 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.818+0000 7fe998ff9700 3 client.4781.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:11:37.819 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.818+0000 7fe998ff9700 3 client.4781.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:11:37.819 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.819+0000 7fe998ff9700 10 client.4781.objecter checking command 1 2023-09-30T02:11:37.819 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.819+0000 7fe998ff9700 1 --2- 172.21.15.59:0/4046959445 >> v2:172.21.15.59:6812/116667 conn(0x7fe98006e8d0 0x7fe980070ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:37.819 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.819+0000 7fe998ff9700 20 client.4781.objecter _get_session s=0x7fe98006e220 osd=3 3 2023-09-30T02:11:37.819 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.819+0000 7fe998ff9700 20 client.4781.objecter put_session s=0x7fe98006e220 osd=3 3 2023-09-30T02:11:37.819 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.819+0000 7fe998ff9700 15 client.4781.objecter _session_command_op_remove -1 1 2023-09-30T02:11:37.819 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.819+0000 7fe998ff9700 20 client.4781.objecter _get_session s=0x7fe98006e220 osd=3 3 2023-09-30T02:11:37.819 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.819+0000 7fe998ff9700 20 client.4781.objecter get_session s=0x7fe98006e220 osd=3 3 2023-09-30T02:11:37.820 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.819+0000 7fe998ff9700 15 client.4781.objecter _session_command_op_assign 3 1 2023-09-30T02:11:37.820 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.819+0000 7fe998ff9700 20 client.4781.objecter put_session s=0x7fe98006e220 osd=3 4 2023-09-30T02:11:37.820 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.819+0000 7fe998ff9700 10 client.4781.objecter _send_command 1 2023-09-30T02:11:37.820 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.819+0000 7fe998ff9700 1 -- 172.21.15.59:0/4046959445 --> v2:172.21.15.59:6812/116667 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe980071350 con 0x7fe98006e8d0 2023-09-30T02:11:37.820 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.819+0000 7fe998ff9700 20 client.4781.objecter dump_active .. 0 homeless 2023-09-30T02:11:37.820 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.819+0000 7fe99bfff700 1 --2- 172.21.15.59:0/4046959445 >> v2:172.21.15.59:6812/116667 conn(0x7fe98006e8d0 0x7fe980070ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:37.821 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.820+0000 7fe99bfff700 10 monclient: get_auth_request con 0x7fe98006e8d0 auth_method 0 2023-09-30T02:11:37.822 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.823+0000 7fe998ff9700 10 client.4781.objecter ms_handle_connect 0x7fe98006e8d0 2023-09-30T02:11:37.822 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.823+0000 7fe99bfff700 1 --2- 172.21.15.59:0/4046959445 >> v2:172.21.15.59:6812/116667 conn(0x7fe98006e8d0 0x7fe980070ce0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:37.850 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.851+0000 7fe998ff9700 1 -- 172.21.15.59:0/4046959445 <== osd.3 v2:172.21.15.59:6812/116667 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7fe980071350 con 0x7fe98006e8d0 2023-09-30T02:11:37.851 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.851+0000 7fe998ff9700 10 client.4781.objecter ms_dispatch 0x7fe99c001550 command_reply(tid 1: 0 ) v1 2023-09-30T02:11:37.851 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.851+0000 7fe998ff9700 10 client.4781.objecter _finish_command 1 = system:0 2023-09-30T02:11:37.851 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.851+0000 7fe998ff9700 20 client.4781.objecter put_session s=0x7fe98006e220 osd=3 4 2023-09-30T02:11:37.851 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.851+0000 7fe998ff9700 15 client.4781.objecter _session_command_op_remove 3 1 2023-09-30T02:11:37.865 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.866+0000 7fe9a2aad700 10 client.4781.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-09-30T02:11:37.865 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.866+0000 7fe9a2aad700 15 client.4781.objecter _session_command_op_assign -1 2 2023-09-30T02:11:37.865 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.866+0000 7fe9a2aad700 20 client.4781.objecter _get_session s=0x7fe98006e220 osd=3 3 2023-09-30T02:11:37.865 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.866+0000 7fe9a2aad700 20 client.4781.objecter put_session s=0x7fe98006e220 osd=3 3 2023-09-30T02:11:37.866 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.866+0000 7fe9a2aad700 20 client.4781.objecter _get_session s=0x7fe98006e220 osd=3 3 2023-09-30T02:11:37.866 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.866+0000 7fe9a2aad700 15 client.4781.objecter _session_command_op_remove -1 2 2023-09-30T02:11:37.866 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.866+0000 7fe9a2aad700 20 client.4781.objecter get_session s=0x7fe98006e220 osd=3 3 2023-09-30T02:11:37.866 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.866+0000 7fe9a2aad700 15 client.4781.objecter _session_command_op_assign 3 2 2023-09-30T02:11:37.866 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.866+0000 7fe9a2aad700 20 client.4781.objecter put_session s=0x7fe98006e220 osd=3 4 2023-09-30T02:11:37.866 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.866+0000 7fe9a2aad700 10 client.4781.objecter _send_command 2 2023-09-30T02:11:37.866 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.866+0000 7fe9a2aad700 1 -- 172.21.15.59:0/4046959445 --> v2:172.21.15.59:6812/116667 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fe99c0ab8f0 con 0x7fe98006e8d0 2023-09-30T02:11:37.892 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.892+0000 7fe998ff9700 1 -- 172.21.15.59:0/4046959445 <== osd.3 v2:172.21.15.59:6812/116667 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fe99c0ab8f0 con 0x7fe98006e8d0 2023-09-30T02:11:37.892 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.892+0000 7fe998ff9700 10 client.4781.objecter ms_dispatch 0x7fe99c001550 command_reply(tid 2: 0 ) v1 2023-09-30T02:11:37.892 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.892+0000 7fe998ff9700 10 client.4781.objecter _finish_command 2 = system:0 2023-09-30T02:11:37.892 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.892+0000 7fe998ff9700 20 client.4781.objecter put_session s=0x7fe98006e220 osd=3 4 2023-09-30T02:11:37.892 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.892+0000 7fe998ff9700 15 client.4781.objecter _session_command_op_remove 3 2 2023-09-30T02:11:37.893 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2023-09-30T02:11:37.893 INFO:teuthology.orchestra.run.smithi059.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T02:11:37.893 INFO:teuthology.orchestra.run.smithi059.stdout:} 2023-09-30T02:11:37.894 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.893+0000 7fe9a2aad700 10 librados: watch_flush enter 2023-09-30T02:11:37.894 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.893+0000 7fe9a2aad700 10 librados: watch_flush exit 2023-09-30T02:11:37.895 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.893+0000 7fe9a2aad700 10 client.4781.objecter close_session for osd.3 2023-09-30T02:11:37.895 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.893+0000 7fe9a2aad700 1 -- 172.21.15.59:0/4046959445 >> v2:172.21.15.59:6812/116667 conn(0x7fe98006e8d0 msgr2=0x7fe980070ce0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:37.895 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.893+0000 7fe9a2aad700 1 --2- 172.21.15.59:0/4046959445 >> v2:172.21.15.59:6812/116667 conn(0x7fe98006e8d0 0x7fe980070ce0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:37.895 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.893+0000 7fe9a2aad700 20 client.4781.objecter put_session s=0x7fe98006e220 osd=3 1 2023-09-30T02:11:37.895 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.893+0000 7fe9a2aad700 20 client.4781.objecter shutdown clearing up homeless session... 2023-09-30T02:11:37.895 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.893+0000 7fe9a2aad700 10 client.4781.objecter successfully canceled tick 2023-09-30T02:11:37.895 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.893+0000 7fe9a2aad700 1 -- 172.21.15.59:0/4046959445 >> v2:172.21.15.110:6800/115870 conn(0x7fe980068960 msgr2=0x7fe98006ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:37.895 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.893+0000 7fe9a2aad700 1 --2- 172.21.15.59:0/4046959445 >> v2:172.21.15.110:6800/115870 conn(0x7fe980068960 0x7fe98006ae10 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fe9900016d0 tx=0x7fe990009040 comp rx=0 tx=0).stop 2023-09-30T02:11:37.895 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.893+0000 7fe9a2aad700 10 monclient: shutdown 2023-09-30T02:11:37.895 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.893+0000 7fe9a2aad700 20 monclient: shutdown canceling and discarding version request 1 2023-09-30T02:11:37.895 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.893+0000 7fe9a2aad700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:11:37.895 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.893+0000 7fe9a2aad700 1 -- 172.21.15.59:0/4046959445 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe99c0afae0 msgr2=0x7fe99c13dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:37.896 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.893+0000 7fe9a2aad700 1 --2- 172.21.15.59:0/4046959445 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe99c0afae0 0x7fe99c13dce0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fe994009df0 tx=0x7fe99400b040 comp rx=0 tx=0).stop 2023-09-30T02:11:37.896 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.893+0000 7fe9a104a700 1 -- 172.21.15.59:0/4046959445 reap_dead start 2023-09-30T02:11:37.896 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.893+0000 7fe9a2aad700 1 -- 172.21.15.59:0/4046959445 shutdown_connections 2023-09-30T02:11:37.896 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.893+0000 7fe9a2aad700 1 -- 172.21.15.59:0/4046959445 >> 172.21.15.59:0/4046959445 conn(0x7fe99c0631f0 msgr2=0x7fe99c143fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:11:37.896 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.893+0000 7fe9a2aad700 1 -- 172.21.15.59:0/4046959445 shutdown_connections 2023-09-30T02:11:37.896 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.893+0000 7fe9a2aad700 1 -- 172.21.15.59:0/4046959445 wait complete. 2023-09-30T02:11:37.896 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:37.893+0000 7fe9a2aad700 1 librados: shutdown 2023-09-30T02:11:37.907 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2023-09-30T02:11:38.049 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.049+0000 7f46c6491700 10 monclient: get_monmap_and_config 2023-09-30T02:11:38.049 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.049+0000 7f46c6491700 10 monclient: build_initial_monmap 2023-09-30T02:11:38.049 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.050+0000 7f46c6491700 10 monclient: monmap: 2023-09-30T02:11:38.049 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:11:38.049 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:38.050 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:11:38.050381+0000 2023-09-30T02:11:38.050 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:11:38.050381+0000 2023-09-30T02:11:38.050 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:11:38.050 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:38.050 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:11:38.050 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:11:38.050 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:11:38.050 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:11:38.050 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:11:38.050 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:38.053 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.050+0000 7f46c6491700 1 Processor -- start 2023-09-30T02:11:38.053 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.050+0000 7f46c6491700 1 -- start start 2023-09-30T02:11:38.053 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.050+0000 7f46c6491700 10 monclient: init 2023-09-30T02:11:38.053 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.051+0000 7f46c6491700 10 monclient: _reopen_session rank -1 2023-09-30T02:11:38.053 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.051+0000 7f46c6491700 10 monclient: _add_conns ranks=[4,2,3,0,1] 2023-09-30T02:11:38.053 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.051+0000 7f46c6491700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f46c00a5b70 0x7f46c00b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:38.053 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.051+0000 7f46c6491700 10 monclient(hunting): picked mon.noname-c con 0x7f46c00a5b70 addr v2:172.21.15.110:3300/0 2023-09-30T02:11:38.053 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.051+0000 7f46c6491700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f46c00a5f40 0x7f46c0141b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:38.053 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.051+0000 7f46c6491700 10 monclient(hunting): picked mon.noname-b con 0x7f46c00a5f40 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:11:38.054 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.051+0000 7f46c6491700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f46c01420c0 0x7f46c013e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:38.054 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.051+0000 7f46c6491700 10 monclient(hunting): picked mon.noname-a con 0x7f46c01420c0 addr v2:172.21.15.59:3300/0 2023-09-30T02:11:38.054 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.051+0000 7f46c6491700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:38.054 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.051+0000 7f46c6491700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46c00a63b0 con 0x7f46c01420c0 2023-09-30T02:11:38.054 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.051+0000 7f46c6491700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:38.054 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.051+0000 7f46c6491700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46c00a75a0 con 0x7f46c00a5b70 2023-09-30T02:11:38.054 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.051+0000 7f46c6491700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:38.054 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.051+0000 7f46c6491700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46c00af3f0 con 0x7f46c00a5f40 2023-09-30T02:11:38.055 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.051+0000 7f46c6491700 10 monclient(hunting): _renew_subs 2023-09-30T02:11:38.055 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.051+0000 7f46c6491700 10 monclient(hunting): authenticate will time out at 2143.353549s 2023-09-30T02:11:38.055 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.052+0000 7f46bf7fe700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f46c00a5f40 0x7f46c0141b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:38.055 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.052+0000 7f46c4a2e700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f46c01420c0 0x7f46c013e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:38.055 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.052+0000 7f46bffff700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f46c00a5b70 0x7f46c00b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:38.055 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.052+0000 7f46bf7fe700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f46c00a5f40 0x7f46c0141b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:46158/0 (socket says 172.21.15.59:46158) 2023-09-30T02:11:38.055 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.052+0000 7f46bf7fe700 1 -- 172.21.15.59:0/1641742072 learned_addr learned my addr 172.21.15.59:0/1641742072 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:11:38.055 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.052+0000 7f46bf7fe700 10 monclient(hunting): get_auth_request con 0x7f46c00a5f40 auth_method 0 2023-09-30T02:11:38.055 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.052+0000 7f46bf7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:38.055 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.052+0000 7f46bf7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:38.056 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.052+0000 7f46bf7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:38.056 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.052+0000 7f46bffff700 10 monclient(hunting): get_auth_request con 0x7f46c00a5b70 auth_method 0 2023-09-30T02:11:38.056 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.052+0000 7f46bffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:38.056 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.052+0000 7f46bffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:38.056 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.052+0000 7f46bffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:38.056 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.052+0000 7f46c4a2e700 10 monclient(hunting): get_auth_request con 0x7f46c01420c0 auth_method 0 2023-09-30T02:11:38.056 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.052+0000 7f46c4a2e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:38.056 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.052+0000 7f46c4a2e700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:38.056 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.052+0000 7f46c4a2e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:38.056 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.053+0000 7f46bffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:38.057 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.053+0000 7f46bffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:38.057 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.053+0000 7f46bffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:38.057 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.053+0000 7f46bf7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:38.057 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.053+0000 7f46bf7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:38.057 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.053+0000 7f46bf7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:38.057 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.053+0000 7f46c4a2e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:38.057 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.053+0000 7f46c4a2e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:38.057 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.053+0000 7f46c4a2e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:38.058 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.055+0000 7f46bffff700 10 monclient(hunting): handle_auth_done global_id 4912 payload 274 2023-09-30T02:11:38.059 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.055+0000 7f46bffff700 1 -- 172.21.15.59:0/1641742072 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f46c00a5f40 msgr2=0x7f46c0141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:11:38.059 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.055+0000 7f46bffff700 1 --2- 172.21.15.59:0/1641742072 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f46c00a5f40 0x7f46c0141b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:38.059 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.055+0000 7f46bffff700 1 -- 172.21.15.59:0/1641742072 >> v2:172.21.15.59:3300/0 conn(0x7f46c01420c0 msgr2=0x7f46c013e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:11:38.059 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.055+0000 7f46bffff700 1 --2- 172.21.15.59:0/1641742072 >> v2:172.21.15.59:3300/0 conn(0x7f46c01420c0 0x7f46c013e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:38.059 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.055+0000 7f46bffff700 10 monclient: _finish_hunting 0 2023-09-30T02:11:38.059 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.055+0000 7f46bffff700 1 monclient: found mon.noname-c 2023-09-30T02:11:38.060 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.055+0000 7f46bffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:11:38.060 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.055+0000 7f46bffff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3300/0 2023-09-30T02:11:38.060 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.055+0000 7f46bffff700 1 -- 172.21.15.59:0/1641742072 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46c0142500 con 0x7f46c00a5b70 2023-09-30T02:11:38.060 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.057+0000 7f46bffff700 1 --2- 172.21.15.59:0/1641742072 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f46c00a5b70 0x7f46c00b1500 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f46c00b49e0 tx=0x7f46b00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a5c743d0e3685fa server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:38.062 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.061+0000 7f46beffd700 1 -- 172.21.15.59:0/1641742072 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b0011630 con 0x7f46c00a5b70 2023-09-30T02:11:38.062 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.061+0000 7f46beffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:38.062 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.061+0000 7f46beffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:11:38.062 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.061+0000 7f46beffd700 10 monclient: dump: 2023-09-30T02:11:38.062 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:38.062 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:38.062 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:38.062 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:38.063 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:38.063 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:38.063 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:38.063 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:38.063 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:38.063 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:38.063 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.061+0000 7f46beffd700 10 monclient: _finish_auth 0 2023-09-30T02:11:38.063 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.061+0000 7f46beffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:11:38.063 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.061+0000 7f46beffd700 1 -- 172.21.15.59:0/1641742072 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46b00117b0 con 0x7f46c00a5b70 2023-09-30T02:11:38.063 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.061+0000 7f46c6491700 5 monclient: authenticate success, global_id 4912 2023-09-30T02:11:38.063 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.061+0000 7f46c6491700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:11:38.064 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.061+0000 7f46beffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:11:38.064 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.061+0000 7f46c6491700 10 monclient: get_monmap_and_config success 2023-09-30T02:11:38.064 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.061+0000 7f46beffd700 1 -- 172.21.15.59:0/1641742072 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b0011b70 con 0x7f46c00a5b70 2023-09-30T02:11:38.064 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.061+0000 7f46beffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:38.064 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.061+0000 7f46c6491700 4 set_mon_vals no callback set 2023-09-30T02:11:38.064 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.061+0000 7f46beffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:11:38.064 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.061+0000 7f46beffd700 10 monclient: dump: 2023-09-30T02:11:38.064 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:38.064 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:38.064 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:38.064 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:38.065 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:38.065 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:38.065 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:38.065 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:38.065 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:38.065 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:38.065 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.061+0000 7f46c6491700 10 monclient: shutdown 2023-09-30T02:11:38.065 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.061+0000 7f46c6491700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:11:38.065 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.061+0000 7f46c6491700 1 -- 172.21.15.59:0/1641742072 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f46c00a5b70 msgr2=0x7f46c00b1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:38.065 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.061+0000 7f46c6491700 1 --2- 172.21.15.59:0/1641742072 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f46c00a5b70 0x7f46c00b1500 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f46c00b49e0 tx=0x7f46b00091a0 comp rx=0 tx=0).stop 2023-09-30T02:11:38.065 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.061+0000 7f46c6491700 1 -- 172.21.15.59:0/1641742072 shutdown_connections 2023-09-30T02:11:38.065 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.061+0000 7f46c6491700 1 --2- 172.21.15.59:0/1641742072 >> v2:172.21.15.59:3300/0 conn(0x7f46c01420c0 0x7f46c013e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:38.066 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.061+0000 7f46c6491700 1 --2- 172.21.15.59:0/1641742072 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f46c00a5b70 0x7f46c00b1500 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:38.066 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.061+0000 7f46c6491700 1 --2- 172.21.15.59:0/1641742072 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f46c00a5f40 0x7f46c0141b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:38.066 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.061+0000 7f46c6491700 1 -- 172.21.15.59:0/1641742072 >> 172.21.15.59:0/1641742072 conn(0x7f46c00631f0 msgr2=0x7f46c00635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:11:38.066 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.062+0000 7f46c6491700 1 -- 172.21.15.59:0/1641742072 shutdown_connections 2023-09-30T02:11:38.066 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.062+0000 7f46c6491700 1 -- 172.21.15.59:0/1641742072 wait complete. 2023-09-30T02:11:38.066 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.062+0000 7f46c6491700 10 monclient: build_initial_monmap 2023-09-30T02:11:38.066 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.062+0000 7f46c6491700 10 monclient: monmap: 2023-09-30T02:11:38.066 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:11:38.066 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:11:38.066 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:11:38.067 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:11:38.067 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:11:38.067 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:11:38.067 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:11:38.067 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:11:38.067 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:11:38.067 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:38.067 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.062+0000 7f46c6491700 1 librados: starting msgr at 2023-09-30T02:11:38.067 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.062+0000 7f46c6491700 1 librados: starting objecter 2023-09-30T02:11:38.067 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.063+0000 7f46c6491700 1 Processor -- start 2023-09-30T02:11:38.070 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.063+0000 7f46c6491700 1 -- start start 2023-09-30T02:11:38.070 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.063+0000 7f46c6491700 1 librados: setting wanted keys 2023-09-30T02:11:38.070 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.063+0000 7f46c6491700 1 librados: calling monclient init 2023-09-30T02:11:38.070 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.063+0000 7f46c6491700 10 monclient: init 2023-09-30T02:11:38.070 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.063+0000 7f46c6491700 10 monclient: _reopen_session rank -1 2023-09-30T02:11:38.071 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.063+0000 7f46c6491700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T02:11:38.071 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.063+0000 7f46c6491700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46c00a5b70 0x7f46c0140900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:38.071 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.063+0000 7f46c6491700 10 monclient(hunting): picked mon.noname-a con 0x7f46c00a5b70 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:11:38.071 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.063+0000 7f46c6491700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f46c00a5f40 0x7f46c0144b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:38.071 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.063+0000 7f46c6491700 10 monclient(hunting): picked mon.noname-c con 0x7f46c00a5f40 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:11:38.071 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.063+0000 7f46c6491700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f46c01420c0 0x7f46c014da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:38.071 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.063+0000 7f46c6491700 10 monclient(hunting): picked mon.noname-b con 0x7f46c01420c0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:11:38.071 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.063+0000 7f46c6491700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:38.071 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.063+0000 7f46c6491700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46c0145390 con 0x7f46c00a5b70 2023-09-30T02:11:38.071 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.063+0000 7f46c6491700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:38.071 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.063+0000 7f46c6491700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46c0145070 con 0x7f46c01420c0 2023-09-30T02:11:38.072 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.063+0000 7f46c6491700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:38.072 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.063+0000 7f46c6491700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46c014b600 con 0x7f46c00a5f40 2023-09-30T02:11:38.072 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.063+0000 7f46c6491700 10 monclient(hunting): _renew_subs 2023-09-30T02:11:38.072 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.063+0000 7f46c6491700 10 monclient(hunting): authenticate will time out at 2143.366000s 2023-09-30T02:11:38.072 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.064+0000 7f46bf7fe700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f46c00a5f40 0x7f46c0144b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:38.072 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.064+0000 7f46c4a2e700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f46c01420c0 0x7f46c014da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:38.072 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.064+0000 7f46bffff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46c00a5b70 0x7f46c0140900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:38.072 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.065+0000 7f46bf7fe700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f46c00a5f40 0x7f46c0144b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:46162/0 (socket says 172.21.15.59:46162) 2023-09-30T02:11:38.072 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.065+0000 7f46bf7fe700 1 -- 172.21.15.59:0/380611716 learned_addr learned my addr 172.21.15.59:0/380611716 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:11:38.072 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.065+0000 7f46bf7fe700 10 monclient(hunting): get_auth_request con 0x7f46c00a5f40 auth_method 0 2023-09-30T02:11:38.073 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.065+0000 7f46bf7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:38.073 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.065+0000 7f46bf7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:38.073 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.065+0000 7f46bf7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:38.073 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.065+0000 7f46c4a2e700 10 monclient(hunting): get_auth_request con 0x7f46c01420c0 auth_method 0 2023-09-30T02:11:38.073 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.065+0000 7f46c4a2e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:38.073 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.065+0000 7f46c4a2e700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:38.073 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.065+0000 7f46c4a2e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:38.073 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.065+0000 7f46bffff700 10 monclient(hunting): get_auth_request con 0x7f46c00a5b70 auth_method 0 2023-09-30T02:11:38.073 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.065+0000 7f46bffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:38.073 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.065+0000 7f46bffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:38.073 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.065+0000 7f46bffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:38.073 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.065+0000 7f46c4a2e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:38.074 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.065+0000 7f46c4a2e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:38.074 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.065+0000 7f46c4a2e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:38.074 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.065+0000 7f46bf7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:38.074 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.065+0000 7f46bf7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:38.074 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.065+0000 7f46bf7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:38.074 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.066+0000 7f46bffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:38.074 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.066+0000 7f46bffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:38.074 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.066+0000 7f46bffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:38.074 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.068+0000 7f46c4a2e700 10 monclient(hunting): handle_auth_done global_id 4915 payload 824 2023-09-30T02:11:38.074 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.068+0000 7f46c4a2e700 1 -- 172.21.15.59:0/380611716 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f46c00a5f40 msgr2=0x7f46c0144b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:38.074 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.068+0000 7f46c4a2e700 1 --2- 172.21.15.59:0/380611716 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f46c00a5f40 0x7f46c0144b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:38.075 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.068+0000 7f46c4a2e700 1 -- 172.21.15.59:0/380611716 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46c00a5b70 msgr2=0x7f46c0140900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:38.075 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.068+0000 7f46c4a2e700 1 --2- 172.21.15.59:0/380611716 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46c00a5b70 0x7f46c0140900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:38.075 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.068+0000 7f46c4a2e700 10 monclient: _finish_hunting 0 2023-09-30T02:11:38.075 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.068+0000 7f46c4a2e700 1 monclient: found mon.noname-b 2023-09-30T02:11:38.075 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.068+0000 7f46c4a2e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:11:38.075 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.068+0000 7f46c4a2e700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:11:38.075 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.068+0000 7f46c4a2e700 1 -- 172.21.15.59:0/380611716 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46b0009420 con 0x7f46c01420c0 2023-09-30T02:11:38.075 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.070+0000 7f46c4a2e700 1 --2- 172.21.15.59:0/380611716 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f46c01420c0 0x7f46c014da60 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f46b800ecc0 tx=0x7f46b800b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:38.075 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.070+0000 7f46a7fff700 10 client.?.objecter ms_handle_connect 0x7f46c01420c0 2023-09-30T02:11:38.075 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.071+0000 7f46a7fff700 10 client.?.objecter resend_mon_ops 2023-09-30T02:11:38.077 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.074+0000 7f46a7fff700 1 -- 172.21.15.59:0/380611716 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b800c8c0 con 0x7f46c01420c0 2023-09-30T02:11:38.077 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.074+0000 7f46a7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:38.077 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.074+0000 7f46a7fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:11:38.077 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.074+0000 7f46a7fff700 10 monclient: dump: 2023-09-30T02:11:38.077 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:38.077 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:38.077 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:38.077 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:38.077 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:38.077 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:38.078 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:38.078 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:38.078 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:38.078 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:38.078 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.074+0000 7f46a7fff700 10 monclient: _finish_auth 0 2023-09-30T02:11:38.078 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.074+0000 7f46a7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:11:38.078 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.074+0000 7f46a7fff700 1 -- 172.21.15.59:0/380611716 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46b800f040 con 0x7f46c01420c0 2023-09-30T02:11:38.078 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.074+0000 7f46a7fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:11:38.078 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.074+0000 7f46a7fff700 1 -- 172.21.15.59:0/380611716 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b800cc90 con 0x7f46c01420c0 2023-09-30T02:11:38.078 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.074+0000 7f46c6491700 5 monclient: authenticate success, global_id 4915 2023-09-30T02:11:38.078 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.074+0000 7f46a7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:38.079 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.074+0000 7f46a7fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:11:38.079 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.074+0000 7f46a7fff700 10 monclient: dump: 2023-09-30T02:11:38.079 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:38.079 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:38.079 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:38.079 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:38.079 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:38.079 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:38.079 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:38.079 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:38.079 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:38.079 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:38.080 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.074+0000 7f46c6491700 10 monclient: _renew_subs 2023-09-30T02:11:38.080 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.074+0000 7f46c6491700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:11:38.080 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.075+0000 7f46c6491700 1 -- 172.21.15.59:0/380611716 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46c0141490 con 0x7f46c01420c0 2023-09-30T02:11:38.080 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.074+0000 7f46be7fc700 4 set_mon_vals no callback set 2023-09-30T02:11:38.080 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.075+0000 7f46c6491700 10 client.4915.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:11:38.080 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.075+0000 7f46c6491700 10 monclient: _renew_subs 2023-09-30T02:11:38.080 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.075+0000 7f46c6491700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:11:38.080 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.075+0000 7f46c6491700 1 -- 172.21.15.59:0/380611716 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46c014dfa0 con 0x7f46c01420c0 2023-09-30T02:11:38.080 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.075+0000 7f46c6491700 1 librados: init done 2023-09-30T02:11:38.080 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.075+0000 7f46c6491700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:11:38.080 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.075+0000 7f46c6491700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:11:38.080 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.075+0000 7f46c6491700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:11:38.080 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.075+0000 7f46c6491700 1 -- 172.21.15.59:0/380611716 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46c0145720 con 0x7f46c01420c0 2023-09-30T02:11:38.319 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.320+0000 7f46a7fff700 1 -- 172.21.15.59:0/380611716 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f46b8014690 con 0x7f46c01420c0 2023-09-30T02:11:38.319 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.320+0000 7f46a7fff700 1 --2- 172.21.15.59:0/380611716 >> v2:172.21.15.110:6800/115870 conn(0x7f46ac068890 0x7f46ac06ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:38.320 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.320+0000 7f46bffff700 1 --2- 172.21.15.59:0/380611716 >> v2:172.21.15.110:6800/115870 conn(0x7f46ac068890 0x7f46ac06ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:38.320 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.320+0000 7f46bffff700 10 monclient: get_auth_request con 0x7f46ac068890 auth_method 0 2023-09-30T02:11:38.320 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.321+0000 7f46bffff700 1 --2- 172.21.15.59:0/380611716 >> v2:172.21.15.110:6800/115870 conn(0x7f46ac068890 0x7f46ac06ad40 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f46c00b49e0 tx=0x7f46b0017000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:38.320 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.321+0000 7f46a7fff700 10 client.4915.objecter ms_handle_connect 0x7f46ac068890 2023-09-30T02:11:38.324 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.324+0000 7f46a7fff700 1 -- 172.21.15.59:0/380611716 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f46b8056ac0 con 0x7f46c01420c0 2023-09-30T02:11:38.324 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.324+0000 7f46a7fff700 10 client.4915.objecter ms_dispatch 0x7f46c0001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:11:38.324 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.324+0000 7f46a7fff700 3 client.4915.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:11:38.324 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.324+0000 7f46a7fff700 3 client.4915.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:11:38.324 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.324+0000 7f46a7fff700 20 client.4915.objecter dump_active .. 0 homeless 2023-09-30T02:11:38.333 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.334+0000 7f46a7fff700 1 -- 172.21.15.59:0/380611716 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f46b8052530 con 0x7f46c01420c0 2023-09-30T02:11:38.333 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.334+0000 7f46a7fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:11:38.334 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.334+0000 7f46a7fff700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:11:38.538 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.539+0000 7f46c6491700 1 -- 172.21.15.59:0/380611716 --> v2:172.21.15.110:6800/115870 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f46c0145500 con 0x7f46ac068890 2023-09-30T02:11:38.539 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.539+0000 7f46a7fff700 1 -- 172.21.15.59:0/380611716 <== mgr.4108 v2:172.21.15.110:6800/115870 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f46c0145500 con 0x7f46ac068890 2023-09-30T02:11:38.542 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.541+0000 7f46c6491700 10 librados: watch_flush enter 2023-09-30T02:11:38.542 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.541+0000 7f46c6491700 10 librados: watch_flush exit 2023-09-30T02:11:38.542 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.541+0000 7f46c6491700 20 client.4915.objecter shutdown clearing up homeless session... 2023-09-30T02:11:38.542 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.541+0000 7f46c6491700 10 client.4915.objecter successfully canceled tick 2023-09-30T02:11:38.542 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.542+0000 7f46c6491700 1 -- 172.21.15.59:0/380611716 >> v2:172.21.15.110:6800/115870 conn(0x7f46ac068890 msgr2=0x7f46ac06ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:38.542 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.542+0000 7f46c6491700 1 --2- 172.21.15.59:0/380611716 >> v2:172.21.15.110:6800/115870 conn(0x7f46ac068890 0x7f46ac06ad40 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f46c00b49e0 tx=0x7f46b0017000 comp rx=0 tx=0).stop 2023-09-30T02:11:38.542 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.542+0000 7f46c6491700 10 monclient: shutdown 2023-09-30T02:11:38.542 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.542+0000 7f46c6491700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:11:38.542 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.542+0000 7f46c6491700 1 -- 172.21.15.59:0/380611716 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f46c01420c0 msgr2=0x7f46c014da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:38.542 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.542+0000 7f46c6491700 1 --2- 172.21.15.59:0/380611716 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f46c01420c0 0x7f46c014da60 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f46b800ecc0 tx=0x7f46b800b040 comp rx=0 tx=0).stop 2023-09-30T02:11:38.543 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.542+0000 7f46c6491700 1 -- 172.21.15.59:0/380611716 shutdown_connections 2023-09-30T02:11:38.543 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.542+0000 7f46c6491700 1 --2- 172.21.15.59:0/380611716 >> v2:172.21.15.110:6800/115870 conn(0x7f46ac068890 0x7f46ac06ad40 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:38.543 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.542+0000 7f46c6491700 1 --2- 172.21.15.59:0/380611716 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f46c01420c0 0x7f46c014da60 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:38.543 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.542+0000 7f46c6491700 1 --2- 172.21.15.59:0/380611716 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46c00a5b70 0x7f46c0140900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:38.543 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.542+0000 7f46c6491700 1 --2- 172.21.15.59:0/380611716 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f46c00a5f40 0x7f46c0144b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:38.543 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.542+0000 7f46c6491700 1 -- 172.21.15.59:0/380611716 >> 172.21.15.59:0/380611716 conn(0x7f46c00631f0 msgr2=0x7f46c00b3bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:11:38.543 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.542+0000 7f46c6491700 1 -- 172.21.15.59:0/380611716 shutdown_connections 2023-09-30T02:11:38.543 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.542+0000 7f46c6491700 1 -- 172.21.15.59:0/380611716 wait complete. 2023-09-30T02:11:38.543 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.542+0000 7f46c6491700 1 librados: shutdown 2023-09-30T02:11:38.544 INFO:teuthology.orchestra.run.smithi059.stderr:instructed osd(s) 3 to deep-scrub 2023-09-30T02:11:38.556 INFO:tasks.ceph:Scrubbing osd.4 2023-09-30T02:11:38.556 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2023-09-30T02:11:38.704 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.705+0000 7fead4732700 10 monclient: get_monmap_and_config 2023-09-30T02:11:38.704 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.705+0000 7fead4732700 10 monclient: build_initial_monmap 2023-09-30T02:11:38.705 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.705+0000 7fead4732700 10 monclient: monmap: 2023-09-30T02:11:38.705 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:11:38.705 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:38.705 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:11:38.705644+0000 2023-09-30T02:11:38.705 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:11:38.705644+0000 2023-09-30T02:11:38.705 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:11:38.705 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:38.705 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:11:38.705 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:11:38.705 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:11:38.705 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:11:38.705 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:11:38.706 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:38.707 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.705+0000 7fead4732700 1 Processor -- start 2023-09-30T02:11:38.708 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.706+0000 7fead4732700 1 -- start start 2023-09-30T02:11:38.708 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.706+0000 7fead4732700 10 monclient: init 2023-09-30T02:11:38.708 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.706+0000 7fead4732700 10 monclient: _reopen_session rank -1 2023-09-30T02:11:38.708 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.706+0000 7fead4732700 10 monclient: _add_conns ranks=[3,4,0,1,2] 2023-09-30T02:11:38.708 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.706+0000 7fead4732700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7feacc0a5730 0x7feacc0a7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:38.708 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.706+0000 7fead4732700 10 monclient(hunting): picked mon.noname-a con 0x7feacc0a5730 addr v2:172.21.15.59:3300/0 2023-09-30T02:11:38.708 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.706+0000 7fead4732700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7feacc1418e0 0x7feacc13ae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:38.708 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.706+0000 7fead4732700 10 monclient(hunting): picked mon.noname-c con 0x7feacc1418e0 addr v2:172.21.15.110:3300/0 2023-09-30T02:11:38.709 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.706+0000 7fead4732700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7feacc13b360 addr v1:172.21.15.59:6789/0 2023-09-30T02:11:38.709 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.707+0000 7fead4732700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feacc0a7b10 con 0x7feacc13b360 2023-09-30T02:11:38.709 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.707+0000 7fead4732700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:38.709 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.707+0000 7fead4732700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feacc0a84e0 con 0x7feacc0a5730 2023-09-30T02:11:38.709 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.707+0000 7fead4732700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:38.709 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.707+0000 7fead4732700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feacc0a5b00 con 0x7feacc1418e0 2023-09-30T02:11:38.709 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.707+0000 7fead4732700 10 monclient(hunting): _renew_subs 2023-09-30T02:11:38.709 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.707+0000 7fead4732700 10 monclient(hunting): authenticate will time out at 2144.009038s 2023-09-30T02:11:38.709 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.707+0000 7fead24ce700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7feacc0a5730 0x7feacc0a7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:38.710 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.707+0000 7fead1ccd700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7feacc1418e0 0x7feacc13ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:38.710 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.708+0000 7fead1ccd700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7feacc1418e0 0x7feacc13ae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:41414/0 (socket says 172.21.15.59:41414) 2023-09-30T02:11:38.710 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.708+0000 7fead1ccd700 1 -- 172.21.15.59:0/3116410849 learned_addr learned my addr 172.21.15.59:0/3116410849 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:11:38.710 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.708+0000 7fead1ccd700 10 monclient(hunting): get_auth_request con 0x7feacc1418e0 auth_method 0 2023-09-30T02:11:38.710 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.708+0000 7fead1ccd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:38.710 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.708+0000 7fead1ccd700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:38.710 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.708+0000 7fead1ccd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:38.710 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.708+0000 7fead24ce700 10 monclient(hunting): get_auth_request con 0x7feacc0a5730 auth_method 0 2023-09-30T02:11:38.710 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.708+0000 7fead24ce700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:38.710 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.708+0000 7fead24ce700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:38.711 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.708+0000 7fead24ce700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:38.711 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.708+0000 7fead1ccd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:38.711 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.708+0000 7fead1ccd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:38.711 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.708+0000 7fead1ccd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:38.712 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.709+0000 7fead24ce700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:38.712 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.709+0000 7fead24ce700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:38.712 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.709+0000 7fead24ce700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:38.712 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.710+0000 7fead1ccd700 10 monclient(hunting): handle_auth_done global_id 4918 payload 274 2023-09-30T02:11:38.712 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.710+0000 7fead1ccd700 1 -- 172.21.15.59:0/3116410849 >> v2:172.21.15.59:3300/0 conn(0x7feacc0a5730 msgr2=0x7feacc0a7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:11:38.712 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.710+0000 7fead1ccd700 1 --2- 172.21.15.59:0/3116410849 >> v2:172.21.15.59:3300/0 conn(0x7feacc0a5730 0x7feacc0a7fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:38.712 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.710+0000 7fead1ccd700 1 -- 172.21.15.59:0/3116410849 >> v1:172.21.15.59:6789/0 conn(0x7feacc13b360 legacy=0x7feacc0af5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:38.713 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.710+0000 7fead1ccd700 10 monclient: _finish_hunting 0 2023-09-30T02:11:38.713 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.710+0000 7fead1ccd700 1 monclient: found mon.noname-c 2023-09-30T02:11:38.713 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.710+0000 7fead1ccd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:11:38.713 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.710+0000 7fead1ccd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3300/0 2023-09-30T02:11:38.713 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.710+0000 7fead1ccd700 1 -- 172.21.15.59:0/3116410849 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feacc0a7750 con 0x7feacc1418e0 2023-09-30T02:11:38.713 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.712+0000 7fead1ccd700 1 --2- 172.21.15.59:0/3116410849 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feacc1418e0 0x7feacc13ae20 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7feacc0a69c0 tx=0x7feabc009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6f5537c04a2b5997 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:38.716 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.716+0000 7fead14cc700 1 -- 172.21.15.59:0/3116410849 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feabc009e70 con 0x7feacc1418e0 2023-09-30T02:11:38.716 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.716+0000 7fead14cc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:38.716 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.716+0000 7fead14cc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:11:38.716 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.716+0000 7fead14cc700 10 monclient: dump: 2023-09-30T02:11:38.716 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:38.716 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:38.716 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:38.716 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:38.717 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:38.717 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:38.717 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:38.717 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:38.717 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:38.717 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:38.717 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.716+0000 7fead14cc700 10 monclient: _finish_auth 0 2023-09-30T02:11:38.717 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.716+0000 7fead14cc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:11:38.717 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.716+0000 7fead14cc700 1 -- 172.21.15.59:0/3116410849 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feabc00b860 con 0x7feacc1418e0 2023-09-30T02:11:38.717 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.716+0000 7fead4732700 5 monclient: authenticate success, global_id 4918 2023-09-30T02:11:38.718 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.716+0000 7fead4732700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:11:38.718 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.716+0000 7fead14cc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:11:38.718 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.716+0000 7fead14cc700 1 -- 172.21.15.59:0/3116410849 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feabc00bc00 con 0x7feacc1418e0 2023-09-30T02:11:38.718 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.716+0000 7fead4732700 10 monclient: get_monmap_and_config success 2023-09-30T02:11:38.718 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.716+0000 7fead14cc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:38.718 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.716+0000 7fead4732700 4 set_mon_vals no callback set 2023-09-30T02:11:38.718 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.716+0000 7fead14cc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:11:38.718 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.716+0000 7fead14cc700 10 monclient: dump: 2023-09-30T02:11:38.718 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:38.719 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:38.719 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:38.719 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:38.719 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:38.719 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:38.719 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:38.719 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:38.719 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:38.720 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:38.723 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.717+0000 7fead4732700 10 monclient: shutdown 2023-09-30T02:11:38.723 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.717+0000 7fead4732700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:11:38.724 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.717+0000 7fead4732700 1 -- 172.21.15.59:0/3116410849 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feacc1418e0 msgr2=0x7feacc13ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:38.724 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.717+0000 7fead4732700 1 --2- 172.21.15.59:0/3116410849 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feacc1418e0 0x7feacc13ae20 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7feacc0a69c0 tx=0x7feabc009250 comp rx=0 tx=0).stop 2023-09-30T02:11:38.724 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.717+0000 7fead4732700 1 -- 172.21.15.59:0/3116410849 shutdown_connections 2023-09-30T02:11:38.724 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.717+0000 7fead4732700 1 --2- 172.21.15.59:0/3116410849 >> v2:172.21.15.59:3300/0 conn(0x7feacc0a5730 0x7feacc0a7fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:38.724 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.717+0000 7fead4732700 1 --2- 172.21.15.59:0/3116410849 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feacc1418e0 0x7feacc13ae20 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:38.725 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.717+0000 7fead4732700 1 -- 172.21.15.59:0/3116410849 >> 172.21.15.59:0/3116410849 conn(0x7feacc0631f0 msgr2=0x7feacc0635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:11:38.725 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.717+0000 7fead4732700 1 -- 172.21.15.59:0/3116410849 shutdown_connections 2023-09-30T02:11:38.725 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.717+0000 7fead4732700 1 -- 172.21.15.59:0/3116410849 wait complete. 2023-09-30T02:11:38.725 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.718+0000 7fead4732700 10 monclient: build_initial_monmap 2023-09-30T02:11:38.725 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.718+0000 7fead4732700 10 monclient: monmap: 2023-09-30T02:11:38.725 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:11:38.725 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:11:38.725 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:11:38.725 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:11:38.726 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:11:38.726 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:11:38.726 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:11:38.726 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:11:38.726 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:11:38.726 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:38.726 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.718+0000 7fead4732700 1 librados: starting msgr at 2023-09-30T02:11:38.726 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.718+0000 7fead4732700 1 librados: starting objecter 2023-09-30T02:11:38.726 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.718+0000 7fead4732700 1 Processor -- start 2023-09-30T02:11:38.726 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.718+0000 7fead4732700 1 -- start start 2023-09-30T02:11:38.727 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.718+0000 7fead4732700 1 librados: setting wanted keys 2023-09-30T02:11:38.727 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.718+0000 7fead4732700 1 librados: calling monclient init 2023-09-30T02:11:38.727 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.718+0000 7fead4732700 10 monclient: init 2023-09-30T02:11:38.727 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.719+0000 7fead4732700 10 monclient: _reopen_session rank -1 2023-09-30T02:11:38.727 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.719+0000 7fead4732700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T02:11:38.727 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.719+0000 7fead4732700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feacc0a5730 0x7feacc0a8870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:38.727 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.719+0000 7fead4732700 10 monclient(hunting): picked mon.noname-a con 0x7feacc0a5730 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:11:38.727 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.719+0000 7fead4732700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7feacc13b360 0x7feacc13eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:38.727 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.719+0000 7fead4732700 10 monclient(hunting): picked mon.noname-c con 0x7feacc13b360 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:11:38.727 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.719+0000 7fead4732700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feacc1418e0 0x7feacc14e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:38.727 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.719+0000 7fead4732700 10 monclient(hunting): picked mon.noname-b con 0x7feacc1418e0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:11:38.728 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.719+0000 7fead4732700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:38.728 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.719+0000 7fead4732700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feacc14e290 con 0x7feacc0a5730 2023-09-30T02:11:38.728 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.719+0000 7fead4732700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:38.728 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.719+0000 7fead4732700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feacc13f0e0 con 0x7feacc1418e0 2023-09-30T02:11:38.728 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.719+0000 7fead4732700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:38.728 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.719+0000 7fead4732700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feacc13dba0 con 0x7feacc13b360 2023-09-30T02:11:38.728 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.719+0000 7fead4732700 10 monclient(hunting): _renew_subs 2023-09-30T02:11:38.728 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.719+0000 7fead4732700 10 monclient(hunting): authenticate will time out at 2144.021526s 2023-09-30T02:11:38.728 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.720+0000 7fead2ccf700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feacc1418e0 0x7feacc14e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:38.729 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.720+0000 7fead1ccd700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7feacc13b360 0x7feacc13eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:38.729 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.720+0000 7fead24ce700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feacc0a5730 0x7feacc0a8870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:38.729 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.720+0000 7fead2ccf700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feacc1418e0 0x7feacc14e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:41420/0 (socket says 172.21.15.59:41420) 2023-09-30T02:11:38.729 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.720+0000 7fead2ccf700 1 -- 172.21.15.59:0/3352344146 learned_addr learned my addr 172.21.15.59:0/3352344146 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:11:38.729 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.720+0000 7fead2ccf700 10 monclient(hunting): get_auth_request con 0x7feacc1418e0 auth_method 0 2023-09-30T02:11:38.729 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.720+0000 7fead2ccf700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:38.729 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.720+0000 7fead2ccf700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:38.729 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.720+0000 7fead2ccf700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:38.729 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.720+0000 7fead24ce700 10 monclient(hunting): get_auth_request con 0x7feacc0a5730 auth_method 0 2023-09-30T02:11:38.730 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.720+0000 7fead24ce700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:38.730 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.720+0000 7fead24ce700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:38.730 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.720+0000 7fead24ce700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:38.730 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.720+0000 7fead1ccd700 10 monclient(hunting): get_auth_request con 0x7feacc13b360 auth_method 0 2023-09-30T02:11:38.730 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.720+0000 7fead1ccd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:38.730 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.720+0000 7fead1ccd700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:38.730 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.720+0000 7fead1ccd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:38.730 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.720+0000 7fead2ccf700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:38.730 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.720+0000 7fead2ccf700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:38.731 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.720+0000 7fead2ccf700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:38.731 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.721+0000 7fead1ccd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:38.731 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.721+0000 7fead1ccd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:38.731 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.721+0000 7fead1ccd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:38.731 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.721+0000 7fead24ce700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:38.731 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.721+0000 7fead24ce700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:38.731 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.721+0000 7fead24ce700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:38.731 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.723+0000 7fead2ccf700 10 monclient(hunting): handle_auth_done global_id 4921 payload 824 2023-09-30T02:11:38.731 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.723+0000 7fead2ccf700 1 -- 172.21.15.59:0/3352344146 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7feacc13b360 msgr2=0x7feacc13eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:38.731 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.723+0000 7fead2ccf700 1 --2- 172.21.15.59:0/3352344146 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7feacc13b360 0x7feacc13eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:38.732 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.723+0000 7fead2ccf700 1 -- 172.21.15.59:0/3352344146 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feacc0a5730 msgr2=0x7feacc0a8870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:38.732 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.723+0000 7fead2ccf700 1 --2- 172.21.15.59:0/3352344146 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feacc0a5730 0x7feacc0a8870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:38.732 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.723+0000 7fead2ccf700 10 monclient: _finish_hunting 0 2023-09-30T02:11:38.732 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.723+0000 7fead2ccf700 1 monclient: found mon.noname-b 2023-09-30T02:11:38.732 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.723+0000 7fead2ccf700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:11:38.732 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.723+0000 7fead2ccf700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:11:38.732 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.723+0000 7fead2ccf700 1 -- 172.21.15.59:0/3352344146 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feabc0094f0 con 0x7feacc1418e0 2023-09-30T02:11:38.733 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.726+0000 7fead2ccf700 1 --2- 172.21.15.59:0/3352344146 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feacc1418e0 0x7feacc14e6e0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7feac400c560 tx=0x7feac4007ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:38.733 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.726+0000 7feac27fc700 10 client.?.objecter ms_handle_connect 0x7feacc1418e0 2023-09-30T02:11:38.734 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.726+0000 7feac27fc700 10 client.?.objecter resend_mon_ops 2023-09-30T02:11:38.734 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.729+0000 7feac27fc700 1 -- 172.21.15.59:0/3352344146 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feac4011590 con 0x7feacc1418e0 2023-09-30T02:11:38.734 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.729+0000 7feac27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:38.734 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.729+0000 7feac27fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:11:38.734 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.729+0000 7feac27fc700 10 monclient: dump: 2023-09-30T02:11:38.734 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:38.734 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:38.734 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:38.734 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:38.735 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:38.735 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:38.735 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:38.735 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:38.735 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:38.735 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:38.735 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.729+0000 7feac27fc700 10 monclient: _finish_auth 0 2023-09-30T02:11:38.735 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.729+0000 7feac27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:11:38.735 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.729+0000 7feac27fc700 1 -- 172.21.15.59:0/3352344146 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feac40116f0 con 0x7feacc1418e0 2023-09-30T02:11:38.735 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.729+0000 7feac27fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:11:38.736 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.729+0000 7feac27fc700 1 -- 172.21.15.59:0/3352344146 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feac4011ab0 con 0x7feacc1418e0 2023-09-30T02:11:38.736 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.729+0000 7fead4732700 5 monclient: authenticate success, global_id 4921 2023-09-30T02:11:38.736 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.729+0000 7feac27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:38.736 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.729+0000 7feac27fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:11:38.736 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.729+0000 7feac27fc700 10 monclient: dump: 2023-09-30T02:11:38.736 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:38.736 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:38.736 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:38.736 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:38.736 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:38.736 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:38.737 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:38.737 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:38.737 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:38.737 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:38.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.729+0000 7fead4732700 10 monclient: _renew_subs 2023-09-30T02:11:38.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.729+0000 7fead4732700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:11:38.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.729+0000 7fead4732700 1 -- 172.21.15.59:0/3352344146 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feacc0aa340 con 0x7feacc1418e0 2023-09-30T02:11:38.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.729+0000 7fead0ccb700 4 set_mon_vals no callback set 2023-09-30T02:11:38.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.730+0000 7fead4732700 10 client.4921.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:11:38.737 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.730+0000 7fead4732700 10 monclient: _renew_subs 2023-09-30T02:11:38.738 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.730+0000 7fead4732700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:11:38.738 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.730+0000 7fead4732700 1 -- 172.21.15.59:0/3352344146 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feacc14ec20 con 0x7feacc1418e0 2023-09-30T02:11:38.738 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.730+0000 7fead4732700 1 librados: init done 2023-09-30T02:11:38.738 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.731+0000 7fead4732700 10 client.4921.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:11:38.738 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.731+0000 7fead4732700 15 client.4921.objecter _session_command_op_assign -1 1 2023-09-30T02:11:38.738 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.731+0000 7fead4732700 20 client.4921.objecter _get_session osd=-1 returning homeless 2023-09-30T02:11:38.738 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.731+0000 7fead4732700 10 client.4921.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:11:38.738 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.731+0000 7fead4732700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:11:38.738 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.731+0000 7fead4732700 1 -- 172.21.15.59:0/3352344146 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7feacc13ddc0 con 0x7feacc1418e0 2023-09-30T02:11:38.755 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.755+0000 7feac27fc700 1 -- 172.21.15.59:0/3352344146 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7feac401b3b0 con 0x7feacc1418e0 2023-09-30T02:11:38.755 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.755+0000 7feac27fc700 1 --2- 172.21.15.59:0/3352344146 >> v2:172.21.15.110:6800/115870 conn(0x7feab0068a10 0x7feab006aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:38.755 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.755+0000 7feac27fc700 1 -- 172.21.15.59:0/3352344146 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7feac4083a00 con 0x7feacc1418e0 2023-09-30T02:11:38.755 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.755+0000 7feac27fc700 10 client.4921.objecter ms_dispatch 0x7feacc001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:11:38.755 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.755+0000 7feac27fc700 3 client.4921.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:11:38.756 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.755+0000 7feac27fc700 3 client.4921.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:11:38.756 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.755+0000 7feac27fc700 10 client.4921.objecter checking command 1 2023-09-30T02:11:38.756 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.755+0000 7feac27fc700 1 --2- 172.21.15.59:0/3352344146 >> v2:172.21.15.110:6801/116560 conn(0x7feab006e980 0x7feab0070d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:38.756 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.755+0000 7fead24ce700 1 --2- 172.21.15.59:0/3352344146 >> v2:172.21.15.110:6800/115870 conn(0x7feab0068a10 0x7feab006aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:38.756 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.755+0000 7feac27fc700 20 client.4921.objecter _get_session s=0x7feab006e2d0 osd=4 3 2023-09-30T02:11:38.756 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.755+0000 7feac27fc700 20 client.4921.objecter put_session s=0x7feab006e2d0 osd=4 3 2023-09-30T02:11:38.756 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.755+0000 7feac27fc700 15 client.4921.objecter _session_command_op_remove -1 1 2023-09-30T02:11:38.756 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.755+0000 7feac27fc700 20 client.4921.objecter _get_session s=0x7feab006e2d0 osd=4 3 2023-09-30T02:11:38.756 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.755+0000 7feac27fc700 20 client.4921.objecter get_session s=0x7feab006e2d0 osd=4 3 2023-09-30T02:11:38.756 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.755+0000 7feac27fc700 15 client.4921.objecter _session_command_op_assign 4 1 2023-09-30T02:11:38.757 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.755+0000 7feac27fc700 20 client.4921.objecter put_session s=0x7feab006e2d0 osd=4 4 2023-09-30T02:11:38.757 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.755+0000 7feac27fc700 10 client.4921.objecter _send_command 1 2023-09-30T02:11:38.757 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.755+0000 7fead24ce700 10 monclient: get_auth_request con 0x7feab0068a10 auth_method 0 2023-09-30T02:11:38.757 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.755+0000 7feac27fc700 1 -- 172.21.15.59:0/3352344146 --> v2:172.21.15.110:6801/116560 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7feab0071400 con 0x7feab006e980 2023-09-30T02:11:38.757 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.755+0000 7feac27fc700 20 client.4921.objecter dump_active .. 0 homeless 2023-09-30T02:11:38.757 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.755+0000 7feac27fc700 1 -- 172.21.15.59:0/3352344146 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7feac40512a0 con 0x7feacc1418e0 2023-09-30T02:11:38.757 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.755+0000 7feac27fc700 10 monclient: handle_get_version_reply finishing 1 version 25 2023-09-30T02:11:38.757 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.756+0000 7fead24ce700 1 --2- 172.21.15.59:0/3352344146 >> v2:172.21.15.110:6800/115870 conn(0x7feab0068a10 0x7feab006aec0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7feac8009fd0 tx=0x7feac800b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:38.757 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.756+0000 7feac27fc700 10 client.4921.objecter ms_handle_connect 0x7feab0068a10 2023-09-30T02:11:38.757 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.756+0000 7fead1ccd700 1 --2- 172.21.15.59:0/3352344146 >> v2:172.21.15.110:6801/116560 conn(0x7feab006e980 0x7feab0070d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:38.758 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.756+0000 7fead1ccd700 10 monclient: get_auth_request con 0x7feab006e980 auth_method 0 2023-09-30T02:11:38.758 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.759+0000 7fead1ccd700 1 --2- 172.21.15.59:0/3352344146 >> v2:172.21.15.110:6801/116560 conn(0x7feab006e980 0x7feab0070d90 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:38.758 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.759+0000 7feac27fc700 10 client.4921.objecter ms_handle_connect 0x7feab006e980 2023-09-30T02:11:38.781 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.781+0000 7feac27fc700 1 -- 172.21.15.59:0/3352344146 <== osd.4 v2:172.21.15.110:6801/116560 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7feab0071400 con 0x7feab006e980 2023-09-30T02:11:38.781 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.781+0000 7feac27fc700 10 client.4921.objecter ms_dispatch 0x7feacc001550 command_reply(tid 1: 0 ) v1 2023-09-30T02:11:38.781 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.781+0000 7feac27fc700 10 client.4921.objecter _finish_command 1 = system:0 2023-09-30T02:11:38.781 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.781+0000 7feac27fc700 20 client.4921.objecter put_session s=0x7feab006e2d0 osd=4 4 2023-09-30T02:11:38.781 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.781+0000 7feac27fc700 15 client.4921.objecter _session_command_op_remove 4 1 2023-09-30T02:11:38.795 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.796+0000 7fead4732700 10 client.4921.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-09-30T02:11:38.796 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.796+0000 7fead4732700 15 client.4921.objecter _session_command_op_assign -1 2 2023-09-30T02:11:38.796 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.796+0000 7fead4732700 20 client.4921.objecter _get_session s=0x7feab006e2d0 osd=4 3 2023-09-30T02:11:38.796 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.796+0000 7fead4732700 20 client.4921.objecter put_session s=0x7feab006e2d0 osd=4 3 2023-09-30T02:11:38.796 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.796+0000 7fead4732700 20 client.4921.objecter _get_session s=0x7feab006e2d0 osd=4 3 2023-09-30T02:11:38.796 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.796+0000 7fead4732700 15 client.4921.objecter _session_command_op_remove -1 2 2023-09-30T02:11:38.797 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.796+0000 7fead4732700 20 client.4921.objecter get_session s=0x7feab006e2d0 osd=4 3 2023-09-30T02:11:38.797 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.796+0000 7fead4732700 15 client.4921.objecter _session_command_op_assign 4 2 2023-09-30T02:11:38.797 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.796+0000 7fead4732700 20 client.4921.objecter put_session s=0x7feab006e2d0 osd=4 4 2023-09-30T02:11:38.797 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.796+0000 7fead4732700 10 client.4921.objecter _send_command 2 2023-09-30T02:11:38.797 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.796+0000 7fead4732700 1 -- 172.21.15.59:0/3352344146 --> v2:172.21.15.110:6801/116560 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7feacc0a7fb0 con 0x7feab006e980 2023-09-30T02:11:38.817 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.818+0000 7feac27fc700 1 -- 172.21.15.59:0/3352344146 <== osd.4 v2:172.21.15.110:6801/116560 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7feacc0a7fb0 con 0x7feab006e980 2023-09-30T02:11:38.817 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.818+0000 7feac27fc700 10 client.4921.objecter ms_dispatch 0x7feacc001550 command_reply(tid 2: 0 ) v1 2023-09-30T02:11:38.817 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.818+0000 7feac27fc700 10 client.4921.objecter _finish_command 2 = system:0 2023-09-30T02:11:38.817 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.818+0000 7feac27fc700 20 client.4921.objecter put_session s=0x7feab006e2d0 osd=4 4 2023-09-30T02:11:38.818 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.818+0000 7feac27fc700 15 client.4921.objecter _session_command_op_remove 4 2 2023-09-30T02:11:38.818 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2023-09-30T02:11:38.818 INFO:teuthology.orchestra.run.smithi059.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T02:11:38.818 INFO:teuthology.orchestra.run.smithi059.stdout:} 2023-09-30T02:11:38.819 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.818+0000 7fead4732700 10 librados: watch_flush enter 2023-09-30T02:11:38.819 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.818+0000 7fead4732700 10 librados: watch_flush exit 2023-09-30T02:11:38.819 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.818+0000 7fead4732700 10 client.4921.objecter close_session for osd.4 2023-09-30T02:11:38.820 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.818+0000 7fead4732700 1 -- 172.21.15.59:0/3352344146 >> v2:172.21.15.110:6801/116560 conn(0x7feab006e980 msgr2=0x7feab0070d90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:38.820 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.818+0000 7fead4732700 1 --2- 172.21.15.59:0/3352344146 >> v2:172.21.15.110:6801/116560 conn(0x7feab006e980 0x7feab0070d90 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:38.820 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.818+0000 7fead4732700 20 client.4921.objecter put_session s=0x7feab006e2d0 osd=4 1 2023-09-30T02:11:38.820 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.818+0000 7fead4732700 20 client.4921.objecter shutdown clearing up homeless session... 2023-09-30T02:11:38.820 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.818+0000 7fead4732700 10 client.4921.objecter successfully canceled tick 2023-09-30T02:11:38.820 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.818+0000 7fead4732700 1 -- 172.21.15.59:0/3352344146 >> v2:172.21.15.110:6800/115870 conn(0x7feab0068a10 msgr2=0x7feab006aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:38.820 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.818+0000 7fead4732700 1 --2- 172.21.15.59:0/3352344146 >> v2:172.21.15.110:6800/115870 conn(0x7feab0068a10 0x7feab006aec0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7feac8009fd0 tx=0x7feac800b040 comp rx=0 tx=0).stop 2023-09-30T02:11:38.820 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.818+0000 7fead4732700 10 monclient: shutdown 2023-09-30T02:11:38.820 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.818+0000 7fead4732700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:11:38.820 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.818+0000 7fead4732700 1 -- 172.21.15.59:0/3352344146 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feacc1418e0 msgr2=0x7feacc14e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:38.820 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.818+0000 7fead4732700 1 --2- 172.21.15.59:0/3352344146 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7feacc1418e0 0x7feacc14e6e0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7feac400c560 tx=0x7feac4007ef0 comp rx=0 tx=0).stop 2023-09-30T02:11:38.821 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.818+0000 7fead2ccf700 1 -- 172.21.15.59:0/3352344146 reap_dead start 2023-09-30T02:11:38.821 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.818+0000 7fead4732700 1 -- 172.21.15.59:0/3352344146 shutdown_connections 2023-09-30T02:11:38.821 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.818+0000 7fead4732700 1 -- 172.21.15.59:0/3352344146 >> 172.21.15.59:0/3352344146 conn(0x7feacc0631f0 msgr2=0x7feacc1392e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:11:38.821 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.818+0000 7fead4732700 1 -- 172.21.15.59:0/3352344146 shutdown_connections 2023-09-30T02:11:38.821 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.818+0000 7fead4732700 1 -- 172.21.15.59:0/3352344146 wait complete. 2023-09-30T02:11:38.821 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.818+0000 7fead4732700 1 librados: shutdown 2023-09-30T02:11:38.832 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2023-09-30T02:11:38.989 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.989+0000 7f1b42dd4700 10 monclient: get_monmap_and_config 2023-09-30T02:11:38.989 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.989+0000 7f1b42dd4700 10 monclient: build_initial_monmap 2023-09-30T02:11:38.989 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.989+0000 7f1b42dd4700 10 monclient: monmap: 2023-09-30T02:11:38.989 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:11:38.989 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:38.989 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:11:38.989968+0000 2023-09-30T02:11:38.989 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:11:38.989968+0000 2023-09-30T02:11:38.989 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:11:38.990 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:38.990 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:11:38.990 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:11:38.990 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:11:38.990 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:11:38.990 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:11:38.990 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:38.991 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.990+0000 7f1b42dd4700 1 Processor -- start 2023-09-30T02:11:38.991 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.990+0000 7f1b42dd4700 1 -- start start 2023-09-30T02:11:38.991 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.990+0000 7f1b42dd4700 10 monclient: init 2023-09-30T02:11:38.991 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.991+0000 7f1b42dd4700 10 monclient: _reopen_session rank -1 2023-09-30T02:11:38.991 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.991+0000 7f1b42dd4700 10 monclient: _add_conns ranks=[3,0,2,4,1] 2023-09-30T02:11:38.992 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.991+0000 7f1b42dd4700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f1b3c0a5b70 0x7f1b3c0b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:38.992 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.991+0000 7f1b42dd4700 10 monclient(hunting): picked mon.noname-a con 0x7f1b3c0a5b70 addr v2:172.21.15.59:3300/0 2023-09-30T02:11:38.992 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.991+0000 7f1b42dd4700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1b3c0a5f40 addr v1:172.21.15.59:6789/0 2023-09-30T02:11:38.992 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.991+0000 7f1b42dd4700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1b3c142290 0x7f1b3c13af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:38.992 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.991+0000 7f1b42dd4700 10 monclient(hunting): picked mon.noname-b con 0x7f1b3c142290 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:11:38.992 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.991+0000 7f1b42dd4700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b3c0a75a0 con 0x7f1b3c0a5f40 2023-09-30T02:11:38.992 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.991+0000 7f1b42dd4700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:38.992 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.991+0000 7f1b42dd4700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b3c1426d0 con 0x7f1b3c0a5b70 2023-09-30T02:11:38.992 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.991+0000 7f1b42dd4700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:38.992 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.991+0000 7f1b42dd4700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b3c0a63e0 con 0x7f1b3c142290 2023-09-30T02:11:38.992 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.991+0000 7f1b42dd4700 10 monclient(hunting): _renew_subs 2023-09-30T02:11:38.993 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.991+0000 7f1b42dd4700 10 monclient(hunting): authenticate will time out at 2144.293553s 2023-09-30T02:11:38.994 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.995+0000 7f1b40b70700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f1b3c0a5b70 0x7f1b3c0b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:38.995 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.996+0000 7f1b3bfff700 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f1b3c0a5f40 0x7f1b3c141b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:49448/0 (socket says 172.21.15.59:49448) 2023-09-30T02:11:38.995 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.996+0000 7f1b3bfff700 1 -- 172.21.15.59:0/3730312145 learned_addr learned my addr 172.21.15.59:0/3730312145 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:11:38.996 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.996+0000 7f1b40b70700 10 monclient(hunting): get_auth_request con 0x7f1b3c0a5b70 auth_method 0 2023-09-30T02:11:38.996 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.996+0000 7f1b40b70700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:38.996 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.996+0000 7f1b40b70700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:38.996 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.996+0000 7f1b40b70700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:38.996 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.997+0000 7f1b40b70700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:38.996 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.997+0000 7f1b40b70700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:38.996 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:38.997+0000 7f1b40b70700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:39.002 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.003+0000 7f1b40b70700 10 monclient(hunting): handle_auth_done global_id 4893 payload 274 2023-09-30T02:11:39.002 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.003+0000 7f1b40b70700 1 -- 172.21.15.59:0/3730312145 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1b3c142290 msgr2=0x7f1b3c13af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:11:39.002 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.003+0000 7f1b40b70700 1 --2- 172.21.15.59:0/3730312145 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1b3c142290 0x7f1b3c13af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:39.002 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.003+0000 7f1b40b70700 1 -- 172.21.15.59:0/3730312145 >> v1:172.21.15.59:6789/0 conn(0x7f1b3c0a5f40 legacy=0x7f1b3c141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:39.002 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.003+0000 7f1b40b70700 10 monclient: _finish_hunting 0 2023-09-30T02:11:39.003 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.003+0000 7f1b40b70700 1 monclient: found mon.noname-a 2023-09-30T02:11:39.003 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.003+0000 7f1b40b70700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:11:39.003 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.003+0000 7f1b40b70700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.59:3300/0 2023-09-30T02:11:39.003 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.003+0000 7f1b40b70700 1 -- 172.21.15.59:0/3730312145 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b3c0af3f0 con 0x7f1b3c0a5b70 2023-09-30T02:11:39.007 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.007+0000 7f1b40b70700 1 --2- 172.21.15.59:0/3730312145 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b3c0a5b70 0x7f1b3c0b1500 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1b30009910 tx=0x7f1b300091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=481e33ee4d4f1d5e server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:39.014 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.015+0000 7f1b3b7fe700 1 -- 172.21.15.59:0/3730312145 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b3000b820 con 0x7f1b3c0a5b70 2023-09-30T02:11:39.014 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.015+0000 7f1b3b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:39.014 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.015+0000 7f1b3b7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T02:11:39.015 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.015+0000 7f1b3b7fe700 10 monclient: dump: 2023-09-30T02:11:39.015 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:39.015 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:39.015 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:39.015 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:39.015 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:39.015 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:39.015 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:39.015 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:39.015 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:39.016 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:39.016 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.015+0000 7f1b3b7fe700 10 monclient: _finish_auth 0 2023-09-30T02:11:39.016 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.015+0000 7f1b3b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:11:39.016 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.015+0000 7f1b3b7fe700 1 -- 172.21.15.59:0/3730312145 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b3000b980 con 0x7f1b3c0a5b70 2023-09-30T02:11:39.016 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.015+0000 7f1b42dd4700 5 monclient: authenticate success, global_id 4893 2023-09-30T02:11:39.016 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.015+0000 7f1b42dd4700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:11:39.016 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.015+0000 7f1b3b7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:11:39.016 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.015+0000 7f1b3b7fe700 1 -- 172.21.15.59:0/3730312145 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b3000bd20 con 0x7f1b3c0a5b70 2023-09-30T02:11:39.016 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.015+0000 7f1b42dd4700 10 monclient: get_monmap_and_config success 2023-09-30T02:11:39.016 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.015+0000 7f1b42dd4700 4 set_mon_vals no callback set 2023-09-30T02:11:39.016 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.015+0000 7f1b3b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:39.017 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.015+0000 7f1b3b7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T02:11:39.017 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.015+0000 7f1b3b7fe700 10 monclient: dump: 2023-09-30T02:11:39.017 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:39.017 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:39.017 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:39.017 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:39.017 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:39.017 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:39.017 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:39.017 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:39.017 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:39.017 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:39.020 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.015+0000 7f1b42dd4700 10 monclient: shutdown 2023-09-30T02:11:39.020 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.015+0000 7f1b42dd4700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:11:39.020 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.015+0000 7f1b42dd4700 1 -- 172.21.15.59:0/3730312145 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b3c0a5b70 msgr2=0x7f1b3c0b1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:39.020 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.015+0000 7f1b42dd4700 1 --2- 172.21.15.59:0/3730312145 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b3c0a5b70 0x7f1b3c0b1500 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f1b30009910 tx=0x7f1b300091a0 comp rx=0 tx=0).stop 2023-09-30T02:11:39.020 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.016+0000 7f1b42dd4700 1 -- 172.21.15.59:0/3730312145 shutdown_connections 2023-09-30T02:11:39.021 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.016+0000 7f1b42dd4700 1 --2- 172.21.15.59:0/3730312145 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b3c0a5b70 0x7f1b3c0b1500 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:39.021 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.016+0000 7f1b42dd4700 1 --2- 172.21.15.59:0/3730312145 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1b3c142290 0x7f1b3c13af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:39.021 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.016+0000 7f1b42dd4700 1 -- 172.21.15.59:0/3730312145 >> 172.21.15.59:0/3730312145 conn(0x7f1b3c0631f0 msgr2=0x7f1b3c0635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:11:39.021 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.016+0000 7f1b42dd4700 1 -- 172.21.15.59:0/3730312145 shutdown_connections 2023-09-30T02:11:39.021 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.016+0000 7f1b42dd4700 1 -- 172.21.15.59:0/3730312145 wait complete. 2023-09-30T02:11:39.021 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.017+0000 7f1b42dd4700 10 monclient: build_initial_monmap 2023-09-30T02:11:39.021 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.017+0000 7f1b42dd4700 10 monclient: monmap: 2023-09-30T02:11:39.021 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:11:39.021 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:11:39.021 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:11:39.021 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:11:39.021 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:11:39.022 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:11:39.022 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:11:39.022 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:11:39.022 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:11:39.022 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:39.022 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.017+0000 7f1b42dd4700 1 librados: starting msgr at 2023-09-30T02:11:39.022 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.017+0000 7f1b42dd4700 1 librados: starting objecter 2023-09-30T02:11:39.022 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.017+0000 7f1b42dd4700 1 Processor -- start 2023-09-30T02:11:39.022 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.017+0000 7f1b42dd4700 1 -- start start 2023-09-30T02:11:39.022 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.017+0000 7f1b42dd4700 1 librados: setting wanted keys 2023-09-30T02:11:39.022 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.017+0000 7f1b42dd4700 1 librados: calling monclient init 2023-09-30T02:11:39.022 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.017+0000 7f1b42dd4700 10 monclient: init 2023-09-30T02:11:39.023 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.018+0000 7f1b42dd4700 10 monclient: _reopen_session rank -1 2023-09-30T02:11:39.023 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.018+0000 7f1b42dd4700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T02:11:39.023 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.018+0000 7f1b42dd4700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1b3c0a5b70 0x7f1b3c0b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:39.023 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.018+0000 7f1b42dd4700 10 monclient(hunting): picked mon.noname-b con 0x7f1b3c0a5b70 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:11:39.023 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.018+0000 7f1b42dd4700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1b3c0a5f40 0x7f1b3c1512d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:39.023 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.018+0000 7f1b42dd4700 10 monclient(hunting): picked mon.noname-c con 0x7f1b3c0a5f40 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:11:39.023 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.018+0000 7f1b42dd4700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b3c142290 0x7f1b3c151810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:39.023 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.018+0000 7f1b42dd4700 10 monclient(hunting): picked mon.noname-a con 0x7f1b3c142290 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:11:39.023 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.018+0000 7f1b42dd4700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:39.023 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.018+0000 7f1b42dd4700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b3c0abd50 con 0x7f1b3c142290 2023-09-30T02:11:39.023 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.018+0000 7f1b42dd4700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:39.023 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.018+0000 7f1b42dd4700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b3c0aa720 con 0x7f1b3c0a5b70 2023-09-30T02:11:39.024 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.018+0000 7f1b42dd4700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:39.024 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.018+0000 7f1b42dd4700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b3c151d50 con 0x7f1b3c0a5f40 2023-09-30T02:11:39.024 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.018+0000 7f1b42dd4700 10 monclient(hunting): _renew_subs 2023-09-30T02:11:39.024 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.018+0000 7f1b42dd4700 10 monclient(hunting): authenticate will time out at 2144.320712s 2023-09-30T02:11:39.024 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.019+0000 7f1b40b70700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1b3c0a5b70 0x7f1b3c0b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:39.024 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.019+0000 7f1b41371700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b3c142290 0x7f1b3c151810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:39.024 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.019+0000 7f1b40b70700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1b3c0a5b70 0x7f1b3c0b4ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:41432/0 (socket says 172.21.15.59:41432) 2023-09-30T02:11:39.024 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.019+0000 7f1b40b70700 1 -- 172.21.15.59:0/3553478319 learned_addr learned my addr 172.21.15.59:0/3553478319 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:11:39.024 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.019+0000 7f1b40b70700 10 monclient(hunting): get_auth_request con 0x7f1b3c0a5b70 auth_method 0 2023-09-30T02:11:39.024 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.019+0000 7f1b40b70700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:39.024 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.019+0000 7f1b40b70700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:39.024 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.019+0000 7f1b40b70700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:39.025 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.020+0000 7f1b41371700 10 monclient(hunting): get_auth_request con 0x7f1b3c142290 auth_method 0 2023-09-30T02:11:39.025 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.020+0000 7f1b41371700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:39.025 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.020+0000 7f1b41371700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:39.025 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.020+0000 7f1b41371700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:39.025 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.020+0000 7f1b40b70700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:39.025 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.020+0000 7f1b40b70700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:39.025 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.020+0000 7f1b40b70700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:39.025 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.020+0000 7f1b41371700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:39.025 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.020+0000 7f1b41371700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:39.025 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.020+0000 7f1b41371700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:39.025 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.020+0000 7f1b3bfff700 1 --2- 172.21.15.59:0/3553478319 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1b3c0a5f40 0x7f1b3c1512d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:39.025 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.021+0000 7f1b3bfff700 10 monclient(hunting): get_auth_request con 0x7f1b3c0a5f40 auth_method 0 2023-09-30T02:11:39.025 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.021+0000 7f1b3bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:39.026 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.021+0000 7f1b3bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:39.026 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.021+0000 7f1b3bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:39.027 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.021+0000 7f1b3bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:39.027 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.021+0000 7f1b3bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:39.027 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.021+0000 7f1b3bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:39.027 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.023+0000 7f1b40b70700 10 monclient(hunting): handle_auth_done global_id 4924 payload 824 2023-09-30T02:11:39.027 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.023+0000 7f1b40b70700 1 -- 172.21.15.59:0/3553478319 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1b3c0a5f40 msgr2=0x7f1b3c1512d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:39.027 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.023+0000 7f1b40b70700 1 --2- 172.21.15.59:0/3553478319 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1b3c0a5f40 0x7f1b3c1512d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:39.027 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.023+0000 7f1b40b70700 1 -- 172.21.15.59:0/3553478319 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b3c142290 msgr2=0x7f1b3c151810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:39.027 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.023+0000 7f1b40b70700 1 --2- 172.21.15.59:0/3553478319 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b3c142290 0x7f1b3c151810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:39.027 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.023+0000 7f1b40b70700 10 monclient: _finish_hunting 0 2023-09-30T02:11:39.028 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.023+0000 7f1b40b70700 1 monclient: found mon.noname-b 2023-09-30T02:11:39.028 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.023+0000 7f1b40b70700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:11:39.028 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.023+0000 7f1b40b70700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:11:39.028 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.023+0000 7f1b40b70700 1 -- 172.21.15.59:0/3553478319 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b30009420 con 0x7f1b3c0a5b70 2023-09-30T02:11:39.028 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.025+0000 7f1b40b70700 1 --2- 172.21.15.59:0/3553478319 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1b3c0a5b70 0x7f1b3c0b4ea0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f1b3000b3c0 tx=0x7f1b30017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:39.028 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.025+0000 7f1b38ff9700 10 client.?.objecter ms_handle_connect 0x7f1b3c0a5b70 2023-09-30T02:11:39.028 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.026+0000 7f1b38ff9700 10 client.?.objecter resend_mon_ops 2023-09-30T02:11:39.029 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.029+0000 7f1b38ff9700 1 -- 172.21.15.59:0/3553478319 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b30011e30 con 0x7f1b3c0a5b70 2023-09-30T02:11:39.029 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.029+0000 7f1b38ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:39.029 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.029+0000 7f1b38ff9700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:11:39.029 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.029+0000 7f1b38ff9700 10 monclient: dump: 2023-09-30T02:11:39.029 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:39.029 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:39.030 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:39.030 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:39.030 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:39.030 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:39.030 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:39.030 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:39.030 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:39.030 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:39.030 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.029+0000 7f1b38ff9700 10 monclient: _finish_auth 0 2023-09-30T02:11:39.030 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.029+0000 7f1b38ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:11:39.030 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.029+0000 7f1b38ff9700 1 -- 172.21.15.59:0/3553478319 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b30024440 con 0x7f1b3c0a5b70 2023-09-30T02:11:39.030 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.029+0000 7f1b42dd4700 5 monclient: authenticate success, global_id 4924 2023-09-30T02:11:39.031 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.029+0000 7f1b42dd4700 10 monclient: _renew_subs 2023-09-30T02:11:39.031 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.029+0000 7f1b42dd4700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:11:39.031 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.029+0000 7f1b42dd4700 1 -- 172.21.15.59:0/3553478319 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b3c0ae9f0 con 0x7f1b3c0a5b70 2023-09-30T02:11:39.031 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.029+0000 7f1b38ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:11:39.031 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.029+0000 7f1b38ff9700 1 -- 172.21.15.59:0/3553478319 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b300247e0 con 0x7f1b3c0a5b70 2023-09-30T02:11:39.031 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.029+0000 7f1b38ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:39.031 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.029+0000 7f1b38ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:11:39.031 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.029+0000 7f1b38ff9700 10 monclient: dump: 2023-09-30T02:11:39.031 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:39.031 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:39.031 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:39.031 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:39.032 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:39.032 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:39.032 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:39.032 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:39.032 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:39.032 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:39.032 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.029+0000 7f1b3affd700 4 set_mon_vals no callback set 2023-09-30T02:11:39.032 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.029+0000 7f1b42dd4700 10 client.4924.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:11:39.032 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.029+0000 7f1b42dd4700 10 monclient: _renew_subs 2023-09-30T02:11:39.032 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.029+0000 7f1b42dd4700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:11:39.032 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.029+0000 7f1b42dd4700 1 -- 172.21.15.59:0/3553478319 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b3c0a9b90 con 0x7f1b3c0a5b70 2023-09-30T02:11:39.032 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.029+0000 7f1b42dd4700 1 librados: init done 2023-09-30T02:11:39.033 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.031+0000 7f1b42dd4700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:11:39.033 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.031+0000 7f1b42dd4700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:11:39.033 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.031+0000 7f1b42dd4700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:11:39.033 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.031+0000 7f1b42dd4700 1 -- 172.21.15.59:0/3553478319 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b3c0a9d80 con 0x7f1b3c0a5b70 2023-09-30T02:11:39.049 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.049+0000 7f1b38ff9700 1 -- 172.21.15.59:0/3553478319 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f1b3002d440 con 0x7f1b3c0a5b70 2023-09-30T02:11:39.049 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.050+0000 7f1b38ff9700 1 --2- 172.21.15.59:0/3553478319 >> v2:172.21.15.110:6800/115870 conn(0x7f1b20068960 0x7f1b2006ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:39.049 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.050+0000 7f1b3bfff700 1 --2- 172.21.15.59:0/3553478319 >> v2:172.21.15.110:6800/115870 conn(0x7f1b20068960 0x7f1b2006ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:39.050 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.050+0000 7f1b3bfff700 10 monclient: get_auth_request con 0x7f1b20068960 auth_method 0 2023-09-30T02:11:39.050 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.051+0000 7f1b3bfff700 1 --2- 172.21.15.59:0/3553478319 >> v2:172.21.15.110:6800/115870 conn(0x7f1b20068960 0x7f1b2006ae10 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c0a8ea0 tx=0x7f1b28007eb0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:39.050 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.051+0000 7f1b38ff9700 10 client.4924.objecter ms_handle_connect 0x7f1b20068960 2023-09-30T02:11:39.141 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.141+0000 7f1b38ff9700 1 -- 172.21.15.59:0/3553478319 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f1b30057800 con 0x7f1b3c0a5b70 2023-09-30T02:11:39.141 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.141+0000 7f1b38ff9700 10 client.4924.objecter ms_dispatch 0x7f1b3c001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:11:39.141 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.141+0000 7f1b38ff9700 3 client.4924.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:11:39.141 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.141+0000 7f1b38ff9700 3 client.4924.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:11:39.141 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.141+0000 7f1b38ff9700 20 client.4924.objecter dump_active .. 0 homeless 2023-09-30T02:11:39.307 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.308+0000 7f1b38ff9700 1 -- 172.21.15.59:0/3553478319 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1b30053440 con 0x7f1b3c0a5b70 2023-09-30T02:11:39.307 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.308+0000 7f1b38ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:11:39.307 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.308+0000 7f1b38ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:11:39.527 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.528+0000 7f1b42dd4700 1 -- 172.21.15.59:0/3553478319 --> v2:172.21.15.110:6800/115870 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f1b3c0aa860 con 0x7f1b20068960 2023-09-30T02:11:39.527 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.528+0000 7f1b38ff9700 1 -- 172.21.15.59:0/3553478319 <== mgr.4108 v2:172.21.15.110:6800/115870 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f1b280013b0 con 0x7f1b20068960 2023-09-30T02:11:39.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.530+0000 7f1b42dd4700 10 librados: watch_flush enter 2023-09-30T02:11:39.530 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.530+0000 7f1b42dd4700 10 librados: watch_flush exit 2023-09-30T02:11:39.531 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.530+0000 7f1b42dd4700 20 client.4924.objecter shutdown clearing up homeless session... 2023-09-30T02:11:39.531 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.530+0000 7f1b42dd4700 10 client.4924.objecter successfully canceled tick 2023-09-30T02:11:39.531 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.530+0000 7f1b42dd4700 1 -- 172.21.15.59:0/3553478319 >> v2:172.21.15.110:6800/115870 conn(0x7f1b20068960 msgr2=0x7f1b2006ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:39.531 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.530+0000 7f1b42dd4700 1 --2- 172.21.15.59:0/3553478319 >> v2:172.21.15.110:6800/115870 conn(0x7f1b20068960 0x7f1b2006ae10 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c0a8ea0 tx=0x7f1b28007eb0 comp rx=0 tx=0).stop 2023-09-30T02:11:39.531 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.530+0000 7f1b42dd4700 10 monclient: shutdown 2023-09-30T02:11:39.531 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.530+0000 7f1b42dd4700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:11:39.531 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.530+0000 7f1b42dd4700 1 -- 172.21.15.59:0/3553478319 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1b3c0a5b70 msgr2=0x7f1b3c0b4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:39.531 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.530+0000 7f1b42dd4700 1 --2- 172.21.15.59:0/3553478319 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1b3c0a5b70 0x7f1b3c0b4ea0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f1b3000b3c0 tx=0x7f1b30017000 comp rx=0 tx=0).stop 2023-09-30T02:11:39.531 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.530+0000 7f1b42dd4700 1 -- 172.21.15.59:0/3553478319 shutdown_connections 2023-09-30T02:11:39.531 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.530+0000 7f1b42dd4700 1 --2- 172.21.15.59:0/3553478319 >> v2:172.21.15.110:6800/115870 conn(0x7f1b20068960 0x7f1b2006ae10 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:39.531 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.530+0000 7f1b42dd4700 1 --2- 172.21.15.59:0/3553478319 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f1b3c0a5b70 0x7f1b3c0b4ea0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:39.532 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.530+0000 7f1b42dd4700 1 --2- 172.21.15.59:0/3553478319 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b3c142290 0x7f1b3c151810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:39.532 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.530+0000 7f1b42dd4700 1 --2- 172.21.15.59:0/3553478319 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f1b3c0a5f40 0x7f1b3c1512d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:39.532 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.530+0000 7f1b42dd4700 1 -- 172.21.15.59:0/3553478319 >> 172.21.15.59:0/3553478319 conn(0x7f1b3c0631f0 msgr2=0x7f1b3c141b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:11:39.532 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.531+0000 7f1b42dd4700 1 -- 172.21.15.59:0/3553478319 shutdown_connections 2023-09-30T02:11:39.532 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.531+0000 7f1b42dd4700 1 -- 172.21.15.59:0/3553478319 wait complete. 2023-09-30T02:11:39.532 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.531+0000 7f1b42dd4700 1 librados: shutdown 2023-09-30T02:11:39.532 INFO:teuthology.orchestra.run.smithi059.stderr:instructed osd(s) 4 to deep-scrub 2023-09-30T02:11:39.544 INFO:tasks.ceph:Scrubbing osd.5 2023-09-30T02:11:39.545 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2023-09-30T02:11:39.680 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.681+0000 7fdb4efe9700 10 monclient: get_monmap_and_config 2023-09-30T02:11:39.681 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.681+0000 7fdb4efe9700 10 monclient: build_initial_monmap 2023-09-30T02:11:39.681 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.681+0000 7fdb4efe9700 10 monclient: monmap: 2023-09-30T02:11:39.681 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:11:39.681 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:39.681 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:11:39.681869+0000 2023-09-30T02:11:39.681 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:11:39.681869+0000 2023-09-30T02:11:39.681 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:11:39.681 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:39.682 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:11:39.682 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:11:39.682 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:11:39.682 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:11:39.682 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:11:39.682 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:39.684 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.682+0000 7fdb4efe9700 1 Processor -- start 2023-09-30T02:11:39.684 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.682+0000 7fdb4efe9700 1 -- start start 2023-09-30T02:11:39.684 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.682+0000 7fdb4efe9700 10 monclient: init 2023-09-30T02:11:39.684 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.682+0000 7fdb4efe9700 10 monclient: _reopen_session rank -1 2023-09-30T02:11:39.685 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.682+0000 7fdb4efe9700 10 monclient: _add_conns ranks=[3,4,2,1,0] 2023-09-30T02:11:39.685 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.682+0000 7fdb4efe9700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fdb480a5730 0x7fdb480a7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:39.685 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.682+0000 7fdb4efe9700 10 monclient(hunting): picked mon.noname-a con 0x7fdb480a5730 addr v2:172.21.15.59:3300/0 2023-09-30T02:11:39.685 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.682+0000 7fdb4efe9700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fdb481418e0 0x7fdb4813ae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:39.685 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.683+0000 7fdb4efe9700 10 monclient(hunting): picked mon.noname-c con 0x7fdb481418e0 addr v2:172.21.15.110:3300/0 2023-09-30T02:11:39.685 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.683+0000 7fdb4efe9700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdb4813b360 0x7fdb4813b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:39.685 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.683+0000 7fdb4efe9700 10 monclient(hunting): picked mon.noname-b con 0x7fdb4813b360 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:11:39.685 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.683+0000 7fdb4efe9700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:39.685 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.683+0000 7fdb4efe9700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb480a7410 con 0x7fdb480a5730 2023-09-30T02:11:39.685 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.683+0000 7fdb4efe9700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:39.685 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.683+0000 7fdb4efe9700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb480a84e0 con 0x7fdb481418e0 2023-09-30T02:11:39.686 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.683+0000 7fdb4efe9700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:39.686 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.683+0000 7fdb4efe9700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb480a7b10 con 0x7fdb4813b360 2023-09-30T02:11:39.686 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.683+0000 7fdb4efe9700 10 monclient(hunting): _renew_subs 2023-09-30T02:11:39.686 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.683+0000 7fdb4efe9700 10 monclient(hunting): authenticate will time out at 2144.985045s 2023-09-30T02:11:39.686 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.683+0000 7fdb4d586700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdb4813b360 0x7fdb4813b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:39.686 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.683+0000 7fdb47fff700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fdb481418e0 0x7fdb4813ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:39.686 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.683+0000 7fdb4cd85700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fdb480a5730 0x7fdb480a7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:39.686 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.684+0000 7fdb4d586700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdb4813b360 0x7fdb4813b7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:46192/0 (socket says 172.21.15.59:46192) 2023-09-30T02:11:39.686 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.684+0000 7fdb4d586700 1 -- 172.21.15.59:0/2352539725 learned_addr learned my addr 172.21.15.59:0/2352539725 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:11:39.686 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.684+0000 7fdb4d586700 10 monclient(hunting): get_auth_request con 0x7fdb4813b360 auth_method 0 2023-09-30T02:11:39.686 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.684+0000 7fdb4d586700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:39.687 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.684+0000 7fdb4d586700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:39.687 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.684+0000 7fdb4d586700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:39.687 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.684+0000 7fdb47fff700 10 monclient(hunting): get_auth_request con 0x7fdb481418e0 auth_method 0 2023-09-30T02:11:39.687 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.684+0000 7fdb47fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:39.687 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.684+0000 7fdb47fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:39.687 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.684+0000 7fdb47fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:39.687 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.684+0000 7fdb4cd85700 10 monclient(hunting): get_auth_request con 0x7fdb480a5730 auth_method 0 2023-09-30T02:11:39.687 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.684+0000 7fdb4cd85700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:39.687 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.684+0000 7fdb4cd85700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:39.687 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.684+0000 7fdb4cd85700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:39.687 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.684+0000 7fdb47fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:39.687 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.684+0000 7fdb47fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:39.688 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.684+0000 7fdb47fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:39.688 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.684+0000 7fdb4d586700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:39.688 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.684+0000 7fdb4d586700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:39.688 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.684+0000 7fdb4d586700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:39.688 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.685+0000 7fdb4cd85700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:39.688 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.685+0000 7fdb4cd85700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:39.688 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.685+0000 7fdb4cd85700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:39.689 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.686+0000 7fdb47fff700 10 monclient(hunting): handle_auth_done global_id 4927 payload 274 2023-09-30T02:11:39.689 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.686+0000 7fdb47fff700 1 -- 172.21.15.59:0/2352539725 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdb4813b360 msgr2=0x7fdb4813b7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:11:39.689 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.686+0000 7fdb47fff700 1 --2- 172.21.15.59:0/2352539725 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdb4813b360 0x7fdb4813b7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:39.689 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.686+0000 7fdb47fff700 1 -- 172.21.15.59:0/2352539725 >> v2:172.21.15.59:3300/0 conn(0x7fdb480a5730 msgr2=0x7fdb480a7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:11:39.689 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.686+0000 7fdb47fff700 1 --2- 172.21.15.59:0/2352539725 >> v2:172.21.15.59:3300/0 conn(0x7fdb480a5730 0x7fdb480a7fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:39.689 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.686+0000 7fdb47fff700 10 monclient: _finish_hunting 0 2023-09-30T02:11:39.689 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.686+0000 7fdb47fff700 1 monclient: found mon.noname-c 2023-09-30T02:11:39.689 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.686+0000 7fdb47fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:11:39.690 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.686+0000 7fdb47fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3300/0 2023-09-30T02:11:39.690 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.686+0000 7fdb47fff700 1 -- 172.21.15.59:0/2352539725 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb480a5b00 con 0x7fdb481418e0 2023-09-30T02:11:39.690 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.688+0000 7fdb47fff700 1 --2- 172.21.15.59:0/2352539725 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdb481418e0 0x7fdb4813ae20 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fdb480a69c0 tx=0x7fdb340091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8385b41aefd1f19c server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:39.692 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.692+0000 7fdb477fe700 1 -- 172.21.15.59:0/2352539725 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb34011630 con 0x7fdb481418e0 2023-09-30T02:11:39.692 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.692+0000 7fdb477fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:39.692 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.692+0000 7fdb477fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:11:39.692 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.692+0000 7fdb477fe700 10 monclient: dump: 2023-09-30T02:11:39.692 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:39.692 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:39.693 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:39.693 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:39.693 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:39.693 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:39.693 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:39.693 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:39.693 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:39.693 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:39.693 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.692+0000 7fdb477fe700 10 monclient: _finish_auth 0 2023-09-30T02:11:39.693 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.692+0000 7fdb477fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:11:39.693 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.692+0000 7fdb477fe700 1 -- 172.21.15.59:0/2352539725 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb340117b0 con 0x7fdb481418e0 2023-09-30T02:11:39.694 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.692+0000 7fdb4efe9700 5 monclient: authenticate success, global_id 4927 2023-09-30T02:11:39.694 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.693+0000 7fdb4efe9700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:11:39.694 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.693+0000 7fdb477fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:11:39.694 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.693+0000 7fdb4efe9700 10 monclient: get_monmap_and_config success 2023-09-30T02:11:39.694 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.693+0000 7fdb477fe700 1 -- 172.21.15.59:0/2352539725 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb34011b70 con 0x7fdb481418e0 2023-09-30T02:11:39.694 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.693+0000 7fdb477fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:39.694 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.693+0000 7fdb4efe9700 4 set_mon_vals no callback set 2023-09-30T02:11:39.694 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.693+0000 7fdb477fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:11:39.694 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.693+0000 7fdb477fe700 10 monclient: dump: 2023-09-30T02:11:39.694 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:39.695 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:39.695 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:39.695 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:39.695 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:39.695 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:39.695 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:39.695 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:39.695 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:39.695 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:39.698 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.693+0000 7fdb4efe9700 10 monclient: shutdown 2023-09-30T02:11:39.698 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.693+0000 7fdb4efe9700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:11:39.698 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.693+0000 7fdb4efe9700 1 -- 172.21.15.59:0/2352539725 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdb481418e0 msgr2=0x7fdb4813ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:39.698 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.693+0000 7fdb4efe9700 1 --2- 172.21.15.59:0/2352539725 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdb481418e0 0x7fdb4813ae20 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fdb480a69c0 tx=0x7fdb340091a0 comp rx=0 tx=0).stop 2023-09-30T02:11:39.699 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.693+0000 7fdb4efe9700 1 -- 172.21.15.59:0/2352539725 shutdown_connections 2023-09-30T02:11:39.699 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.693+0000 7fdb4efe9700 1 --2- 172.21.15.59:0/2352539725 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdb4813b360 0x7fdb4813b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:39.699 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.693+0000 7fdb4efe9700 1 --2- 172.21.15.59:0/2352539725 >> v2:172.21.15.59:3300/0 conn(0x7fdb480a5730 0x7fdb480a7fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:39.699 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.693+0000 7fdb4efe9700 1 --2- 172.21.15.59:0/2352539725 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdb481418e0 0x7fdb4813ae20 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:39.699 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.693+0000 7fdb4efe9700 1 -- 172.21.15.59:0/2352539725 >> 172.21.15.59:0/2352539725 conn(0x7fdb480631f0 msgr2=0x7fdb480635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:11:39.699 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.693+0000 7fdb4efe9700 1 -- 172.21.15.59:0/2352539725 shutdown_connections 2023-09-30T02:11:39.700 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.693+0000 7fdb4efe9700 1 -- 172.21.15.59:0/2352539725 wait complete. 2023-09-30T02:11:39.700 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.694+0000 7fdb4efe9700 10 monclient: build_initial_monmap 2023-09-30T02:11:39.700 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.694+0000 7fdb4efe9700 10 monclient: monmap: 2023-09-30T02:11:39.700 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:11:39.700 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:11:39.700 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:11:39.700 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:11:39.700 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:11:39.700 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:11:39.700 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:11:39.700 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:11:39.700 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:11:39.701 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:39.701 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.694+0000 7fdb4efe9700 1 librados: starting msgr at 2023-09-30T02:11:39.701 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.694+0000 7fdb4efe9700 1 librados: starting objecter 2023-09-30T02:11:39.701 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.694+0000 7fdb4efe9700 1 Processor -- start 2023-09-30T02:11:39.701 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.695+0000 7fdb4efe9700 1 -- start start 2023-09-30T02:11:39.701 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.695+0000 7fdb4efe9700 1 librados: setting wanted keys 2023-09-30T02:11:39.701 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.695+0000 7fdb4efe9700 1 librados: calling monclient init 2023-09-30T02:11:39.701 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.695+0000 7fdb4efe9700 10 monclient: init 2023-09-30T02:11:39.701 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.695+0000 7fdb4efe9700 10 monclient: _reopen_session rank -1 2023-09-30T02:11:39.701 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.695+0000 7fdb4efe9700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T02:11:39.701 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.695+0000 7fdb4efe9700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdb480a5730 0x7fdb4813d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:39.702 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.695+0000 7fdb4efe9700 10 monclient(hunting): picked mon.noname-c con 0x7fdb480a5730 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:11:39.702 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.695+0000 7fdb4efe9700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb4813b360 0x7fdb48151a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:39.702 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.695+0000 7fdb4efe9700 10 monclient(hunting): picked mon.noname-a con 0x7fdb4813b360 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:11:39.702 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.695+0000 7fdb4efe9700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdb481418e0 0x7fdb48151fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:39.702 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.695+0000 7fdb4efe9700 10 monclient(hunting): picked mon.noname-b con 0x7fdb481418e0 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:11:39.702 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.695+0000 7fdb4efe9700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:39.702 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.695+0000 7fdb4efe9700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb480ac390 con 0x7fdb4813b360 2023-09-30T02:11:39.702 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.695+0000 7fdb4efe9700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:39.702 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.695+0000 7fdb4efe9700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb4813d740 con 0x7fdb481418e0 2023-09-30T02:11:39.702 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.695+0000 7fdb4efe9700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:39.703 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.695+0000 7fdb4efe9700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb48152510 con 0x7fdb480a5730 2023-09-30T02:11:39.703 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.695+0000 7fdb4efe9700 10 monclient(hunting): _renew_subs 2023-09-30T02:11:39.703 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.695+0000 7fdb4efe9700 10 monclient(hunting): authenticate will time out at 2144.997865s 2023-09-30T02:11:39.703 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.696+0000 7fdb4d586700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdb481418e0 0x7fdb48151fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:39.703 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.696+0000 7fdb47fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb4813b360 0x7fdb48151a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:39.703 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.696+0000 7fdb4cd85700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdb480a5730 0x7fdb4813d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:39.703 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.696+0000 7fdb4d586700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdb481418e0 0x7fdb48151fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:41446/0 (socket says 172.21.15.59:41446) 2023-09-30T02:11:39.703 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.696+0000 7fdb4d586700 1 -- 172.21.15.59:0/3475538861 learned_addr learned my addr 172.21.15.59:0/3475538861 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:11:39.703 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.696+0000 7fdb4d586700 10 monclient(hunting): get_auth_request con 0x7fdb481418e0 auth_method 0 2023-09-30T02:11:39.703 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.696+0000 7fdb4d586700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:39.703 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.696+0000 7fdb4d586700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:39.704 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.696+0000 7fdb4d586700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:39.704 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.696+0000 7fdb4cd85700 10 monclient(hunting): get_auth_request con 0x7fdb480a5730 auth_method 0 2023-09-30T02:11:39.704 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.696+0000 7fdb4cd85700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:39.704 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.696+0000 7fdb4cd85700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:39.704 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.696+0000 7fdb4cd85700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:39.704 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.696+0000 7fdb47fff700 10 monclient(hunting): get_auth_request con 0x7fdb4813b360 auth_method 0 2023-09-30T02:11:39.704 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.696+0000 7fdb47fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:39.704 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.696+0000 7fdb47fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:39.704 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.696+0000 7fdb47fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:39.704 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.697+0000 7fdb4d586700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:39.704 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.697+0000 7fdb4d586700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:39.705 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.697+0000 7fdb4d586700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:39.705 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.697+0000 7fdb4cd85700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:39.705 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.697+0000 7fdb4cd85700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:39.705 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.697+0000 7fdb4cd85700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:39.705 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.697+0000 7fdb47fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:39.705 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.697+0000 7fdb47fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:39.705 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.697+0000 7fdb47fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:39.707 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.700+0000 7fdb4d586700 10 monclient(hunting): handle_auth_done global_id 4930 payload 824 2023-09-30T02:11:39.707 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.700+0000 7fdb4d586700 1 -- 172.21.15.59:0/3475538861 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdb480a5730 msgr2=0x7fdb4813d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:39.707 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.700+0000 7fdb4d586700 1 --2- 172.21.15.59:0/3475538861 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fdb480a5730 0x7fdb4813d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:39.707 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.700+0000 7fdb4d586700 1 -- 172.21.15.59:0/3475538861 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb4813b360 msgr2=0x7fdb48151a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:39.707 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.700+0000 7fdb4d586700 1 --2- 172.21.15.59:0/3475538861 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdb4813b360 0x7fdb48151a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:39.708 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.700+0000 7fdb4d586700 10 monclient: _finish_hunting 0 2023-09-30T02:11:39.708 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.700+0000 7fdb4d586700 1 monclient: found mon.noname-b 2023-09-30T02:11:39.708 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.700+0000 7fdb4d586700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:11:39.708 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.700+0000 7fdb4d586700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:11:39.708 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.700+0000 7fdb4d586700 1 -- 172.21.15.59:0/3475538861 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb34009420 con 0x7fdb481418e0 2023-09-30T02:11:39.708 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.702+0000 7fdb4d586700 1 --2- 172.21.15.59:0/3475538861 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdb481418e0 0x7fdb48151fd0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fdb40009e80 tx=0x7fdb4000b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:39.708 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.702+0000 7fdb44ff9700 10 client.?.objecter ms_handle_connect 0x7fdb481418e0 2023-09-30T02:11:39.708 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.702+0000 7fdb44ff9700 10 client.?.objecter resend_mon_ops 2023-09-30T02:11:39.708 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.706+0000 7fdb44ff9700 1 -- 172.21.15.59:0/3475538861 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb4000c890 con 0x7fdb481418e0 2023-09-30T02:11:39.708 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.706+0000 7fdb44ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:39.708 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.706+0000 7fdb44ff9700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:11:39.708 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.706+0000 7fdb44ff9700 10 monclient: dump: 2023-09-30T02:11:39.709 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:39.709 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:39.709 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:39.709 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:39.709 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:39.709 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:39.709 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:39.709 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:39.709 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:39.709 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:39.709 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.706+0000 7fdb44ff9700 10 monclient: _finish_auth 0 2023-09-30T02:11:39.710 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.706+0000 7fdb44ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:11:39.710 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.706+0000 7fdb44ff9700 1 -- 172.21.15.59:0/3475538861 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb4000f040 con 0x7fdb481418e0 2023-09-30T02:11:39.710 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.706+0000 7fdb44ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:11:39.710 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.706+0000 7fdb44ff9700 1 -- 172.21.15.59:0/3475538861 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb4000cc40 con 0x7fdb481418e0 2023-09-30T02:11:39.710 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.706+0000 7fdb4efe9700 5 monclient: authenticate success, global_id 4930 2023-09-30T02:11:39.710 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.706+0000 7fdb4efe9700 10 monclient: _renew_subs 2023-09-30T02:11:39.710 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.706+0000 7fdb4efe9700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:11:39.710 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.706+0000 7fdb4efe9700 1 -- 172.21.15.59:0/3475538861 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb480ae270 con 0x7fdb481418e0 2023-09-30T02:11:39.710 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.706+0000 7fdb46ffd700 4 set_mon_vals no callback set 2023-09-30T02:11:39.710 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.706+0000 7fdb44ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:39.710 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.706+0000 7fdb44ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:11:39.710 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.706+0000 7fdb44ff9700 10 monclient: dump: 2023-09-30T02:11:39.711 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:39.711 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:39.711 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:39.711 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:39.711 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:39.711 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:39.711 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:39.711 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:39.711 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:39.711 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:39.711 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.706+0000 7fdb4efe9700 10 client.4930.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:11:39.711 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.706+0000 7fdb4efe9700 10 monclient: _renew_subs 2023-09-30T02:11:39.712 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.706+0000 7fdb4efe9700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:11:39.712 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.706+0000 7fdb4efe9700 1 -- 172.21.15.59:0/3475538861 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb4813cd20 con 0x7fdb481418e0 2023-09-30T02:11:39.712 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.706+0000 7fdb4efe9700 1 librados: init done 2023-09-30T02:11:39.712 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.707+0000 7fdb4efe9700 10 client.4930.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:11:39.712 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.707+0000 7fdb4efe9700 15 client.4930.objecter _session_command_op_assign -1 1 2023-09-30T02:11:39.712 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.707+0000 7fdb4efe9700 20 client.4930.objecter _get_session osd=-1 returning homeless 2023-09-30T02:11:39.712 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.707+0000 7fdb4efe9700 10 client.4930.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:11:39.712 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.707+0000 7fdb4efe9700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:11:39.712 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.707+0000 7fdb4efe9700 1 -- 172.21.15.59:0/3475538861 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fdb480a8210 con 0x7fdb481418e0 2023-09-30T02:11:39.724 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.725+0000 7fdb44ff9700 1 -- 172.21.15.59:0/3475538861 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fdb40014650 con 0x7fdb481418e0 2023-09-30T02:11:39.724 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.725+0000 7fdb44ff9700 1 --2- 172.21.15.59:0/3475538861 >> v2:172.21.15.110:6800/115870 conn(0x7fdb2c068890 0x7fdb2c06ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:39.725 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.725+0000 7fdb4cd85700 1 --2- 172.21.15.59:0/3475538861 >> v2:172.21.15.110:6800/115870 conn(0x7fdb2c068890 0x7fdb2c06ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:39.725 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.725+0000 7fdb4cd85700 10 monclient: get_auth_request con 0x7fdb2c068890 auth_method 0 2023-09-30T02:11:39.725 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.726+0000 7fdb44ff9700 10 client.4930.objecter ms_handle_connect 0x7fdb2c068890 2023-09-30T02:11:39.725 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.726+0000 7fdb4cd85700 1 --2- 172.21.15.59:0/3475538861 >> v2:172.21.15.110:6800/115870 conn(0x7fdb2c068890 0x7fdb2c06ad40 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fdb3c009d20 tx=0x7fdb3c0091a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:39.730 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.730+0000 7fdb44ff9700 1 -- 172.21.15.59:0/3475538861 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fdb40009320 con 0x7fdb481418e0 2023-09-30T02:11:39.730 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.730+0000 7fdb44ff9700 10 client.4930.objecter ms_dispatch 0x7fdb48001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:11:39.730 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.730+0000 7fdb44ff9700 3 client.4930.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:11:39.730 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.730+0000 7fdb44ff9700 3 client.4930.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:11:39.730 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.730+0000 7fdb44ff9700 10 client.4930.objecter checking command 1 2023-09-30T02:11:39.730 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.730+0000 7fdb44ff9700 1 --2- 172.21.15.59:0/3475538861 >> v2:172.21.15.110:6809/116562 conn(0x7fdb2c06e800 0x7fdb2c070c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:39.731 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.730+0000 7fdb44ff9700 20 client.4930.objecter _get_session s=0x7fdb2c06e150 osd=5 3 2023-09-30T02:11:39.731 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.730+0000 7fdb44ff9700 20 client.4930.objecter put_session s=0x7fdb2c06e150 osd=5 3 2023-09-30T02:11:39.731 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.730+0000 7fdb44ff9700 15 client.4930.objecter _session_command_op_remove -1 1 2023-09-30T02:11:39.731 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.730+0000 7fdb44ff9700 20 client.4930.objecter _get_session s=0x7fdb2c06e150 osd=5 3 2023-09-30T02:11:39.731 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.730+0000 7fdb44ff9700 20 client.4930.objecter get_session s=0x7fdb2c06e150 osd=5 3 2023-09-30T02:11:39.731 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.730+0000 7fdb44ff9700 15 client.4930.objecter _session_command_op_assign 5 1 2023-09-30T02:11:39.731 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.730+0000 7fdb44ff9700 20 client.4930.objecter put_session s=0x7fdb2c06e150 osd=5 4 2023-09-30T02:11:39.731 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.730+0000 7fdb44ff9700 10 client.4930.objecter _send_command 1 2023-09-30T02:11:39.731 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.730+0000 7fdb44ff9700 1 -- 172.21.15.59:0/3475538861 --> v2:172.21.15.110:6809/116562 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdb2c071280 con 0x7fdb2c06e800 2023-09-30T02:11:39.731 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.730+0000 7fdb44ff9700 20 client.4930.objecter dump_active .. 0 homeless 2023-09-30T02:11:39.731 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.730+0000 7fdb44ff9700 1 -- 172.21.15.59:0/3475538861 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdb40052d10 con 0x7fdb481418e0 2023-09-30T02:11:39.731 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.730+0000 7fdb44ff9700 10 monclient: handle_get_version_reply finishing 1 version 25 2023-09-30T02:11:39.732 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.731+0000 7fdb47fff700 1 --2- 172.21.15.59:0/3475538861 >> v2:172.21.15.110:6809/116562 conn(0x7fdb2c06e800 0x7fdb2c070c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:39.732 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.731+0000 7fdb47fff700 10 monclient: get_auth_request con 0x7fdb2c06e800 auth_method 0 2023-09-30T02:11:39.734 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.735+0000 7fdb47fff700 1 --2- 172.21.15.59:0/3475538861 >> v2:172.21.15.110:6809/116562 conn(0x7fdb2c06e800 0x7fdb2c070c10 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:39.734 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.735+0000 7fdb44ff9700 10 client.4930.objecter ms_handle_connect 0x7fdb2c06e800 2023-09-30T02:11:39.756 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.757+0000 7fdb44ff9700 1 -- 172.21.15.59:0/3475538861 <== osd.5 v2:172.21.15.110:6809/116562 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7fdb2c071280 con 0x7fdb2c06e800 2023-09-30T02:11:39.757 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.757+0000 7fdb44ff9700 10 client.4930.objecter ms_dispatch 0x7fdb48001550 command_reply(tid 1: 0 ) v1 2023-09-30T02:11:39.757 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.757+0000 7fdb44ff9700 10 client.4930.objecter _finish_command 1 = system:0 2023-09-30T02:11:39.757 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.757+0000 7fdb44ff9700 20 client.4930.objecter put_session s=0x7fdb2c06e150 osd=5 4 2023-09-30T02:11:39.757 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.757+0000 7fdb44ff9700 15 client.4930.objecter _session_command_op_remove 5 1 2023-09-30T02:11:39.771 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.772+0000 7fdb4efe9700 10 client.4930.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-09-30T02:11:39.772 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.772+0000 7fdb4efe9700 15 client.4930.objecter _session_command_op_assign -1 2 2023-09-30T02:11:39.772 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.772+0000 7fdb4efe9700 20 client.4930.objecter _get_session s=0x7fdb2c06e150 osd=5 3 2023-09-30T02:11:39.772 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.772+0000 7fdb4efe9700 20 client.4930.objecter put_session s=0x7fdb2c06e150 osd=5 3 2023-09-30T02:11:39.772 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.772+0000 7fdb4efe9700 20 client.4930.objecter _get_session s=0x7fdb2c06e150 osd=5 3 2023-09-30T02:11:39.772 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.772+0000 7fdb4efe9700 15 client.4930.objecter _session_command_op_remove -1 2 2023-09-30T02:11:39.772 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.772+0000 7fdb4efe9700 20 client.4930.objecter get_session s=0x7fdb2c06e150 osd=5 3 2023-09-30T02:11:39.772 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.772+0000 7fdb4efe9700 15 client.4930.objecter _session_command_op_assign 5 2 2023-09-30T02:11:39.772 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.772+0000 7fdb4efe9700 20 client.4930.objecter put_session s=0x7fdb2c06e150 osd=5 4 2023-09-30T02:11:39.772 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.772+0000 7fdb4efe9700 10 client.4930.objecter _send_command 2 2023-09-30T02:11:39.772 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.772+0000 7fdb4efe9700 1 -- 172.21.15.59:0/3475538861 --> v2:172.21.15.110:6809/116562 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fdb480a8010 con 0x7fdb2c06e800 2023-09-30T02:11:39.793 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.793+0000 7fdb44ff9700 1 -- 172.21.15.59:0/3475538861 <== osd.5 v2:172.21.15.110:6809/116562 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fdb480a8010 con 0x7fdb2c06e800 2023-09-30T02:11:39.793 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.793+0000 7fdb44ff9700 10 client.4930.objecter ms_dispatch 0x7fdb48001550 command_reply(tid 2: 0 ) v1 2023-09-30T02:11:39.793 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.793+0000 7fdb44ff9700 10 client.4930.objecter _finish_command 2 = system:0 2023-09-30T02:11:39.793 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.793+0000 7fdb44ff9700 20 client.4930.objecter put_session s=0x7fdb2c06e150 osd=5 4 2023-09-30T02:11:39.793 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.793+0000 7fdb44ff9700 15 client.4930.objecter _session_command_op_remove 5 2 2023-09-30T02:11:39.793 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2023-09-30T02:11:39.794 INFO:teuthology.orchestra.run.smithi059.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T02:11:39.794 INFO:teuthology.orchestra.run.smithi059.stdout:} 2023-09-30T02:11:39.795 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.794+0000 7fdb4efe9700 10 librados: watch_flush enter 2023-09-30T02:11:39.795 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.794+0000 7fdb4efe9700 10 librados: watch_flush exit 2023-09-30T02:11:39.795 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.794+0000 7fdb4efe9700 10 client.4930.objecter close_session for osd.5 2023-09-30T02:11:39.795 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.794+0000 7fdb4efe9700 1 -- 172.21.15.59:0/3475538861 >> v2:172.21.15.110:6809/116562 conn(0x7fdb2c06e800 msgr2=0x7fdb2c070c10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:39.795 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.794+0000 7fdb4efe9700 1 --2- 172.21.15.59:0/3475538861 >> v2:172.21.15.110:6809/116562 conn(0x7fdb2c06e800 0x7fdb2c070c10 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:39.795 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.794+0000 7fdb4efe9700 20 client.4930.objecter put_session s=0x7fdb2c06e150 osd=5 1 2023-09-30T02:11:39.795 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.794+0000 7fdb4efe9700 20 client.4930.objecter shutdown clearing up homeless session... 2023-09-30T02:11:39.796 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.794+0000 7fdb4efe9700 10 client.4930.objecter successfully canceled tick 2023-09-30T02:11:39.796 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.794+0000 7fdb4efe9700 1 -- 172.21.15.59:0/3475538861 >> v2:172.21.15.110:6800/115870 conn(0x7fdb2c068890 msgr2=0x7fdb2c06ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:39.796 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.794+0000 7fdb4efe9700 1 --2- 172.21.15.59:0/3475538861 >> v2:172.21.15.110:6800/115870 conn(0x7fdb2c068890 0x7fdb2c06ad40 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fdb3c009d20 tx=0x7fdb3c0091a0 comp rx=0 tx=0).stop 2023-09-30T02:11:39.796 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.794+0000 7fdb4efe9700 10 monclient: shutdown 2023-09-30T02:11:39.796 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.794+0000 7fdb4efe9700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:11:39.796 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.794+0000 7fdb4efe9700 1 -- 172.21.15.59:0/3475538861 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdb481418e0 msgr2=0x7fdb48151fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:39.796 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.794+0000 7fdb4efe9700 1 --2- 172.21.15.59:0/3475538861 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fdb481418e0 0x7fdb48151fd0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fdb40009e80 tx=0x7fdb4000b040 comp rx=0 tx=0).stop 2023-09-30T02:11:39.796 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.794+0000 7fdb4d586700 1 -- 172.21.15.59:0/3475538861 reap_dead start 2023-09-30T02:11:39.796 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.794+0000 7fdb4efe9700 1 -- 172.21.15.59:0/3475538861 shutdown_connections 2023-09-30T02:11:39.796 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.794+0000 7fdb4efe9700 1 -- 172.21.15.59:0/3475538861 >> 172.21.15.59:0/3475538861 conn(0x7fdb480631f0 msgr2=0x7fdb480b5360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:11:39.796 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.794+0000 7fdb4efe9700 1 -- 172.21.15.59:0/3475538861 shutdown_connections 2023-09-30T02:11:39.797 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.794+0000 7fdb4efe9700 1 -- 172.21.15.59:0/3475538861 wait complete. 2023-09-30T02:11:39.797 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.794+0000 7fdb4efe9700 1 librados: shutdown 2023-09-30T02:11:39.807 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2023-09-30T02:11:39.947 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.947+0000 7f362c949700 10 monclient: get_monmap_and_config 2023-09-30T02:11:39.947 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.947+0000 7f362c949700 10 monclient: build_initial_monmap 2023-09-30T02:11:39.947 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.947+0000 7f362c949700 10 monclient: monmap: 2023-09-30T02:11:39.947 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:11:39.947 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:39.947 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:11:39.948219+0000 2023-09-30T02:11:39.947 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:11:39.948219+0000 2023-09-30T02:11:39.947 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:11:39.948 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:39.948 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:11:39.948 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:11:39.948 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:11:39.948 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:11:39.948 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:11:39.948 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:39.949 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.948+0000 7f362c949700 1 Processor -- start 2023-09-30T02:11:39.950 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.948+0000 7f362c949700 1 -- start start 2023-09-30T02:11:39.950 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.948+0000 7f362c949700 10 monclient: init 2023-09-30T02:11:39.950 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.949+0000 7f362c949700 10 monclient: _reopen_session rank -1 2023-09-30T02:11:39.950 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.949+0000 7f362c949700 10 monclient: _add_conns ranks=[0,1,4,3,2] 2023-09-30T02:11:39.950 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.949+0000 7f362c949700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f36280a5b70 addr v1:172.21.15.59:6789/0 2023-09-30T02:11:39.950 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.949+0000 7f362c949700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3628142290 addr v1:172.21.15.110:6789/0 2023-09-30T02:11:39.950 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.949+0000 7f362c949700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f36280b1500 0x7f36280a5f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:39.950 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.949+0000 7f362c949700 10 monclient(hunting): picked mon.noname-c con 0x7f36280b1500 addr v2:172.21.15.110:3300/0 2023-09-30T02:11:39.950 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.949+0000 7f362c949700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36280b1920 con 0x7f36280a5b70 2023-09-30T02:11:39.950 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.949+0000 7f362c949700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36280a75a0 con 0x7f3628142290 2023-09-30T02:11:39.950 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.949+0000 7f362c949700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:39.951 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.950+0000 7f362c949700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36280af3f0 con 0x7f36280b1500 2023-09-30T02:11:39.951 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.950+0000 7f362c949700 10 monclient(hunting): _renew_subs 2023-09-30T02:11:39.951 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.950+0000 7f362c949700 10 monclient(hunting): authenticate will time out at 2145.252020s 2023-09-30T02:11:39.995 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.995+0000 7f3626d9d700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f36280b1500 0x7f36280a5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:39.995 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.995+0000 7f3625d9b700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f3628142290 0x7f362813af30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.59:50278/0 (socket says 172.21.15.59:50278) 2023-09-30T02:11:39.995 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.995+0000 7f3625d9b700 1 -- 172.21.15.59:0/447558550 learned_addr learned my addr 172.21.15.59:0/447558550 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:11:39.995 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.995+0000 7f3626d9d700 10 monclient(hunting): get_auth_request con 0x7f36280b1500 auth_method 0 2023-09-30T02:11:39.995 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.995+0000 7f3626d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:39.995 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.995+0000 7f3626d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:39.995 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.995+0000 7f3626d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:39.996 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.996+0000 7f3626d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:39.996 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.996+0000 7f3626d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:39.996 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.996+0000 7f3626d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:39.997 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.998+0000 7f3626d9d700 10 monclient(hunting): handle_auth_done global_id 4933 payload 274 2023-09-30T02:11:39.998 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.998+0000 7f3626d9d700 1 -- 172.21.15.59:0/447558550 >> v1:172.21.15.110:6789/0 conn(0x7f3628142290 legacy=0x7f362813af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:39.998 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.998+0000 7f3626d9d700 1 -- 172.21.15.59:0/447558550 >> v1:172.21.15.59:6789/0 conn(0x7f36280a5b70 legacy=0x7f3628141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:11:39.998 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.998+0000 7f3626d9d700 10 monclient: _finish_hunting 0 2023-09-30T02:11:39.998 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.998+0000 7f3626d9d700 1 monclient: found mon.noname-c 2023-09-30T02:11:39.998 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.998+0000 7f3626d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:11:39.998 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.998+0000 7f3626d9d700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3300/0 2023-09-30T02:11:39.998 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:39.998+0000 7f3626d9d700 1 -- 172.21.15.59:0/447558550 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f362813b640 con 0x7f36280b1500 2023-09-30T02:11:39.999 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.000+0000 7f3626d9d700 1 --2- 172.21.15.59:0/447558550 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f36280b1500 0x7f36280a5f40 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f3620009e60 tx=0x7f3620009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=240d54d13e421fb1 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:40.006 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.006+0000 7f362559a700 1 -- 172.21.15.59:0/447558550 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f362000e070 con 0x7f36280b1500 2023-09-30T02:11:40.007 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.006+0000 7f362559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:40.007 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.006+0000 7f362559a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:11:40.007 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.006+0000 7f362559a700 10 monclient: dump: 2023-09-30T02:11:40.007 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:40.007 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:40.007 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:40.008 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:40.008 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:40.009 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:40.009 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:40.009 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:40.009 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:40.009 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:40.009 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.006+0000 7f362559a700 10 monclient: _finish_auth 0 2023-09-30T02:11:40.009 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.006+0000 7f362559a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:11:40.010 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.006+0000 7f362c949700 5 monclient: authenticate success, global_id 4933 2023-09-30T02:11:40.010 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.006+0000 7f362c949700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:11:40.010 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.006+0000 7f362559a700 1 -- 172.21.15.59:0/447558550 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f362000bbf0 con 0x7f36280b1500 2023-09-30T02:11:40.010 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.006+0000 7f362559a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:11:40.010 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.006+0000 7f362c949700 10 monclient: get_monmap_and_config success 2023-09-30T02:11:40.010 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.006+0000 7f362559a700 1 -- 172.21.15.59:0/447558550 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3620012430 con 0x7f36280b1500 2023-09-30T02:11:40.010 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.006+0000 7f362559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:40.010 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.006+0000 7f362c949700 4 set_mon_vals no callback set 2023-09-30T02:11:40.011 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.006+0000 7f362559a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:11:40.011 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.006+0000 7f362559a700 10 monclient: dump: 2023-09-30T02:11:40.011 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:40.011 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:40.011 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:40.011 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:40.011 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:40.011 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:40.011 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:40.012 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:40.012 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:40.012 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:40.012 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.007+0000 7f362c949700 10 monclient: shutdown 2023-09-30T02:11:40.012 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.007+0000 7f362c949700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:11:40.012 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.007+0000 7f362c949700 1 -- 172.21.15.59:0/447558550 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f36280b1500 msgr2=0x7f36280a5f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:40.012 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.007+0000 7f362c949700 1 --2- 172.21.15.59:0/447558550 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f36280b1500 0x7f36280a5f40 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f3620009e60 tx=0x7f3620009780 comp rx=0 tx=0).stop 2023-09-30T02:11:40.012 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.007+0000 7f362c949700 1 -- 172.21.15.59:0/447558550 shutdown_connections 2023-09-30T02:11:40.012 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.007+0000 7f362c949700 1 --2- 172.21.15.59:0/447558550 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f36280b1500 0x7f36280a5f40 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:40.013 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.007+0000 7f362c949700 1 -- 172.21.15.59:0/447558550 >> 172.21.15.59:0/447558550 conn(0x7f36280631f0 msgr2=0x7f36280635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:11:40.013 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.007+0000 7f362c949700 1 -- 172.21.15.59:0/447558550 shutdown_connections 2023-09-30T02:11:40.013 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.007+0000 7f362c949700 1 -- 172.21.15.59:0/447558550 wait complete. 2023-09-30T02:11:40.018 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.008+0000 7f362c949700 10 monclient: build_initial_monmap 2023-09-30T02:11:40.018 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.008+0000 7f362c949700 10 monclient: monmap: 2023-09-30T02:11:40.018 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:11:40.018 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:11:40.018 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:11:40.018 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:11:40.018 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:11:40.018 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:11:40.019 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:11:40.019 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:11:40.019 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:11:40.019 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:40.019 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.008+0000 7f362c949700 1 librados: starting msgr at 2023-09-30T02:11:40.019 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.008+0000 7f362c949700 1 librados: starting objecter 2023-09-30T02:11:40.019 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.009+0000 7f362c949700 1 Processor -- start 2023-09-30T02:11:40.019 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.009+0000 7f362c949700 1 -- start start 2023-09-30T02:11:40.019 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.009+0000 7f362c949700 1 librados: setting wanted keys 2023-09-30T02:11:40.019 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.009+0000 7f362c949700 1 librados: calling monclient init 2023-09-30T02:11:40.019 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.009+0000 7f362c949700 10 monclient: init 2023-09-30T02:11:40.019 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.009+0000 7f362c949700 10 monclient: _reopen_session rank -1 2023-09-30T02:11:40.020 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.009+0000 7f362c949700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T02:11:40.020 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.009+0000 7f362c949700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36280a5b70 0x7f3628144f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:40.020 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.009+0000 7f362c949700 10 monclient(hunting): picked mon.noname-a con 0x7f36280a5b70 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:11:40.020 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.009+0000 7f362c949700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f36280b1500 0x7f362813e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:40.020 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.009+0000 7f362c949700 10 monclient(hunting): picked mon.noname-c con 0x7f36280b1500 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:11:40.020 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.009+0000 7f362c949700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3628142290 0x7f36280ae000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:40.020 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.009+0000 7f362c949700 10 monclient(hunting): picked mon.noname-b con 0x7f3628142290 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:11:40.020 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.009+0000 7f362c949700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:40.020 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.009+0000 7f362c949700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36280a8b00 con 0x7f36280a5b70 2023-09-30T02:11:40.020 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.009+0000 7f362c949700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:40.020 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.009+0000 7f362c949700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f362813d210 con 0x7f3628142290 2023-09-30T02:11:40.020 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.009+0000 7f362c949700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:40.021 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.009+0000 7f362c949700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36280a78a0 con 0x7f36280b1500 2023-09-30T02:11:40.021 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.009+0000 7f362c949700 10 monclient(hunting): _renew_subs 2023-09-30T02:11:40.021 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.009+0000 7f362c949700 10 monclient(hunting): authenticate will time out at 2145.311997s 2023-09-30T02:11:40.021 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.010+0000 7f3626d9d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3628142290 0x7f36280ae000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:40.021 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.011+0000 7f3626d9d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3628142290 0x7f36280ae000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:41466/0 (socket says 172.21.15.59:41466) 2023-09-30T02:11:40.021 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.011+0000 7f3626d9d700 1 -- 172.21.15.59:0/3427094597 learned_addr learned my addr 172.21.15.59:0/3427094597 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:11:40.021 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.011+0000 7f3626d9d700 10 monclient(hunting): get_auth_request con 0x7f3628142290 auth_method 0 2023-09-30T02:11:40.021 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.011+0000 7f3626d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:40.021 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.011+0000 7f3626d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:40.021 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.011+0000 7f3626d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:40.021 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.011+0000 7f3625d9b700 1 --2- 172.21.15.59:0/3427094597 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f36280b1500 0x7f362813e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:40.022 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.011+0000 7f3625d9b700 10 monclient(hunting): get_auth_request con 0x7f36280b1500 auth_method 0 2023-09-30T02:11:40.022 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.011+0000 7f3625d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:40.022 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.011+0000 7f3625d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:40.022 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.011+0000 7f3625d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:40.022 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.011+0000 7f3626d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:40.022 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.011+0000 7f3626d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:40.022 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.011+0000 7f3626d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:40.022 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.012+0000 7f3625d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:40.022 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.012+0000 7f3625d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:40.022 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.012+0000 7f3625d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:40.022 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.015+0000 7f362659c700 1 --2- 172.21.15.59:0/3427094597 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36280a5b70 0x7f3628144f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:40.022 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.016+0000 7f3626d9d700 10 monclient(hunting): handle_auth_done global_id 4936 payload 824 2023-09-30T02:11:40.023 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.016+0000 7f3626d9d700 1 -- 172.21.15.59:0/3427094597 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f36280b1500 msgr2=0x7f362813e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:40.023 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.016+0000 7f3626d9d700 1 --2- 172.21.15.59:0/3427094597 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f36280b1500 0x7f362813e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:40.023 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.016+0000 7f3626d9d700 1 -- 172.21.15.59:0/3427094597 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36280a5b70 msgr2=0x7f3628144f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:40.023 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.016+0000 7f3626d9d700 1 --2- 172.21.15.59:0/3427094597 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36280a5b70 0x7f3628144f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:40.023 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.016+0000 7f3626d9d700 10 monclient: _finish_hunting 0 2023-09-30T02:11:40.023 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.016+0000 7f3626d9d700 1 monclient: found mon.noname-b 2023-09-30T02:11:40.023 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.016+0000 7f3626d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:11:40.023 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.016+0000 7f3626d9d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:11:40.023 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.016+0000 7f3626d9d700 1 -- 172.21.15.59:0/3427094597 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3620009a20 con 0x7f3628142290 2023-09-30T02:11:40.025 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.021+0000 7f3626d9d700 1 --2- 172.21.15.59:0/3427094597 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3628142290 0x7f36280ae000 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f36280a6510 tx=0x7f3620018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:40.025 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.021+0000 7f36167fc700 10 client.?.objecter ms_handle_connect 0x7f3628142290 2023-09-30T02:11:40.025 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.021+0000 7f36167fc700 10 client.?.objecter resend_mon_ops 2023-09-30T02:11:40.025 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.023+0000 7f36167fc700 1 -- 172.21.15.59:0/3427094597 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3620012e70 con 0x7f3628142290 2023-09-30T02:11:40.025 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.023+0000 7f36167fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:40.026 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.023+0000 7f36167fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:11:40.026 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.023+0000 7f36167fc700 10 monclient: dump: 2023-09-30T02:11:40.026 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:40.027 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:40.027 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:40.027 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:40.027 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:40.027 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:40.027 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:40.028 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:40.028 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:40.028 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:40.028 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.023+0000 7f36167fc700 10 monclient: _finish_auth 0 2023-09-30T02:11:40.028 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.023+0000 7f36167fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:11:40.028 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.023+0000 7f362c949700 5 monclient: authenticate success, global_id 4936 2023-09-30T02:11:40.029 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.023+0000 7f362c949700 10 monclient: _renew_subs 2023-09-30T02:11:40.029 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.023+0000 7f362c949700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:11:40.029 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.023+0000 7f362c949700 1 -- 172.21.15.59:0/3427094597 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36280a7a60 con 0x7f3628142290 2023-09-30T02:11:40.029 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.023+0000 7f362c949700 10 client.4936.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:11:40.029 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.023+0000 7f362c949700 10 monclient: _renew_subs 2023-09-30T02:11:40.030 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.023+0000 7f362c949700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:11:40.030 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.023+0000 7f362c949700 1 -- 172.21.15.59:0/3427094597 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f362813e070 con 0x7f3628142290 2023-09-30T02:11:40.030 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.023+0000 7f362c949700 1 librados: init done 2023-09-30T02:11:40.030 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.024+0000 7f362c949700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:11:40.030 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.024+0000 7f362c949700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:11:40.030 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.024+0000 7f362c949700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:11:40.031 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.024+0000 7f362c949700 1 -- 172.21.15.59:0/3427094597 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3628064a00 con 0x7f3628142290 2023-09-30T02:11:40.031 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.025+0000 7f36167fc700 1 -- 172.21.15.59:0/3427094597 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f362001e850 con 0x7f3628142290 2023-09-30T02:11:40.031 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.025+0000 7f36167fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:11:40.031 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.025+0000 7f36167fc700 1 -- 172.21.15.59:0/3427094597 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f362001e9c0 con 0x7f3628142290 2023-09-30T02:11:40.031 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.025+0000 7f36167fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:40.031 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.025+0000 7f36167fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:11:40.032 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.025+0000 7f3624d99700 4 set_mon_vals no callback set 2023-09-30T02:11:40.032 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.025+0000 7f36167fc700 10 monclient: dump: 2023-09-30T02:11:40.032 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:40.032 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:40.032 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:40.032 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:40.033 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:40.033 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:40.033 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:40.033 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:40.033 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:40.033 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:40.274 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.274+0000 7f36167fc700 1 -- 172.21.15.59:0/3427094597 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f3620024070 con 0x7f3628142290 2023-09-30T02:11:40.274 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.275+0000 7f36167fc700 1 --2- 172.21.15.59:0/3427094597 >> v2:172.21.15.110:6800/115870 conn(0x7f3608068890 0x7f360806ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:40.274 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.275+0000 7f362659c700 1 --2- 172.21.15.59:0/3427094597 >> v2:172.21.15.110:6800/115870 conn(0x7f3608068890 0x7f360806ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:40.275 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.275+0000 7f362659c700 10 monclient: get_auth_request con 0x7f3608068890 auth_method 0 2023-09-30T02:11:40.275 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.275+0000 7f36167fc700 10 client.4936.objecter ms_handle_connect 0x7f3608068890 2023-09-30T02:11:40.275 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.275+0000 7f362659c700 1 --2- 172.21.15.59:0/3427094597 >> v2:172.21.15.110:6800/115870 conn(0x7f3608068890 0x7f360806ad40 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f361800a8c0 tx=0x7f3618008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:40.278 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.278+0000 7f36167fc700 1 -- 172.21.15.59:0/3427094597 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f3620058590 con 0x7f3628142290 2023-09-30T02:11:40.278 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.278+0000 7f36167fc700 10 client.4936.objecter ms_dispatch 0x7f3628001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:11:40.278 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.278+0000 7f36167fc700 3 client.4936.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:11:40.278 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.278+0000 7f36167fc700 3 client.4936.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:11:40.278 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.278+0000 7f36167fc700 20 client.4936.objecter dump_active .. 0 homeless 2023-09-30T02:11:40.288 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.288+0000 7f36167fc700 1 -- 172.21.15.59:0/3427094597 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f36200541d0 con 0x7f3628142290 2023-09-30T02:11:40.288 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.288+0000 7f36167fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:11:40.288 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.288+0000 7f36167fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:11:40.500 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.500+0000 7f362c949700 1 -- 172.21.15.59:0/3427094597 --> v2:172.21.15.110:6800/115870 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7f36280b1920 con 0x7f3608068890 2023-09-30T02:11:40.500 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.501+0000 7f36167fc700 1 -- 172.21.15.59:0/3427094597 <== mgr.4108 v2:172.21.15.110:6800/115870 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f36280b1920 con 0x7f3608068890 2023-09-30T02:11:40.504 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.504+0000 7f362c949700 10 librados: watch_flush enter 2023-09-30T02:11:40.505 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.504+0000 7f362c949700 10 librados: watch_flush exit 2023-09-30T02:11:40.505 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.504+0000 7f362c949700 20 client.4936.objecter shutdown clearing up homeless session... 2023-09-30T02:11:40.505 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.504+0000 7f362c949700 10 client.4936.objecter successfully canceled tick 2023-09-30T02:11:40.505 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.504+0000 7f362c949700 1 -- 172.21.15.59:0/3427094597 >> v2:172.21.15.110:6800/115870 conn(0x7f3608068890 msgr2=0x7f360806ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:40.505 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.504+0000 7f362c949700 1 --2- 172.21.15.59:0/3427094597 >> v2:172.21.15.110:6800/115870 conn(0x7f3608068890 0x7f360806ad40 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f361800a8c0 tx=0x7f3618008040 comp rx=0 tx=0).stop 2023-09-30T02:11:40.505 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.504+0000 7f362c949700 10 monclient: shutdown 2023-09-30T02:11:40.505 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.504+0000 7f362c949700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:11:40.505 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.504+0000 7f362c949700 1 -- 172.21.15.59:0/3427094597 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3628142290 msgr2=0x7f36280ae000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:40.505 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.504+0000 7f362c949700 1 --2- 172.21.15.59:0/3427094597 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3628142290 0x7f36280ae000 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f36280a6510 tx=0x7f3620018000 comp rx=0 tx=0).stop 2023-09-30T02:11:40.505 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.504+0000 7f362c949700 1 -- 172.21.15.59:0/3427094597 shutdown_connections 2023-09-30T02:11:40.505 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.504+0000 7f362c949700 1 --2- 172.21.15.59:0/3427094597 >> v2:172.21.15.110:6800/115870 conn(0x7f3608068890 0x7f360806ad40 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:40.506 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.504+0000 7f362c949700 1 --2- 172.21.15.59:0/3427094597 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f3628142290 0x7f36280ae000 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:40.506 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.504+0000 7f362c949700 1 --2- 172.21.15.59:0/3427094597 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f36280a5b70 0x7f3628144f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:40.506 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.504+0000 7f362c949700 1 --2- 172.21.15.59:0/3427094597 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f36280b1500 0x7f362813e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:40.506 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.504+0000 7f362c949700 1 -- 172.21.15.59:0/3427094597 >> 172.21.15.59:0/3427094597 conn(0x7f36280631f0 msgr2=0x7f3628070f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:11:40.506 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.505+0000 7f362c949700 1 -- 172.21.15.59:0/3427094597 shutdown_connections 2023-09-30T02:11:40.506 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.505+0000 7f362c949700 1 -- 172.21.15.59:0/3427094597 wait complete. 2023-09-30T02:11:40.506 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.505+0000 7f362c949700 1 librados: shutdown 2023-09-30T02:11:40.506 INFO:teuthology.orchestra.run.smithi059.stderr:instructed osd(s) 5 to deep-scrub 2023-09-30T02:11:40.517 INFO:tasks.ceph:Scrubbing osd.6 2023-09-30T02:11:40.517 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2023-09-30T02:11:40.654 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.655+0000 7f74ab7d0700 10 monclient: get_monmap_and_config 2023-09-30T02:11:40.655 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.655+0000 7f74ab7d0700 10 monclient: build_initial_monmap 2023-09-30T02:11:40.655 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.655+0000 7f74ab7d0700 10 monclient: monmap: 2023-09-30T02:11:40.655 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:11:40.655 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:40.655 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:11:40.655914+0000 2023-09-30T02:11:40.655 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:11:40.655914+0000 2023-09-30T02:11:40.655 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:11:40.655 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:40.656 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:11:40.656 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:11:40.656 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:11:40.656 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:11:40.656 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:11:40.656 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:40.658 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.656+0000 7f74ab7d0700 1 Processor -- start 2023-09-30T02:11:40.658 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.656+0000 7f74ab7d0700 1 -- start start 2023-09-30T02:11:40.658 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.656+0000 7f74ab7d0700 10 monclient: init 2023-09-30T02:11:40.658 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.656+0000 7f74ab7d0700 10 monclient: _reopen_session rank -1 2023-09-30T02:11:40.658 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.657+0000 7f74ab7d0700 10 monclient: _add_conns ranks=[2,4,0,3,1] 2023-09-30T02:11:40.658 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.657+0000 7f74ab7d0700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f74a40a5730 0x7f74a40a7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:40.659 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.657+0000 7f74ab7d0700 10 monclient(hunting): picked mon.noname-b con 0x7f74a40a5730 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:11:40.659 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.657+0000 7f74ab7d0700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f74a41418e0 0x7f74a413ae20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:40.659 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.657+0000 7f74ab7d0700 10 monclient(hunting): picked mon.noname-c con 0x7f74a41418e0 addr v2:172.21.15.110:3300/0 2023-09-30T02:11:40.659 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.657+0000 7f74ab7d0700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f74a413b360 addr v1:172.21.15.59:6789/0 2023-09-30T02:11:40.659 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.657+0000 7f74ab7d0700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74a40a7b10 con 0x7f74a413b360 2023-09-30T02:11:40.659 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.657+0000 7f74ab7d0700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:40.659 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.657+0000 7f74ab7d0700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74a40a84e0 con 0x7f74a41418e0 2023-09-30T02:11:40.659 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.657+0000 7f74ab7d0700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:40.659 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.657+0000 7f74ab7d0700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74a40a5b00 con 0x7f74a40a5730 2023-09-30T02:11:40.659 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.657+0000 7f74ab7d0700 10 monclient(hunting): _renew_subs 2023-09-30T02:11:40.660 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.657+0000 7f74ab7d0700 10 monclient(hunting): authenticate will time out at 2145.959109s 2023-09-30T02:11:40.660 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.657+0000 7f74a956c700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f74a40a5730 0x7f74a40a7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:40.660 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.657+0000 7f74a9d6d700 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f74a413b360 0x7f74a40af5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:49460/0 (socket says 172.21.15.59:49460) 2023-09-30T02:11:40.660 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.657+0000 7f74a9d6d700 1 -- 172.21.15.59:0/263515121 learned_addr learned my addr 172.21.15.59:0/263515121 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:11:40.660 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.658+0000 7f74a8d6b700 1 --2- 172.21.15.59:0/263515121 >> v2:172.21.15.110:3300/0 conn(0x7f74a41418e0 0x7f74a413ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:40.660 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.658+0000 7f74a956c700 10 monclient(hunting): get_auth_request con 0x7f74a40a5730 auth_method 0 2023-09-30T02:11:40.660 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.658+0000 7f74a956c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:40.660 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.658+0000 7f74a956c700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:40.660 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.658+0000 7f74a956c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:40.660 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.658+0000 7f74a8d6b700 10 monclient(hunting): get_auth_request con 0x7f74a41418e0 auth_method 0 2023-09-30T02:11:40.661 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.658+0000 7f74a8d6b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:40.661 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.658+0000 7f74a8d6b700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:40.661 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.658+0000 7f74a8d6b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:40.661 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.659+0000 7f74a8d6b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:40.661 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.659+0000 7f74a8d6b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:40.661 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.659+0000 7f74a8d6b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:40.661 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.659+0000 7f74a956c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:40.661 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.659+0000 7f74a956c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:40.661 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.659+0000 7f74a956c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:40.662 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.660+0000 7f7497fff700 1 -- 172.21.15.59:0/263515121 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3398768302 0 0) 0x7f74a40a7b10 con 0x7f74a413b360 2023-09-30T02:11:40.663 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.661+0000 7f7497fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:40.663 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.661+0000 7f7497fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:40.663 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.661+0000 7f7497fff700 10 monclient(hunting): my global_id is 4908 2023-09-30T02:11:40.663 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.661+0000 7f7497fff700 1 -- 172.21.15.59:0/263515121 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7490003570 con 0x7f74a413b360 2023-09-30T02:11:40.663 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.661+0000 7f74a8d6b700 10 monclient(hunting): handle_auth_done global_id 4939 payload 274 2023-09-30T02:11:40.663 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.661+0000 7f74a8d6b700 1 -- 172.21.15.59:0/263515121 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f74a40a5730 msgr2=0x7f74a40a7fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:11:40.663 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.661+0000 7f74a8d6b700 1 --2- 172.21.15.59:0/263515121 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f74a40a5730 0x7f74a40a7fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:40.663 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.661+0000 7f74a8d6b700 1 -- 172.21.15.59:0/263515121 >> v1:172.21.15.59:6789/0 conn(0x7f74a413b360 legacy=0x7f74a40af5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:40.663 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.661+0000 7f74a8d6b700 10 monclient: _finish_hunting 0 2023-09-30T02:11:40.664 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.661+0000 7f74a8d6b700 1 monclient: found mon.noname-c 2023-09-30T02:11:40.664 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.661+0000 7f74a8d6b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:11:40.664 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.661+0000 7f74a8d6b700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3300/0 2023-09-30T02:11:40.664 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.661+0000 7f74a8d6b700 1 -- 172.21.15.59:0/263515121 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74a40a7750 con 0x7f74a41418e0 2023-09-30T02:11:40.664 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.663+0000 7f74a8d6b700 1 --2- 172.21.15.59:0/263515121 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f74a41418e0 0x7f74a413ae20 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f74980098e0 tx=0x7f74980091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c4394d820b5e9db server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:40.667 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.667+0000 7f7497fff700 1 -- 172.21.15.59:0/263515121 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7498011630 con 0x7f74a41418e0 2023-09-30T02:11:40.667 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.667+0000 7f7497fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:40.667 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.667+0000 7f7497fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:11:40.667 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.667+0000 7f7497fff700 10 monclient: dump: 2023-09-30T02:11:40.667 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:40.667 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:40.667 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:40.667 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:40.668 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:40.668 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:40.668 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:40.668 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:40.668 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:40.668 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:40.668 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.667+0000 7f7497fff700 10 monclient: _finish_auth 0 2023-09-30T02:11:40.668 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.667+0000 7f7497fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:11:40.668 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.667+0000 7f7497fff700 1 -- 172.21.15.59:0/263515121 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7498011790 con 0x7f74a41418e0 2023-09-30T02:11:40.668 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.667+0000 7f74ab7d0700 5 monclient: authenticate success, global_id 4939 2023-09-30T02:11:40.668 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.667+0000 7f74ab7d0700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:11:40.669 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.667+0000 7f7497fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:11:40.669 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.667+0000 7f7497fff700 1 -- 172.21.15.59:0/263515121 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7498011b30 con 0x7f74a41418e0 2023-09-30T02:11:40.669 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.667+0000 7f74ab7d0700 10 monclient: get_monmap_and_config success 2023-09-30T02:11:40.669 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.667+0000 7f74ab7d0700 4 set_mon_vals no callback set 2023-09-30T02:11:40.669 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.667+0000 7f7497fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:40.669 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.667+0000 7f7497fff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:11:40.669 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.667+0000 7f7497fff700 10 monclient: dump: 2023-09-30T02:11:40.669 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:40.669 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:40.670 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:40.670 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:40.670 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:40.670 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:40.670 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:40.670 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:40.670 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:40.670 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:40.670 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.667+0000 7f74ab7d0700 10 monclient: shutdown 2023-09-30T02:11:40.670 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.667+0000 7f74ab7d0700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:11:40.671 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.667+0000 7f74ab7d0700 1 -- 172.21.15.59:0/263515121 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f74a41418e0 msgr2=0x7f74a413ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:40.671 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.667+0000 7f74ab7d0700 1 --2- 172.21.15.59:0/263515121 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f74a41418e0 0x7f74a413ae20 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f74980098e0 tx=0x7f74980091a0 comp rx=0 tx=0).stop 2023-09-30T02:11:40.671 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.667+0000 7f74ab7d0700 1 -- 172.21.15.59:0/263515121 shutdown_connections 2023-09-30T02:11:40.671 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.667+0000 7f74ab7d0700 1 --2- 172.21.15.59:0/263515121 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f74a40a5730 0x7f74a40a7fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:40.671 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.667+0000 7f74ab7d0700 1 --2- 172.21.15.59:0/263515121 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f74a41418e0 0x7f74a413ae20 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:40.671 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.667+0000 7f74ab7d0700 1 -- 172.21.15.59:0/263515121 >> 172.21.15.59:0/263515121 conn(0x7f74a40631f0 msgr2=0x7f74a40635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:11:40.671 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.668+0000 7f74ab7d0700 1 -- 172.21.15.59:0/263515121 shutdown_connections 2023-09-30T02:11:40.671 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.668+0000 7f74ab7d0700 1 -- 172.21.15.59:0/263515121 wait complete. 2023-09-30T02:11:40.674 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.668+0000 7f74ab7d0700 10 monclient: build_initial_monmap 2023-09-30T02:11:40.674 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.668+0000 7f74ab7d0700 10 monclient: monmap: 2023-09-30T02:11:40.674 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:11:40.674 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:11:40.674 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:11:40.675 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:11:40.675 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:11:40.675 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:11:40.675 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:11:40.675 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:11:40.675 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:11:40.675 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:40.675 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.669+0000 7f74ab7d0700 1 librados: starting msgr at 2023-09-30T02:11:40.675 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.669+0000 7f74ab7d0700 1 librados: starting objecter 2023-09-30T02:11:40.675 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.669+0000 7f74ab7d0700 1 Processor -- start 2023-09-30T02:11:40.675 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.669+0000 7f74ab7d0700 1 -- start start 2023-09-30T02:11:40.676 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.669+0000 7f74ab7d0700 1 librados: setting wanted keys 2023-09-30T02:11:40.676 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.669+0000 7f74ab7d0700 1 librados: calling monclient init 2023-09-30T02:11:40.676 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.669+0000 7f74ab7d0700 10 monclient: init 2023-09-30T02:11:40.676 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.669+0000 7f74ab7d0700 10 monclient: _reopen_session rank -1 2023-09-30T02:11:40.676 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.669+0000 7f74ab7d0700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T02:11:40.676 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.670+0000 7f74ab7d0700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f74a40a5730 0x7f74a40ad5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:40.676 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.670+0000 7f74ab7d0700 10 monclient(hunting): picked mon.noname-b con 0x7f74a40a5730 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:11:40.676 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.670+0000 7f74ab7d0700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f74a413b360 0x7f74a413c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:40.676 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.670+0000 7f74ab7d0700 10 monclient(hunting): picked mon.noname-c con 0x7f74a413b360 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:11:40.676 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.670+0000 7f74ab7d0700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f74a41418e0 0x7f74a413cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:40.676 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.670+0000 7f74ab7d0700 10 monclient(hunting): picked mon.noname-a con 0x7f74a41418e0 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:11:40.676 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.670+0000 7f74ab7d0700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:40.677 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.670+0000 7f74ab7d0700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74a40b3950 con 0x7f74a41418e0 2023-09-30T02:11:40.677 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.670+0000 7f74ab7d0700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:40.677 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.670+0000 7f74ab7d0700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74a413d240 con 0x7f74a40a5730 2023-09-30T02:11:40.677 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.670+0000 7f74ab7d0700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:40.677 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.670+0000 7f74ab7d0700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74a413dff0 con 0x7f74a413b360 2023-09-30T02:11:40.677 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.670+0000 7f74ab7d0700 10 monclient(hunting): _renew_subs 2023-09-30T02:11:40.677 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.670+0000 7f74ab7d0700 10 monclient(hunting): authenticate will time out at 2145.972090s 2023-09-30T02:11:40.677 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.670+0000 7f74a956c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f74a40a5730 0x7f74a40ad5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:40.677 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.670+0000 7f74a8d6b700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f74a413b360 0x7f74a413c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:40.677 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.670+0000 7f74a9d6d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f74a41418e0 0x7f74a413cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:40.678 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.670+0000 7f74a956c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f74a40a5730 0x7f74a40ad5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:41478/0 (socket says 172.21.15.59:41478) 2023-09-30T02:11:40.678 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.670+0000 7f74a956c700 1 -- 172.21.15.59:0/2948143369 learned_addr learned my addr 172.21.15.59:0/2948143369 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:11:40.678 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.670+0000 7f74a956c700 10 monclient(hunting): get_auth_request con 0x7f74a40a5730 auth_method 0 2023-09-30T02:11:40.678 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.670+0000 7f74a956c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:40.678 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.670+0000 7f74a956c700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:40.678 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.670+0000 7f74a956c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:40.678 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.671+0000 7f74a8d6b700 10 monclient(hunting): get_auth_request con 0x7f74a413b360 auth_method 0 2023-09-30T02:11:40.678 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.671+0000 7f74a8d6b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:40.678 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.671+0000 7f74a8d6b700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:40.678 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.671+0000 7f74a8d6b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:40.679 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.671+0000 7f74a9d6d700 10 monclient(hunting): get_auth_request con 0x7f74a41418e0 auth_method 0 2023-09-30T02:11:40.679 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.671+0000 7f74a9d6d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:40.679 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.671+0000 7f74a9d6d700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:40.679 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.671+0000 7f74a9d6d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:40.679 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.671+0000 7f74a956c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:40.679 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.671+0000 7f74a956c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:40.679 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.671+0000 7f74a956c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:40.679 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.671+0000 7f74a8d6b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:40.679 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.671+0000 7f74a8d6b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:40.679 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.671+0000 7f74a8d6b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:40.680 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.672+0000 7f74a9d6d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:40.680 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.672+0000 7f74a9d6d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:40.680 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.672+0000 7f74a9d6d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:40.680 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.674+0000 7f74a956c700 10 monclient(hunting): handle_auth_done global_id 4942 payload 824 2023-09-30T02:11:40.680 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.674+0000 7f74a956c700 1 -- 172.21.15.59:0/2948143369 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f74a413b360 msgr2=0x7f74a413c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:40.680 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.674+0000 7f74a956c700 1 --2- 172.21.15.59:0/2948143369 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f74a413b360 0x7f74a413c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:40.680 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.674+0000 7f74a956c700 1 -- 172.21.15.59:0/2948143369 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f74a41418e0 msgr2=0x7f74a413cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:40.680 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.674+0000 7f74a956c700 1 --2- 172.21.15.59:0/2948143369 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f74a41418e0 0x7f74a413cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:40.680 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.674+0000 7f74a956c700 10 monclient: _finish_hunting 0 2023-09-30T02:11:40.680 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.674+0000 7f74a956c700 1 monclient: found mon.noname-b 2023-09-30T02:11:40.680 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.674+0000 7f74a956c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:11:40.681 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.674+0000 7f74a956c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:11:40.681 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.674+0000 7f74a956c700 1 -- 172.21.15.59:0/2948143369 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7498009420 con 0x7f74a40a5730 2023-09-30T02:11:40.682 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.676+0000 7f74a956c700 1 --2- 172.21.15.59:0/2948143369 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f74a40a5730 0x7f74a40ad5d0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f74a000e930 tx=0x7f74a00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:40.682 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.676+0000 7f74957fa700 10 client.?.objecter ms_handle_connect 0x7f74a40a5730 2023-09-30T02:11:40.682 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.676+0000 7f74957fa700 10 client.?.objecter resend_mon_ops 2023-09-30T02:11:40.682 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.680+0000 7f74957fa700 1 -- 172.21.15.59:0/2948143369 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74a0013680 con 0x7f74a40a5730 2023-09-30T02:11:40.682 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.680+0000 7f74957fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:40.682 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.680+0000 7f74957fa700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:11:40.683 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.680+0000 7f74957fa700 10 monclient: dump: 2023-09-30T02:11:40.683 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:40.683 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:40.683 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:40.683 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:40.683 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:40.683 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:40.683 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:40.683 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:40.683 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:40.684 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:40.684 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.680+0000 7f74957fa700 10 monclient: _finish_auth 0 2023-09-30T02:11:40.684 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.680+0000 7f74957fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:11:40.684 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.680+0000 7f74957fa700 1 -- 172.21.15.59:0/2948143369 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74a00137e0 con 0x7f74a40a5730 2023-09-30T02:11:40.684 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.680+0000 7f74957fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:11:40.684 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.680+0000 7f74957fa700 1 -- 172.21.15.59:0/2948143369 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74a0013b80 con 0x7f74a40a5730 2023-09-30T02:11:40.684 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.680+0000 7f74957fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:40.684 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.680+0000 7f74957fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:11:40.684 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.680+0000 7f74957fa700 10 monclient: dump: 2023-09-30T02:11:40.684 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:40.685 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:40.685 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:40.685 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:40.685 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:40.685 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:40.685 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:40.685 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:40.685 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:40.685 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:40.685 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.680+0000 7f74ab7d0700 5 monclient: authenticate success, global_id 4942 2023-09-30T02:11:40.686 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.680+0000 7f74ab7d0700 10 monclient: _renew_subs 2023-09-30T02:11:40.686 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.680+0000 7f74ab7d0700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:11:40.686 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.680+0000 7f74977fe700 4 set_mon_vals no callback set 2023-09-30T02:11:40.686 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.680+0000 7f74ab7d0700 1 -- 172.21.15.59:0/2948143369 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74a413e270 con 0x7f74a40a5730 2023-09-30T02:11:40.686 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.680+0000 7f74ab7d0700 10 client.4942.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:11:40.686 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.680+0000 7f74ab7d0700 10 monclient: _renew_subs 2023-09-30T02:11:40.686 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.680+0000 7f74ab7d0700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:11:40.686 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.680+0000 7f74ab7d0700 1 -- 172.21.15.59:0/2948143369 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74a413e760 con 0x7f74a40a5730 2023-09-30T02:11:40.686 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.680+0000 7f74ab7d0700 1 librados: init done 2023-09-30T02:11:40.687 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.681+0000 7f74ab7d0700 10 client.4942.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:11:40.687 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.681+0000 7f74ab7d0700 15 client.4942.objecter _session_command_op_assign -1 1 2023-09-30T02:11:40.687 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.681+0000 7f74ab7d0700 20 client.4942.objecter _get_session osd=-1 returning homeless 2023-09-30T02:11:40.687 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.681+0000 7f74ab7d0700 10 client.4942.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:11:40.687 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.681+0000 7f74ab7d0700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:11:40.687 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.681+0000 7f74ab7d0700 1 -- 172.21.15.59:0/2948143369 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f74a4142220 con 0x7f74a40a5730 2023-09-30T02:11:40.706 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.705+0000 7f74957fa700 1 -- 172.21.15.59:0/2948143369 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f74a0025420 con 0x7f74a40a5730 2023-09-30T02:11:40.706 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.705+0000 7f74957fa700 1 --2- 172.21.15.59:0/2948143369 >> v2:172.21.15.110:6800/115870 conn(0x7f74880689e0 0x7f748806ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:40.706 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.705+0000 7f74957fa700 1 -- 172.21.15.59:0/2948143369 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f74a0084320 con 0x7f74a40a5730 2023-09-30T02:11:40.706 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.705+0000 7f74957fa700 10 client.4942.objecter ms_dispatch 0x7f74a4001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:11:40.706 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.705+0000 7f74957fa700 3 client.4942.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:11:40.707 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.705+0000 7f74957fa700 3 client.4942.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:11:40.707 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.706+0000 7f74957fa700 10 client.4942.objecter checking command 1 2023-09-30T02:11:40.707 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.706+0000 7f74957fa700 1 --2- 172.21.15.59:0/2948143369 >> v2:172.21.15.110:6813/116563 conn(0x7f748806e950 0x7f7488070d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:40.707 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.706+0000 7f74957fa700 20 client.4942.objecter _get_session s=0x7f748806e2a0 osd=6 3 2023-09-30T02:11:40.707 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.706+0000 7f74957fa700 20 client.4942.objecter put_session s=0x7f748806e2a0 osd=6 3 2023-09-30T02:11:40.707 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.706+0000 7f74957fa700 15 client.4942.objecter _session_command_op_remove -1 1 2023-09-30T02:11:40.707 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.706+0000 7f74957fa700 20 client.4942.objecter _get_session s=0x7f748806e2a0 osd=6 3 2023-09-30T02:11:40.707 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.706+0000 7f74957fa700 20 client.4942.objecter get_session s=0x7f748806e2a0 osd=6 3 2023-09-30T02:11:40.707 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.706+0000 7f74957fa700 15 client.4942.objecter _session_command_op_assign 6 1 2023-09-30T02:11:40.707 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.706+0000 7f74957fa700 20 client.4942.objecter put_session s=0x7f748806e2a0 osd=6 4 2023-09-30T02:11:40.708 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.706+0000 7f74957fa700 10 client.4942.objecter _send_command 1 2023-09-30T02:11:40.708 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.706+0000 7f74957fa700 1 -- 172.21.15.59:0/2948143369 --> v2:172.21.15.110:6813/116563 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f74880713d0 con 0x7f748806e950 2023-09-30T02:11:40.708 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.706+0000 7f74957fa700 20 client.4942.objecter dump_active .. 0 homeless 2023-09-30T02:11:40.708 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.706+0000 7f74957fa700 1 -- 172.21.15.59:0/2948143369 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f74a0084650 con 0x7f74a40a5730 2023-09-30T02:11:40.708 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.706+0000 7f74957fa700 10 monclient: handle_get_version_reply finishing 1 version 25 2023-09-30T02:11:40.708 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.706+0000 7f74a8d6b700 1 --2- 172.21.15.59:0/2948143369 >> v2:172.21.15.110:6800/115870 conn(0x7f74880689e0 0x7f748806ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:40.708 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.706+0000 7f74a8d6b700 10 monclient: get_auth_request con 0x7f74880689e0 auth_method 0 2023-09-30T02:11:40.708 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.706+0000 7f74a8d6b700 1 --2- 172.21.15.59:0/2948143369 >> v2:172.21.15.110:6800/115870 conn(0x7f74880689e0 0x7f748806ae90 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f74980091e0 tx=0x7f7498017000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:40.708 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.706+0000 7f74957fa700 10 client.4942.objecter ms_handle_connect 0x7f74880689e0 2023-09-30T02:11:40.708 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.706+0000 7f74a9d6d700 1 --2- 172.21.15.59:0/2948143369 >> v2:172.21.15.110:6813/116563 conn(0x7f748806e950 0x7f7488070d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:40.708 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.707+0000 7f74a9d6d700 10 monclient: get_auth_request con 0x7f748806e950 auth_method 0 2023-09-30T02:11:40.709 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.709+0000 7f74a9d6d700 1 --2- 172.21.15.59:0/2948143369 >> v2:172.21.15.110:6813/116563 conn(0x7f748806e950 0x7f7488070d60 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:40.709 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.709+0000 7f74957fa700 10 client.4942.objecter ms_handle_connect 0x7f748806e950 2023-09-30T02:11:40.736 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.736+0000 7f74957fa700 1 -- 172.21.15.59:0/2948143369 <== osd.6 v2:172.21.15.110:6813/116563 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f74880713d0 con 0x7f748806e950 2023-09-30T02:11:40.736 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.736+0000 7f74957fa700 10 client.4942.objecter ms_dispatch 0x7f74a4001550 command_reply(tid 1: 0 ) v1 2023-09-30T02:11:40.736 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.736+0000 7f74957fa700 10 client.4942.objecter _finish_command 1 = system:0 2023-09-30T02:11:40.736 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.736+0000 7f74957fa700 20 client.4942.objecter put_session s=0x7f748806e2a0 osd=6 4 2023-09-30T02:11:40.736 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.736+0000 7f74957fa700 15 client.4942.objecter _session_command_op_remove 6 1 2023-09-30T02:11:40.751 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.752+0000 7f74ab7d0700 10 client.4942.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-09-30T02:11:40.751 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.752+0000 7f74ab7d0700 15 client.4942.objecter _session_command_op_assign -1 2 2023-09-30T02:11:40.751 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.752+0000 7f74ab7d0700 20 client.4942.objecter _get_session s=0x7f748806e2a0 osd=6 3 2023-09-30T02:11:40.751 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.752+0000 7f74ab7d0700 20 client.4942.objecter put_session s=0x7f748806e2a0 osd=6 3 2023-09-30T02:11:40.752 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.752+0000 7f74ab7d0700 20 client.4942.objecter _get_session s=0x7f748806e2a0 osd=6 3 2023-09-30T02:11:40.752 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.752+0000 7f74ab7d0700 15 client.4942.objecter _session_command_op_remove -1 2 2023-09-30T02:11:40.752 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.752+0000 7f74ab7d0700 20 client.4942.objecter get_session s=0x7f748806e2a0 osd=6 3 2023-09-30T02:11:40.752 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.752+0000 7f74ab7d0700 15 client.4942.objecter _session_command_op_assign 6 2 2023-09-30T02:11:40.752 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.752+0000 7f74ab7d0700 20 client.4942.objecter put_session s=0x7f748806e2a0 osd=6 4 2023-09-30T02:11:40.752 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.752+0000 7f74ab7d0700 10 client.4942.objecter _send_command 2 2023-09-30T02:11:40.752 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.752+0000 7f74ab7d0700 1 -- 172.21.15.59:0/2948143369 --> v2:172.21.15.110:6813/116563 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f74a4064a00 con 0x7f748806e950 2023-09-30T02:11:40.773 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.773+0000 7f74957fa700 1 -- 172.21.15.59:0/2948143369 <== osd.6 v2:172.21.15.110:6813/116563 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f74a4064a00 con 0x7f748806e950 2023-09-30T02:11:40.773 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.773+0000 7f74957fa700 10 client.4942.objecter ms_dispatch 0x7f74a4001550 command_reply(tid 2: 0 ) v1 2023-09-30T02:11:40.773 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.773+0000 7f74957fa700 10 client.4942.objecter _finish_command 2 = system:0 2023-09-30T02:11:40.773 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.773+0000 7f74957fa700 20 client.4942.objecter put_session s=0x7f748806e2a0 osd=6 4 2023-09-30T02:11:40.773 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.773+0000 7f74957fa700 15 client.4942.objecter _session_command_op_remove 6 2 2023-09-30T02:11:40.773 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2023-09-30T02:11:40.774 INFO:teuthology.orchestra.run.smithi059.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T02:11:40.774 INFO:teuthology.orchestra.run.smithi059.stdout:} 2023-09-30T02:11:40.775 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.774+0000 7f74ab7d0700 10 librados: watch_flush enter 2023-09-30T02:11:40.775 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.774+0000 7f74ab7d0700 10 librados: watch_flush exit 2023-09-30T02:11:40.775 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.774+0000 7f74ab7d0700 10 client.4942.objecter close_session for osd.6 2023-09-30T02:11:40.775 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.774+0000 7f74ab7d0700 1 -- 172.21.15.59:0/2948143369 >> v2:172.21.15.110:6813/116563 conn(0x7f748806e950 msgr2=0x7f7488070d60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:40.776 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.774+0000 7f74ab7d0700 1 --2- 172.21.15.59:0/2948143369 >> v2:172.21.15.110:6813/116563 conn(0x7f748806e950 0x7f7488070d60 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:40.776 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.774+0000 7f74ab7d0700 20 client.4942.objecter put_session s=0x7f748806e2a0 osd=6 1 2023-09-30T02:11:40.776 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.774+0000 7f74ab7d0700 20 client.4942.objecter shutdown clearing up homeless session... 2023-09-30T02:11:40.776 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.774+0000 7f74ab7d0700 10 client.4942.objecter successfully canceled tick 2023-09-30T02:11:40.776 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.774+0000 7f74ab7d0700 1 -- 172.21.15.59:0/2948143369 >> v2:172.21.15.110:6800/115870 conn(0x7f74880689e0 msgr2=0x7f748806ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:40.776 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.774+0000 7f74ab7d0700 1 --2- 172.21.15.59:0/2948143369 >> v2:172.21.15.110:6800/115870 conn(0x7f74880689e0 0x7f748806ae90 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f74980091e0 tx=0x7f7498017000 comp rx=0 tx=0).stop 2023-09-30T02:11:40.776 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.774+0000 7f74ab7d0700 10 monclient: shutdown 2023-09-30T02:11:40.776 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.774+0000 7f74ab7d0700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:11:40.776 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.774+0000 7f74ab7d0700 1 -- 172.21.15.59:0/2948143369 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f74a40a5730 msgr2=0x7f74a40ad5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:40.777 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.774+0000 7f74ab7d0700 1 --2- 172.21.15.59:0/2948143369 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f74a40a5730 0x7f74a40ad5d0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f74a000e930 tx=0x7f74a00091a0 comp rx=0 tx=0).stop 2023-09-30T02:11:40.777 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.774+0000 7f74a9d6d700 1 -- 172.21.15.59:0/2948143369 reap_dead start 2023-09-30T02:11:40.777 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.774+0000 7f74ab7d0700 1 -- 172.21.15.59:0/2948143369 shutdown_connections 2023-09-30T02:11:40.777 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.774+0000 7f74ab7d0700 1 -- 172.21.15.59:0/2948143369 >> 172.21.15.59:0/2948143369 conn(0x7f74a40631f0 msgr2=0x7f74a40635c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:11:40.777 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.774+0000 7f74ab7d0700 1 -- 172.21.15.59:0/2948143369 shutdown_connections 2023-09-30T02:11:40.777 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.774+0000 7f74ab7d0700 1 -- 172.21.15.59:0/2948143369 wait complete. 2023-09-30T02:11:40.777 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.774+0000 7f74ab7d0700 1 librados: shutdown 2023-09-30T02:11:40.786 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2023-09-30T02:11:40.920 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.921+0000 7ff35a8eb700 10 monclient: get_monmap_and_config 2023-09-30T02:11:40.921 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.921+0000 7ff35a8eb700 10 monclient: build_initial_monmap 2023-09-30T02:11:40.921 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.921+0000 7ff35a8eb700 10 monclient: monmap: 2023-09-30T02:11:40.921 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:11:40.921 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:40.921 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:11:40.921840+0000 2023-09-30T02:11:40.921 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:11:40.921840+0000 2023-09-30T02:11:40.921 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:11:40.921 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:40.921 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:11:40.922 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:11:40.922 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:11:40.922 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:11:40.922 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:11:40.922 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:40.923 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.922+0000 7ff35a8eb700 1 Processor -- start 2023-09-30T02:11:40.924 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.922+0000 7ff35a8eb700 1 -- start start 2023-09-30T02:11:40.924 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.922+0000 7ff35a8eb700 10 monclient: init 2023-09-30T02:11:40.924 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.922+0000 7ff35a8eb700 10 monclient: _reopen_session rank -1 2023-09-30T02:11:40.924 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.922+0000 7ff35a8eb700 10 monclient: _add_conns ranks=[1,2,0,3,4] 2023-09-30T02:11:40.924 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.923+0000 7ff35a8eb700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff3540a5b70 addr v1:172.21.15.110:6789/0 2023-09-30T02:11:40.924 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.923+0000 7ff35a8eb700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff354142290 0x7ff3540b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:40.924 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.923+0000 7ff35a8eb700 10 monclient(hunting): picked mon.noname-b con 0x7ff354142290 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:11:40.924 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.923+0000 7ff35a8eb700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff3540a5f40 addr v1:172.21.15.59:6789/0 2023-09-30T02:11:40.924 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.923+0000 7ff35a8eb700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3540a6380 con 0x7ff3540a5f40 2023-09-30T02:11:40.924 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.923+0000 7ff35a8eb700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3540a75a0 con 0x7ff3540a5b70 2023-09-30T02:11:40.924 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.923+0000 7ff35a8eb700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:40.924 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.923+0000 7ff35a8eb700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3540af3f0 con 0x7ff354142290 2023-09-30T02:11:40.925 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.923+0000 7ff35a8eb700 10 monclient(hunting): _renew_subs 2023-09-30T02:11:40.925 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.923+0000 7ff35a8eb700 10 monclient(hunting): authenticate will time out at 2146.225117s 2023-09-30T02:11:40.925 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.923+0000 7ff3537fe700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff354142290 0x7ff3540b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:40.925 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.923+0000 7ff358e88700 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7ff3540a5f40 0x7ff35413af30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:49466/0 (socket says 172.21.15.59:49466) 2023-09-30T02:11:40.925 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.923+0000 7ff358e88700 1 -- 172.21.15.59:0/956095484 learned_addr learned my addr 172.21.15.59:0/956095484 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:11:40.925 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.924+0000 7ff3537fe700 10 monclient(hunting): get_auth_request con 0x7ff354142290 auth_method 0 2023-09-30T02:11:40.925 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.924+0000 7ff3537fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:40.925 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.924+0000 7ff3537fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:40.925 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.924+0000 7ff3537fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:40.925 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.924+0000 7ff3537fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:40.925 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.924+0000 7ff3537fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:40.926 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.924+0000 7ff3537fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:40.927 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.926+0000 7ff352ffd700 1 -- 172.21.15.59:0/956095484 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2429363518 0 0) 0x7ff3540a6380 con 0x7ff3540a5f40 2023-09-30T02:11:40.927 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.926+0000 7ff352ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:40.927 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.926+0000 7ff352ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:40.927 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.926+0000 7ff352ffd700 10 monclient(hunting): my global_id is 4914 2023-09-30T02:11:40.927 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.926+0000 7ff352ffd700 1 -- 172.21.15.59:0/956095484 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff340003570 con 0x7ff3540a5f40 2023-09-30T02:11:40.927 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.927+0000 7ff352ffd700 1 -- 172.21.15.59:0/956095484 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1102230609 0 0) 0x7ff3540a75a0 con 0x7ff3540a5b70 2023-09-30T02:11:40.927 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.927+0000 7ff352ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:40.928 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.927+0000 7ff352ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:40.928 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.927+0000 7ff352ffd700 10 monclient(hunting): my global_id is 4945 2023-09-30T02:11:40.928 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.927+0000 7ff352ffd700 1 -- 172.21.15.59:0/956095484 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3540a6380 con 0x7ff3540a5b70 2023-09-30T02:11:40.928 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.927+0000 7ff3537fe700 10 monclient(hunting): handle_auth_done global_id 4811 payload 274 2023-09-30T02:11:40.928 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.927+0000 7ff3537fe700 1 -- 172.21.15.59:0/956095484 >> v1:172.21.15.110:6789/0 conn(0x7ff3540a5b70 legacy=0x7ff354141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:40.928 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.927+0000 7ff3537fe700 1 -- 172.21.15.59:0/956095484 >> v1:172.21.15.59:6789/0 conn(0x7ff3540a5f40 legacy=0x7ff35413af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:40.928 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.927+0000 7ff3537fe700 10 monclient: _finish_hunting 0 2023-09-30T02:11:40.928 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.927+0000 7ff3537fe700 1 monclient: found mon.noname-b 2023-09-30T02:11:40.928 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.927+0000 7ff3537fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:11:40.928 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.927+0000 7ff3537fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.59:3301/0 2023-09-30T02:11:40.928 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.927+0000 7ff3537fe700 1 -- 172.21.15.59:0/956095484 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff35413b640 con 0x7ff354142290 2023-09-30T02:11:40.929 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.929+0000 7ff3537fe700 1 --2- 172.21.15.59:0/956095484 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff354142290 0x7ff3540b1500 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ff344009910 tx=0x7ff3440091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=61c09f64533feb6b server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:40.933 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.933+0000 7ff352ffd700 1 -- 172.21.15.59:0/956095484 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff344011630 con 0x7ff354142290 2023-09-30T02:11:40.933 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.933+0000 7ff352ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:40.933 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.933+0000 7ff352ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:11:40.933 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.933+0000 7ff352ffd700 10 monclient: dump: 2023-09-30T02:11:40.933 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:40.933 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:40.933 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:40.934 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:40.934 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:40.934 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:40.934 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:40.934 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:40.934 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:40.934 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:40.934 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.933+0000 7ff352ffd700 10 monclient: _finish_auth 0 2023-09-30T02:11:40.934 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.933+0000 7ff352ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:11:40.934 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.933+0000 7ff352ffd700 1 -- 172.21.15.59:0/956095484 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff344011790 con 0x7ff354142290 2023-09-30T02:11:40.934 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.933+0000 7ff352ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:11:40.935 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.933+0000 7ff352ffd700 1 -- 172.21.15.59:0/956095484 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff344011b30 con 0x7ff354142290 2023-09-30T02:11:40.935 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.933+0000 7ff352ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:40.935 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.933+0000 7ff352ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:11:40.935 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.933+0000 7ff352ffd700 10 monclient: dump: 2023-09-30T02:11:40.935 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:40.935 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:40.935 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:40.935 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:40.935 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:40.935 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:40.935 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:40.935 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:40.936 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:40.936 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:40.936 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.933+0000 7ff35a8eb700 5 monclient: authenticate success, global_id 4811 2023-09-30T02:11:40.936 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.933+0000 7ff35a8eb700 10 monclient: get_monmap_and_config success 2023-09-30T02:11:40.936 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.933+0000 7ff35a8eb700 4 set_mon_vals no callback set 2023-09-30T02:11:40.939 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.934+0000 7ff35a8eb700 10 monclient: shutdown 2023-09-30T02:11:40.939 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.934+0000 7ff35a8eb700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:11:40.940 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.934+0000 7ff35a8eb700 1 -- 172.21.15.59:0/956095484 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff354142290 msgr2=0x7ff3540b1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:40.940 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.934+0000 7ff35a8eb700 1 --2- 172.21.15.59:0/956095484 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff354142290 0x7ff3540b1500 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7ff344009910 tx=0x7ff3440091a0 comp rx=0 tx=0).stop 2023-09-30T02:11:40.940 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.934+0000 7ff35a8eb700 1 -- 172.21.15.59:0/956095484 shutdown_connections 2023-09-30T02:11:40.940 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.934+0000 7ff35a8eb700 1 --2- 172.21.15.59:0/956095484 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff354142290 0x7ff3540b1500 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:40.940 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.934+0000 7ff35a8eb700 1 -- 172.21.15.59:0/956095484 >> 172.21.15.59:0/956095484 conn(0x7ff3540631f0 msgr2=0x7ff3540635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:11:40.940 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.934+0000 7ff35a8eb700 1 -- 172.21.15.59:0/956095484 shutdown_connections 2023-09-30T02:11:40.940 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.934+0000 7ff35a8eb700 1 -- 172.21.15.59:0/956095484 wait complete. 2023-09-30T02:11:40.940 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.935+0000 7ff35a8eb700 10 monclient: build_initial_monmap 2023-09-30T02:11:40.940 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.935+0000 7ff35a8eb700 10 monclient: monmap: 2023-09-30T02:11:40.940 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:11:40.940 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:11:40.940 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:11:40.941 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:11:40.941 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:11:40.941 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:11:40.941 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:11:40.941 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:11:40.941 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:11:40.941 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:40.941 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.935+0000 7ff35a8eb700 1 librados: starting msgr at 2023-09-30T02:11:40.941 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.935+0000 7ff35a8eb700 1 librados: starting objecter 2023-09-30T02:11:40.941 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.935+0000 7ff35a8eb700 1 Processor -- start 2023-09-30T02:11:40.941 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.936+0000 7ff35a8eb700 1 -- start start 2023-09-30T02:11:40.941 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.936+0000 7ff35a8eb700 1 librados: setting wanted keys 2023-09-30T02:11:40.942 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.936+0000 7ff35a8eb700 1 librados: calling monclient init 2023-09-30T02:11:40.942 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.936+0000 7ff35a8eb700 10 monclient: init 2023-09-30T02:11:40.942 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.936+0000 7ff35a8eb700 10 monclient: _reopen_session rank -1 2023-09-30T02:11:40.942 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.936+0000 7ff35a8eb700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T02:11:40.942 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.936+0000 7ff35a8eb700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff3540a5b70 0x7ff3540a7b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:40.942 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.936+0000 7ff35a8eb700 10 monclient(hunting): picked mon.noname-c con 0x7ff3540a5b70 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:11:40.942 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.936+0000 7ff35a8eb700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3540a5f40 0x7ff3540a8210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:40.942 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.936+0000 7ff35a8eb700 10 monclient(hunting): picked mon.noname-a con 0x7ff3540a5f40 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:11:40.942 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.936+0000 7ff35a8eb700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff354142290 0x7ff354150310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:40.942 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.936+0000 7ff35a8eb700 10 monclient(hunting): picked mon.noname-b con 0x7ff354142290 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:11:40.942 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.936+0000 7ff35a8eb700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:40.943 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.936+0000 7ff35a8eb700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3540aa020 con 0x7ff3540a5f40 2023-09-30T02:11:40.943 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.936+0000 7ff35a8eb700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:40.943 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.936+0000 7ff35a8eb700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3540a8b00 con 0x7ff354142290 2023-09-30T02:11:40.943 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.936+0000 7ff35a8eb700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:40.943 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.936+0000 7ff35a8eb700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3540a8750 con 0x7ff3540a5b70 2023-09-30T02:11:40.943 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.936+0000 7ff35a8eb700 10 monclient(hunting): _renew_subs 2023-09-30T02:11:40.943 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.936+0000 7ff35a8eb700 10 monclient(hunting): authenticate will time out at 2146.238971s 2023-09-30T02:11:40.943 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.937+0000 7ff353fff700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff3540a5b70 0x7ff3540a7b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:40.943 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.937+0000 7ff358e88700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff354142290 0x7ff354150310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:40.943 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.937+0000 7ff3537fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3540a5f40 0x7ff3540a8210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:40.943 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.937+0000 7ff353fff700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff3540a5b70 0x7ff3540a7b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:46244/0 (socket says 172.21.15.59:46244) 2023-09-30T02:11:40.943 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.937+0000 7ff353fff700 1 -- 172.21.15.59:0/3256288335 learned_addr learned my addr 172.21.15.59:0/3256288335 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:11:40.944 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.937+0000 7ff353fff700 10 monclient(hunting): get_auth_request con 0x7ff3540a5b70 auth_method 0 2023-09-30T02:11:40.944 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.937+0000 7ff353fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:40.944 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.937+0000 7ff353fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:40.944 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.937+0000 7ff353fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:40.944 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.937+0000 7ff358e88700 10 monclient(hunting): get_auth_request con 0x7ff354142290 auth_method 0 2023-09-30T02:11:40.944 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.937+0000 7ff358e88700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:40.944 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.937+0000 7ff358e88700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:40.944 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.937+0000 7ff358e88700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:40.944 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.938+0000 7ff3537fe700 10 monclient(hunting): get_auth_request con 0x7ff3540a5f40 auth_method 0 2023-09-30T02:11:40.944 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.938+0000 7ff3537fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:40.944 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.938+0000 7ff3537fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:40.944 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.938+0000 7ff3537fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:40.944 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.938+0000 7ff353fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:40.945 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.938+0000 7ff353fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:40.945 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.938+0000 7ff353fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:40.945 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.938+0000 7ff358e88700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:40.945 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.938+0000 7ff358e88700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:40.945 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.938+0000 7ff358e88700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:40.945 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.938+0000 7ff3537fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:40.945 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.938+0000 7ff3537fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:40.945 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.938+0000 7ff3537fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:40.946 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.941+0000 7ff358e88700 10 monclient(hunting): handle_auth_done global_id 4948 payload 824 2023-09-30T02:11:40.946 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.941+0000 7ff358e88700 1 -- 172.21.15.59:0/3256288335 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff3540a5b70 msgr2=0x7ff3540a7b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:40.946 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.941+0000 7ff358e88700 1 --2- 172.21.15.59:0/3256288335 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff3540a5b70 0x7ff3540a7b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:40.946 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.941+0000 7ff358e88700 1 -- 172.21.15.59:0/3256288335 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3540a5f40 msgr2=0x7ff3540a8210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:40.946 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.941+0000 7ff358e88700 1 --2- 172.21.15.59:0/3256288335 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3540a5f40 0x7ff3540a8210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:40.946 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.941+0000 7ff358e88700 10 monclient: _finish_hunting 0 2023-09-30T02:11:40.946 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.941+0000 7ff358e88700 1 monclient: found mon.noname-b 2023-09-30T02:11:40.947 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.941+0000 7ff358e88700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:11:40.947 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.941+0000 7ff358e88700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:11:40.947 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.941+0000 7ff358e88700 1 -- 172.21.15.59:0/3256288335 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff344009420 con 0x7ff354142290 2023-09-30T02:11:40.947 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.944+0000 7ff358e88700 1 --2- 172.21.15.59:0/3256288335 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff354142290 0x7ff354150310 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7ff34c005c50 tx=0x7ff34c001a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:40.947 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.944+0000 7ff34bfff700 10 client.?.objecter ms_handle_connect 0x7ff354142290 2023-09-30T02:11:40.947 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.944+0000 7ff34bfff700 10 client.?.objecter resend_mon_ops 2023-09-30T02:11:40.948 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.948+0000 7ff34bfff700 1 -- 172.21.15.59:0/3256288335 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff34c00a840 con 0x7ff354142290 2023-09-30T02:11:40.948 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.948+0000 7ff34bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:40.949 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.948+0000 7ff34bfff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:11:40.949 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.948+0000 7ff34bfff700 10 monclient: dump: 2023-09-30T02:11:40.949 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:40.949 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:40.949 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:40.949 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:40.949 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:40.949 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:40.949 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:40.949 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:40.949 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:40.949 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:40.950 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.948+0000 7ff34bfff700 10 monclient: _finish_auth 0 2023-09-30T02:11:40.950 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.948+0000 7ff34bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:11:40.950 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.948+0000 7ff34bfff700 1 -- 172.21.15.59:0/3256288335 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff34c00a9a0 con 0x7ff354142290 2023-09-30T02:11:40.950 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.948+0000 7ff35a8eb700 5 monclient: authenticate success, global_id 4948 2023-09-30T02:11:40.950 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.948+0000 7ff35a8eb700 10 monclient: _renew_subs 2023-09-30T02:11:40.950 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.948+0000 7ff35a8eb700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:11:40.950 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.948+0000 7ff35a8eb700 1 -- 172.21.15.59:0/3256288335 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff35413e630 con 0x7ff354142290 2023-09-30T02:11:40.950 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.948+0000 7ff34bfff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:11:40.950 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.948+0000 7ff34bfff700 1 -- 172.21.15.59:0/3256288335 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff34c00ad60 con 0x7ff354142290 2023-09-30T02:11:40.950 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.948+0000 7ff34bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:40.950 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.948+0000 7ff34bfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:11:40.951 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.948+0000 7ff34bfff700 10 monclient: dump: 2023-09-30T02:11:40.951 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:40.951 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:40.951 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:40.951 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:40.951 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:40.951 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:40.951 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:40.951 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:40.951 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:40.951 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:40.951 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.948+0000 7ff3527fc700 4 set_mon_vals no callback set 2023-09-30T02:11:40.952 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.948+0000 7ff35a8eb700 10 client.4948.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:11:40.952 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.948+0000 7ff35a8eb700 10 monclient: _renew_subs 2023-09-30T02:11:40.952 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.948+0000 7ff35a8eb700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:11:40.952 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.948+0000 7ff35a8eb700 1 -- 172.21.15.59:0/3256288335 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff354150a90 con 0x7ff354142290 2023-09-30T02:11:40.952 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.948+0000 7ff35a8eb700 1 librados: init done 2023-09-30T02:11:40.952 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.949+0000 7ff35a8eb700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:11:40.952 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.949+0000 7ff35a8eb700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:11:40.952 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.949+0000 7ff35a8eb700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:11:40.952 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.949+0000 7ff35a8eb700 1 -- 172.21.15.59:0/3256288335 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3540b0de0 con 0x7ff354142290 2023-09-30T02:11:40.972 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.973+0000 7ff34bfff700 1 -- 172.21.15.59:0/3256288335 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7ff34c007300 con 0x7ff354142290 2023-09-30T02:11:40.973 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.973+0000 7ff34bfff700 1 --2- 172.21.15.59:0/3256288335 >> v2:172.21.15.110:6800/115870 conn(0x7ff334068890 0x7ff33406ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:40.973 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.973+0000 7ff34bfff700 1 -- 172.21.15.59:0/3256288335 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7ff34c084d20 con 0x7ff354142290 2023-09-30T02:11:40.973 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.973+0000 7ff34bfff700 10 client.4948.objecter ms_dispatch 0x7ff354001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:11:40.973 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.973+0000 7ff34bfff700 3 client.4948.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:11:40.973 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.973+0000 7ff34bfff700 3 client.4948.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:11:40.973 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.973+0000 7ff34bfff700 20 client.4948.objecter dump_active .. 0 homeless 2023-09-30T02:11:40.973 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.973+0000 7ff353fff700 1 --2- 172.21.15.59:0/3256288335 >> v2:172.21.15.110:6800/115870 conn(0x7ff334068890 0x7ff33406ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:40.973 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.973+0000 7ff353fff700 10 monclient: get_auth_request con 0x7ff334068890 auth_method 0 2023-09-30T02:11:40.973 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.974+0000 7ff34bfff700 10 client.4948.objecter ms_handle_connect 0x7ff334068890 2023-09-30T02:11:40.974 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:40.974+0000 7ff353fff700 1 --2- 172.21.15.59:0/3256288335 >> v2:172.21.15.110:6800/115870 conn(0x7ff334068890 0x7ff33406ad40 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff33c007fd0 tx=0x7ff33c007e10 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:41.206 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.207+0000 7ff34bfff700 1 -- 172.21.15.59:0/3256288335 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff34c052640 con 0x7ff354142290 2023-09-30T02:11:41.207 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.207+0000 7ff34bfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:11:41.207 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.207+0000 7ff34bfff700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:11:41.443 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.443+0000 7ff35a8eb700 1 -- 172.21.15.59:0/3256288335 --> v2:172.21.15.110:6800/115870 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7ff3540b0fe0 con 0x7ff334068890 2023-09-30T02:11:41.443 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.444+0000 7ff34bfff700 1 -- 172.21.15.59:0/3256288335 <== mgr.4108 v2:172.21.15.110:6800/115870 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7ff3540b0fe0 con 0x7ff334068890 2023-09-30T02:11:41.446 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.446+0000 7ff35a8eb700 10 librados: watch_flush enter 2023-09-30T02:11:41.447 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.446+0000 7ff35a8eb700 10 librados: watch_flush exit 2023-09-30T02:11:41.447 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.446+0000 7ff35a8eb700 20 client.4948.objecter shutdown clearing up homeless session... 2023-09-30T02:11:41.447 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.446+0000 7ff35a8eb700 10 client.4948.objecter successfully canceled tick 2023-09-30T02:11:41.447 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.447+0000 7ff35a8eb700 1 -- 172.21.15.59:0/3256288335 >> v2:172.21.15.110:6800/115870 conn(0x7ff334068890 msgr2=0x7ff33406ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:41.447 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.447+0000 7ff35a8eb700 1 --2- 172.21.15.59:0/3256288335 >> v2:172.21.15.110:6800/115870 conn(0x7ff334068890 0x7ff33406ad40 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff33c007fd0 tx=0x7ff33c007e10 comp rx=0 tx=0).stop 2023-09-30T02:11:41.447 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.447+0000 7ff35a8eb700 10 monclient: shutdown 2023-09-30T02:11:41.447 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.447+0000 7ff35a8eb700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:11:41.447 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.447+0000 7ff35a8eb700 1 -- 172.21.15.59:0/3256288335 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff354142290 msgr2=0x7ff354150310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:41.447 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.447+0000 7ff35a8eb700 1 --2- 172.21.15.59:0/3256288335 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff354142290 0x7ff354150310 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7ff34c005c50 tx=0x7ff34c001a40 comp rx=0 tx=0).stop 2023-09-30T02:11:41.447 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.447+0000 7ff35a8eb700 1 -- 172.21.15.59:0/3256288335 shutdown_connections 2023-09-30T02:11:41.447 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.447+0000 7ff35a8eb700 1 --2- 172.21.15.59:0/3256288335 >> v2:172.21.15.110:6800/115870 conn(0x7ff334068890 0x7ff33406ad40 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:41.448 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.447+0000 7ff35a8eb700 1 --2- 172.21.15.59:0/3256288335 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7ff354142290 0x7ff354150310 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:41.448 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.447+0000 7ff35a8eb700 1 --2- 172.21.15.59:0/3256288335 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7ff3540a5b70 0x7ff3540a7b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:41.448 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.447+0000 7ff35a8eb700 1 --2- 172.21.15.59:0/3256288335 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3540a5f40 0x7ff3540a8210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:41.448 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.447+0000 7ff35a8eb700 1 -- 172.21.15.59:0/3256288335 >> 172.21.15.59:0/3256288335 conn(0x7ff3540631f0 msgr2=0x7ff354144400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:11:41.448 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.447+0000 7ff35a8eb700 1 -- 172.21.15.59:0/3256288335 shutdown_connections 2023-09-30T02:11:41.448 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.448+0000 7ff35a8eb700 1 -- 172.21.15.59:0/3256288335 wait complete. 2023-09-30T02:11:41.448 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.448+0000 7ff35a8eb700 1 librados: shutdown 2023-09-30T02:11:41.449 INFO:teuthology.orchestra.run.smithi059.stderr:instructed osd(s) 6 to deep-scrub 2023-09-30T02:11:41.463 INFO:tasks.ceph:Scrubbing osd.7 2023-09-30T02:11:41.463 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2023-09-30T02:11:41.598 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.598+0000 7fc901ddf700 10 monclient: get_monmap_and_config 2023-09-30T02:11:41.598 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.598+0000 7fc901ddf700 10 monclient: build_initial_monmap 2023-09-30T02:11:41.598 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.598+0000 7fc901ddf700 10 monclient: monmap: 2023-09-30T02:11:41.598 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:11:41.598 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:41.598 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:11:41.599116+0000 2023-09-30T02:11:41.599 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:11:41.599116+0000 2023-09-30T02:11:41.599 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:11:41.599 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:41.599 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:11:41.599 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:11:41.599 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:11:41.599 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:11:41.599 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:11:41.599 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:41.601 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.599+0000 7fc901ddf700 1 Processor -- start 2023-09-30T02:11:41.601 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.599+0000 7fc901ddf700 1 -- start start 2023-09-30T02:11:41.601 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.599+0000 7fc901ddf700 10 monclient: init 2023-09-30T02:11:41.601 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.600+0000 7fc901ddf700 10 monclient: _reopen_session rank -1 2023-09-30T02:11:41.602 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.600+0000 7fc901ddf700 10 monclient: _add_conns ranks=[2,1,0,4,3] 2023-09-30T02:11:41.602 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.600+0000 7fc901ddf700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc8fc0a5730 0x7fc8fc0a7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:41.602 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.600+0000 7fc901ddf700 10 monclient(hunting): picked mon.noname-b con 0x7fc8fc0a5730 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:11:41.602 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.600+0000 7fc901ddf700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc8fc1418e0 addr v1:172.21.15.110:6789/0 2023-09-30T02:11:41.602 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.600+0000 7fc901ddf700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc8fc13b530 addr v1:172.21.15.59:6789/0 2023-09-30T02:11:41.602 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.600+0000 7fc901ddf700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8fc0a7b10 con 0x7fc8fc13b530 2023-09-30T02:11:41.602 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.600+0000 7fc901ddf700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8fc0a5b00 con 0x7fc8fc1418e0 2023-09-30T02:11:41.602 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.600+0000 7fc901ddf700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:41.602 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.600+0000 7fc901ddf700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8fc0a84e0 con 0x7fc8fc0a5730 2023-09-30T02:11:41.602 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.600+0000 7fc901ddf700 10 monclient(hunting): _renew_subs 2023-09-30T02:11:41.602 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.600+0000 7fc901ddf700 10 monclient(hunting): authenticate will time out at 2146.902383s 2023-09-30T02:11:41.603 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.600+0000 7fc8fb7fe700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc8fc0a5730 0x7fc8fc0a7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:41.603 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.601+0000 7fc8fbfff700 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7fc8fc13b530 0x7fc8fc148f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:49472/0 (socket says 172.21.15.59:49472) 2023-09-30T02:11:41.603 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.601+0000 7fc8fbfff700 1 -- 172.21.15.59:0/4288530684 learned_addr learned my addr 172.21.15.59:0/4288530684 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:11:41.603 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.601+0000 7fc8faffd700 1 --1- 172.21.15.59:0/4288530684 >> v1:172.21.15.110:6789/0 conn(0x7fc8fc1418e0 0x7fc8fc13ae20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.59:50286/0 (socket says 172.21.15.59:50286) 2023-09-30T02:11:41.603 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.601+0000 7fc8fb7fe700 10 monclient(hunting): get_auth_request con 0x7fc8fc0a5730 auth_method 0 2023-09-30T02:11:41.603 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.601+0000 7fc8fb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:41.603 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.601+0000 7fc8fb7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:41.603 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.601+0000 7fc8fb7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:41.603 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.602+0000 7fc8fb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:41.603 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.602+0000 7fc8fb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:41.603 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.602+0000 7fc8fb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:41.604 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.604+0000 7fc8fa7fc700 1 -- 172.21.15.59:0/4288530684 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2478823802 0 0) 0x7fc8fc0a5b00 con 0x7fc8fc1418e0 2023-09-30T02:11:41.604 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.604+0000 7fc8fa7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:41.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.604+0000 7fc8fa7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:41.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.604+0000 7fc8fa7fc700 10 monclient(hunting): my global_id is 4951 2023-09-30T02:11:41.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.604+0000 7fc8fa7fc700 1 -- 172.21.15.59:0/4288530684 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8e8003570 con 0x7fc8fc1418e0 2023-09-30T02:11:41.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.604+0000 7fc8fa7fc700 1 -- 172.21.15.59:0/4288530684 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 990700082 0 0) 0x7fc8fc0a7b10 con 0x7fc8fc13b530 2023-09-30T02:11:41.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.604+0000 7fc8fa7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:41.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.604+0000 7fc8fa7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:41.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.604+0000 7fc8fa7fc700 10 monclient(hunting): my global_id is 4920 2023-09-30T02:11:41.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.604+0000 7fc8fa7fc700 1 -- 172.21.15.59:0/4288530684 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8fc0a5b00 con 0x7fc8fc13b530 2023-09-30T02:11:41.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.604+0000 7fc8fb7fe700 10 monclient(hunting): handle_auth_done global_id 4817 payload 274 2023-09-30T02:11:41.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.604+0000 7fc8fb7fe700 1 -- 172.21.15.59:0/4288530684 >> v1:172.21.15.110:6789/0 conn(0x7fc8fc1418e0 legacy=0x7fc8fc13ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:41.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.604+0000 7fc8fb7fe700 1 -- 172.21.15.59:0/4288530684 >> v1:172.21.15.59:6789/0 conn(0x7fc8fc13b530 legacy=0x7fc8fc148f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:41.606 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.604+0000 7fc8fb7fe700 10 monclient: _finish_hunting 0 2023-09-30T02:11:41.606 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.604+0000 7fc8fb7fe700 1 monclient: found mon.noname-b 2023-09-30T02:11:41.606 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.604+0000 7fc8fb7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:11:41.606 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.604+0000 7fc8fb7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.59:3301/0 2023-09-30T02:11:41.606 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.604+0000 7fc8fb7fe700 1 -- 172.21.15.59:0/4288530684 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8fc0a7750 con 0x7fc8fc0a5730 2023-09-30T02:11:41.606 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.606+0000 7fc8fb7fe700 1 --2- 172.21.15.59:0/4288530684 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc8fc0a5730 0x7fc8fc0a7fa0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc8e4009910 tx=0x7fc8e40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=960b88a94646a2f3 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:41.609 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.610+0000 7fc8fa7fc700 1 -- 172.21.15.59:0/4288530684 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e4011630 con 0x7fc8fc0a5730 2023-09-30T02:11:41.610 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.610+0000 7fc8fa7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:41.610 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.610+0000 7fc8fa7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:11:41.610 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.610+0000 7fc8fa7fc700 10 monclient: dump: 2023-09-30T02:11:41.610 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:41.610 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:41.610 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:41.610 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:41.610 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:41.610 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:41.610 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:41.611 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:41.611 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:41.611 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:41.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.610+0000 7fc8fa7fc700 10 monclient: _finish_auth 0 2023-09-30T02:11:41.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.610+0000 7fc8fa7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:11:41.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.610+0000 7fc8fa7fc700 1 -- 172.21.15.59:0/4288530684 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8e4011790 con 0x7fc8fc0a5730 2023-09-30T02:11:41.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.610+0000 7fc901ddf700 5 monclient: authenticate success, global_id 4817 2023-09-30T02:11:41.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.610+0000 7fc901ddf700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:11:41.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.610+0000 7fc8fa7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:11:41.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.610+0000 7fc901ddf700 10 monclient: get_monmap_and_config success 2023-09-30T02:11:41.611 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.610+0000 7fc901ddf700 4 set_mon_vals no callback set 2023-09-30T02:11:41.612 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.610+0000 7fc8fa7fc700 1 -- 172.21.15.59:0/4288530684 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e4011b30 con 0x7fc8fc0a5730 2023-09-30T02:11:41.612 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.610+0000 7fc8fa7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:41.612 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.610+0000 7fc8fa7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:11:41.612 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.610+0000 7fc8fa7fc700 10 monclient: dump: 2023-09-30T02:11:41.612 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:41.612 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:41.612 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:41.612 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:41.612 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:41.612 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:41.613 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:41.613 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:41.613 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:41.613 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:41.616 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.611+0000 7fc901ddf700 10 monclient: shutdown 2023-09-30T02:11:41.616 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.611+0000 7fc901ddf700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:11:41.616 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.611+0000 7fc901ddf700 1 -- 172.21.15.59:0/4288530684 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc8fc0a5730 msgr2=0x7fc8fc0a7fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:41.616 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.611+0000 7fc901ddf700 1 --2- 172.21.15.59:0/4288530684 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc8fc0a5730 0x7fc8fc0a7fa0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc8e4009910 tx=0x7fc8e40091a0 comp rx=0 tx=0).stop 2023-09-30T02:11:41.616 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.611+0000 7fc901ddf700 1 -- 172.21.15.59:0/4288530684 shutdown_connections 2023-09-30T02:11:41.616 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.611+0000 7fc901ddf700 1 --2- 172.21.15.59:0/4288530684 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc8fc0a5730 0x7fc8fc0a7fa0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:41.616 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.611+0000 7fc901ddf700 1 -- 172.21.15.59:0/4288530684 >> 172.21.15.59:0/4288530684 conn(0x7fc8fc0631f0 msgr2=0x7fc8fc0635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:11:41.616 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.611+0000 7fc901ddf700 1 -- 172.21.15.59:0/4288530684 shutdown_connections 2023-09-30T02:11:41.616 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.611+0000 7fc901ddf700 1 -- 172.21.15.59:0/4288530684 wait complete. 2023-09-30T02:11:41.617 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.612+0000 7fc901ddf700 10 monclient: build_initial_monmap 2023-09-30T02:11:41.617 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.612+0000 7fc901ddf700 10 monclient: monmap: 2023-09-30T02:11:41.617 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:11:41.617 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:11:41.617 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:11:41.617 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:11:41.617 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:11:41.617 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:11:41.617 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:11:41.617 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:11:41.617 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:11:41.618 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:41.618 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.612+0000 7fc901ddf700 1 librados: starting msgr at 2023-09-30T02:11:41.618 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.612+0000 7fc901ddf700 1 librados: starting objecter 2023-09-30T02:11:41.618 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.612+0000 7fc901ddf700 1 Processor -- start 2023-09-30T02:11:41.618 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.612+0000 7fc901ddf700 1 -- start start 2023-09-30T02:11:41.618 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.612+0000 7fc901ddf700 1 librados: setting wanted keys 2023-09-30T02:11:41.618 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.612+0000 7fc901ddf700 1 librados: calling monclient init 2023-09-30T02:11:41.618 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.612+0000 7fc901ddf700 10 monclient: init 2023-09-30T02:11:41.618 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.613+0000 7fc901ddf700 10 monclient: _reopen_session rank -1 2023-09-30T02:11:41.618 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.613+0000 7fc901ddf700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T02:11:41.618 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.613+0000 7fc901ddf700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc8fc0a5730 0x7fc8fc13a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:41.618 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.613+0000 7fc901ddf700 10 monclient(hunting): picked mon.noname-b con 0x7fc8fc0a5730 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:11:41.619 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.613+0000 7fc901ddf700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc8fc13b530 0x7fc8fc14e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:41.619 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.613+0000 7fc901ddf700 10 monclient(hunting): picked mon.noname-c con 0x7fc8fc13b530 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:11:41.619 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.613+0000 7fc901ddf700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8fc1418e0 0x7fc8fc14e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:41.619 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.613+0000 7fc901ddf700 10 monclient(hunting): picked mon.noname-a con 0x7fc8fc1418e0 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:11:41.619 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.613+0000 7fc901ddf700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:41.619 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.613+0000 7fc901ddf700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8fc0ae640 con 0x7fc8fc1418e0 2023-09-30T02:11:41.619 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.613+0000 7fc901ddf700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:41.619 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.613+0000 7fc901ddf700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8fc14eb60 con 0x7fc8fc0a5730 2023-09-30T02:11:41.619 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.613+0000 7fc901ddf700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:41.619 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.613+0000 7fc901ddf700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8fc0ae120 con 0x7fc8fc13b530 2023-09-30T02:11:41.619 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.613+0000 7fc901ddf700 10 monclient(hunting): _renew_subs 2023-09-30T02:11:41.619 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.613+0000 7fc901ddf700 10 monclient(hunting): authenticate will time out at 2146.915461s 2023-09-30T02:11:41.619 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.614+0000 7fc8fbfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8fc1418e0 0x7fc8fc14e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:41.620 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.614+0000 7fc8fb7fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc8fc0a5730 0x7fc8fc13a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:41.620 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.614+0000 7fc8faffd700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc8fc13b530 0x7fc8fc14e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:41.620 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.614+0000 7fc8fb7fe700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc8fc0a5730 0x7fc8fc13a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:41496/0 (socket says 172.21.15.59:41496) 2023-09-30T02:11:41.620 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.614+0000 7fc8fb7fe700 1 -- 172.21.15.59:0/343258870 learned_addr learned my addr 172.21.15.59:0/343258870 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:11:41.620 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.614+0000 7fc8fb7fe700 10 monclient(hunting): get_auth_request con 0x7fc8fc0a5730 auth_method 0 2023-09-30T02:11:41.620 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.614+0000 7fc8fb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:41.620 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.614+0000 7fc8fb7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:41.620 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.614+0000 7fc8fb7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:41.620 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.614+0000 7fc8faffd700 10 monclient(hunting): get_auth_request con 0x7fc8fc13b530 auth_method 0 2023-09-30T02:11:41.620 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.614+0000 7fc8faffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:41.620 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.614+0000 7fc8faffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:41.620 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.614+0000 7fc8faffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:41.621 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.614+0000 7fc8fbfff700 10 monclient(hunting): get_auth_request con 0x7fc8fc1418e0 auth_method 0 2023-09-30T02:11:41.621 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.614+0000 7fc8fbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:41.621 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.614+0000 7fc8fbfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:41.621 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.614+0000 7fc8fbfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:41.621 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.614+0000 7fc8fb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:41.621 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.614+0000 7fc8fb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:41.621 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.614+0000 7fc8fb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:41.621 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.615+0000 7fc8faffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:41.621 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.615+0000 7fc8faffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:41.621 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.615+0000 7fc8faffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:41.621 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.615+0000 7fc8fbfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:41.622 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.615+0000 7fc8fbfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:41.622 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.615+0000 7fc8fbfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:41.622 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.618+0000 7fc8fb7fe700 10 monclient(hunting): handle_auth_done global_id 4954 payload 824 2023-09-30T02:11:41.623 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.618+0000 7fc8fb7fe700 1 -- 172.21.15.59:0/343258870 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc8fc13b530 msgr2=0x7fc8fc14e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:41.623 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.618+0000 7fc8fb7fe700 1 --2- 172.21.15.59:0/343258870 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fc8fc13b530 0x7fc8fc14e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:41.623 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.618+0000 7fc8fb7fe700 1 -- 172.21.15.59:0/343258870 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8fc1418e0 msgr2=0x7fc8fc14e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:41.623 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.618+0000 7fc8fb7fe700 1 --2- 172.21.15.59:0/343258870 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8fc1418e0 0x7fc8fc14e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:41.623 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.618+0000 7fc8fb7fe700 10 monclient: _finish_hunting 0 2023-09-30T02:11:41.623 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.618+0000 7fc8fb7fe700 1 monclient: found mon.noname-b 2023-09-30T02:11:41.623 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.618+0000 7fc8fb7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:11:41.623 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.618+0000 7fc8fb7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:11:41.623 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.618+0000 7fc8fb7fe700 1 -- 172.21.15.59:0/343258870 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8e4009420 con 0x7fc8fc0a5730 2023-09-30T02:11:41.623 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.620+0000 7fc8fb7fe700 1 --2- 172.21.15.59:0/343258870 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc8fc0a5730 0x7fc8fc13a5a0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fc8e4009f00 tx=0x7fc8e4017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:41.624 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.620+0000 7fc8f37fe700 10 client.?.objecter ms_handle_connect 0x7fc8fc0a5730 2023-09-30T02:11:41.624 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.620+0000 7fc8f37fe700 10 client.?.objecter resend_mon_ops 2023-09-30T02:11:41.625 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.624+0000 7fc8f37fe700 1 -- 172.21.15.59:0/343258870 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e4011900 con 0x7fc8fc0a5730 2023-09-30T02:11:41.625 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.624+0000 7fc8f37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:41.625 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.624+0000 7fc8f37fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:11:41.625 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.624+0000 7fc8f37fe700 10 monclient: dump: 2023-09-30T02:11:41.625 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:41.625 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:41.625 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:41.625 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:41.626 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:41.626 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:41.626 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:41.626 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:41.626 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:41.626 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:41.626 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.625+0000 7fc8f37fe700 10 monclient: _finish_auth 0 2023-09-30T02:11:41.626 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.625+0000 7fc8f37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:11:41.626 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.625+0000 7fc8f37fe700 1 -- 172.21.15.59:0/343258870 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8e4011a60 con 0x7fc8fc0a5730 2023-09-30T02:11:41.626 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.625+0000 7fc8f37fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:11:41.626 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.625+0000 7fc8f37fe700 1 -- 172.21.15.59:0/343258870 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8e4025650 con 0x7fc8fc0a5730 2023-09-30T02:11:41.627 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.625+0000 7fc901ddf700 5 monclient: authenticate success, global_id 4954 2023-09-30T02:11:41.627 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.625+0000 7fc901ddf700 10 monclient: _renew_subs 2023-09-30T02:11:41.627 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.625+0000 7fc901ddf700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:11:41.627 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.625+0000 7fc901ddf700 1 -- 172.21.15.59:0/343258870 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8fc13c610 con 0x7fc8fc0a5730 2023-09-30T02:11:41.627 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.625+0000 7fc8f37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:41.627 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.625+0000 7fc8f9ffb700 4 set_mon_vals no callback set 2023-09-30T02:11:41.627 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.625+0000 7fc8f37fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:11:41.627 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.625+0000 7fc8f37fe700 10 monclient: dump: 2023-09-30T02:11:41.627 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:41.627 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:41.627 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:41.627 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:41.628 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:41.628 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:41.628 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:41.628 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:41.628 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:41.628 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:41.628 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.625+0000 7fc901ddf700 10 client.4954.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:11:41.628 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.625+0000 7fc901ddf700 10 monclient: _renew_subs 2023-09-30T02:11:41.628 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.625+0000 7fc901ddf700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:11:41.628 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.625+0000 7fc901ddf700 1 -- 172.21.15.59:0/343258870 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8fc0ade50 con 0x7fc8fc0a5730 2023-09-30T02:11:41.628 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.625+0000 7fc901ddf700 1 librados: init done 2023-09-30T02:11:41.628 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.626+0000 7fc901ddf700 10 client.4954.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:11:41.629 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.626+0000 7fc901ddf700 15 client.4954.objecter _session_command_op_assign -1 1 2023-09-30T02:11:41.629 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.626+0000 7fc901ddf700 20 client.4954.objecter _get_session osd=-1 returning homeless 2023-09-30T02:11:41.629 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.626+0000 7fc901ddf700 10 client.4954.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:11:41.629 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.626+0000 7fc901ddf700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:11:41.629 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.626+0000 7fc901ddf700 1 -- 172.21.15.59:0/343258870 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc8fc064a00 con 0x7fc8fc0a5730 2023-09-30T02:11:41.649 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.649+0000 7fc8f37fe700 1 -- 172.21.15.59:0/343258870 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fc8e40193b0 con 0x7fc8fc0a5730 2023-09-30T02:11:41.650 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.650+0000 7fc8f37fe700 1 --2- 172.21.15.59:0/343258870 >> v2:172.21.15.110:6800/115870 conn(0x7fc8dc0689e0 0x7fc8dc06ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:41.650 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.650+0000 7fc8f37fe700 1 -- 172.21.15.59:0/343258870 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fc8e4085d70 con 0x7fc8fc0a5730 2023-09-30T02:11:41.650 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.650+0000 7fc8f37fe700 10 client.4954.objecter ms_dispatch 0x7fc8fc001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:11:41.651 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.650+0000 7fc8f37fe700 3 client.4954.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:11:41.651 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.650+0000 7fc8f37fe700 3 client.4954.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:11:41.651 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.650+0000 7fc8f37fe700 10 client.4954.objecter checking command 1 2023-09-30T02:11:41.651 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.650+0000 7fc8f37fe700 1 --2- 172.21.15.59:0/343258870 >> v2:172.21.15.110:6805/116561 conn(0x7fc8dc06e950 0x7fc8dc070d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:41.651 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.650+0000 7fc8f37fe700 20 client.4954.objecter _get_session s=0x7fc8dc06e2a0 osd=7 3 2023-09-30T02:11:41.651 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.650+0000 7fc8f37fe700 20 client.4954.objecter put_session s=0x7fc8dc06e2a0 osd=7 3 2023-09-30T02:11:41.651 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.650+0000 7fc8f37fe700 15 client.4954.objecter _session_command_op_remove -1 1 2023-09-30T02:11:41.651 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.650+0000 7fc8f37fe700 20 client.4954.objecter _get_session s=0x7fc8dc06e2a0 osd=7 3 2023-09-30T02:11:41.651 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.650+0000 7fc8f37fe700 20 client.4954.objecter get_session s=0x7fc8dc06e2a0 osd=7 3 2023-09-30T02:11:41.651 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.650+0000 7fc8f37fe700 15 client.4954.objecter _session_command_op_assign 7 1 2023-09-30T02:11:41.651 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.650+0000 7fc8f37fe700 20 client.4954.objecter put_session s=0x7fc8dc06e2a0 osd=7 4 2023-09-30T02:11:41.651 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.650+0000 7fc8f37fe700 10 client.4954.objecter _send_command 1 2023-09-30T02:11:41.652 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.650+0000 7fc8f37fe700 1 -- 172.21.15.59:0/343258870 --> v2:172.21.15.110:6805/116561 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc8dc0713d0 con 0x7fc8dc06e950 2023-09-30T02:11:41.652 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.650+0000 7fc8f37fe700 20 client.4954.objecter dump_active .. 0 homeless 2023-09-30T02:11:41.652 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.650+0000 7fc8faffd700 1 --2- 172.21.15.59:0/343258870 >> v2:172.21.15.110:6800/115870 conn(0x7fc8dc0689e0 0x7fc8dc06ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:41.652 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.650+0000 7fc8f37fe700 1 -- 172.21.15.59:0/343258870 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc8e40860a0 con 0x7fc8fc0a5730 2023-09-30T02:11:41.652 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.650+0000 7fc8f37fe700 10 monclient: handle_get_version_reply finishing 1 version 25 2023-09-30T02:11:41.652 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.650+0000 7fc8faffd700 10 monclient: get_auth_request con 0x7fc8dc0689e0 auth_method 0 2023-09-30T02:11:41.652 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.650+0000 7fc8faffd700 1 --2- 172.21.15.59:0/343258870 >> v2:172.21.15.110:6800/115870 conn(0x7fc8dc0689e0 0x7fc8dc06ae90 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fc8fc13d290 tx=0x7fc8ec007d60 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:41.652 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.650+0000 7fc8f37fe700 10 client.4954.objecter ms_handle_connect 0x7fc8dc0689e0 2023-09-30T02:11:41.652 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.651+0000 7fc8fbfff700 1 --2- 172.21.15.59:0/343258870 >> v2:172.21.15.110:6805/116561 conn(0x7fc8dc06e950 0x7fc8dc070d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:41.652 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.651+0000 7fc8fbfff700 10 monclient: get_auth_request con 0x7fc8dc06e950 auth_method 0 2023-09-30T02:11:41.653 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.654+0000 7fc8fbfff700 1 --2- 172.21.15.59:0/343258870 >> v2:172.21.15.110:6805/116561 conn(0x7fc8dc06e950 0x7fc8dc070d60 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:41.653 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.654+0000 7fc8f37fe700 10 client.4954.objecter ms_handle_connect 0x7fc8dc06e950 2023-09-30T02:11:41.675 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.676+0000 7fc8f37fe700 1 -- 172.21.15.59:0/343258870 <== osd.7 v2:172.21.15.110:6805/116561 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7fc8dc0713d0 con 0x7fc8dc06e950 2023-09-30T02:11:41.675 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.676+0000 7fc8f37fe700 10 client.4954.objecter ms_dispatch 0x7fc8fc001550 command_reply(tid 1: 0 ) v1 2023-09-30T02:11:41.676 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.676+0000 7fc8f37fe700 10 client.4954.objecter _finish_command 1 = system:0 2023-09-30T02:11:41.676 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.676+0000 7fc8f37fe700 20 client.4954.objecter put_session s=0x7fc8dc06e2a0 osd=7 4 2023-09-30T02:11:41.676 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.676+0000 7fc8f37fe700 15 client.4954.objecter _session_command_op_remove 7 1 2023-09-30T02:11:41.690 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.690+0000 7fc901ddf700 10 client.4954.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2023-09-30T02:11:41.690 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.690+0000 7fc901ddf700 15 client.4954.objecter _session_command_op_assign -1 2 2023-09-30T02:11:41.690 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.690+0000 7fc901ddf700 20 client.4954.objecter _get_session s=0x7fc8dc06e2a0 osd=7 3 2023-09-30T02:11:41.690 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.690+0000 7fc901ddf700 20 client.4954.objecter put_session s=0x7fc8dc06e2a0 osd=7 3 2023-09-30T02:11:41.690 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.690+0000 7fc901ddf700 20 client.4954.objecter _get_session s=0x7fc8dc06e2a0 osd=7 3 2023-09-30T02:11:41.690 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.690+0000 7fc901ddf700 15 client.4954.objecter _session_command_op_remove -1 2 2023-09-30T02:11:41.690 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.690+0000 7fc901ddf700 20 client.4954.objecter get_session s=0x7fc8dc06e2a0 osd=7 3 2023-09-30T02:11:41.690 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.690+0000 7fc901ddf700 15 client.4954.objecter _session_command_op_assign 7 2 2023-09-30T02:11:41.691 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.690+0000 7fc901ddf700 20 client.4954.objecter put_session s=0x7fc8dc06e2a0 osd=7 4 2023-09-30T02:11:41.691 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.690+0000 7fc901ddf700 10 client.4954.objecter _send_command 2 2023-09-30T02:11:41.691 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.690+0000 7fc901ddf700 1 -- 172.21.15.59:0/343258870 --> v2:172.21.15.110:6805/116561 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fc8fc0a7050 con 0x7fc8dc06e950 2023-09-30T02:11:41.712 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.713+0000 7fc8f37fe700 1 -- 172.21.15.59:0/343258870 <== osd.7 v2:172.21.15.110:6805/116561 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fc8fc0a7050 con 0x7fc8dc06e950 2023-09-30T02:11:41.712 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.713+0000 7fc8f37fe700 10 client.4954.objecter ms_dispatch 0x7fc8fc001550 command_reply(tid 2: 0 ) v1 2023-09-30T02:11:41.713 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.713+0000 7fc8f37fe700 10 client.4954.objecter _finish_command 2 = system:0 2023-09-30T02:11:41.713 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.713+0000 7fc8f37fe700 20 client.4954.objecter put_session s=0x7fc8dc06e2a0 osd=7 4 2023-09-30T02:11:41.713 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.713+0000 7fc8f37fe700 15 client.4954.objecter _session_command_op_remove 7 2 2023-09-30T02:11:41.713 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2023-09-30T02:11:41.713 INFO:teuthology.orchestra.run.smithi059.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T02:11:41.713 INFO:teuthology.orchestra.run.smithi059.stdout:} 2023-09-30T02:11:41.715 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.714+0000 7fc901ddf700 10 librados: watch_flush enter 2023-09-30T02:11:41.715 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.714+0000 7fc901ddf700 10 librados: watch_flush exit 2023-09-30T02:11:41.715 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.714+0000 7fc901ddf700 10 client.4954.objecter close_session for osd.7 2023-09-30T02:11:41.715 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.714+0000 7fc901ddf700 1 -- 172.21.15.59:0/343258870 >> v2:172.21.15.110:6805/116561 conn(0x7fc8dc06e950 msgr2=0x7fc8dc070d60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:41.715 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.714+0000 7fc901ddf700 1 --2- 172.21.15.59:0/343258870 >> v2:172.21.15.110:6805/116561 conn(0x7fc8dc06e950 0x7fc8dc070d60 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:41.715 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.714+0000 7fc901ddf700 20 client.4954.objecter put_session s=0x7fc8dc06e2a0 osd=7 1 2023-09-30T02:11:41.715 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.714+0000 7fc901ddf700 20 client.4954.objecter shutdown clearing up homeless session... 2023-09-30T02:11:41.715 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.714+0000 7fc901ddf700 10 client.4954.objecter successfully canceled tick 2023-09-30T02:11:41.715 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.714+0000 7fc901ddf700 1 -- 172.21.15.59:0/343258870 >> v2:172.21.15.110:6800/115870 conn(0x7fc8dc0689e0 msgr2=0x7fc8dc06ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:41.715 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.714+0000 7fc901ddf700 1 --2- 172.21.15.59:0/343258870 >> v2:172.21.15.110:6800/115870 conn(0x7fc8dc0689e0 0x7fc8dc06ae90 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fc8fc13d290 tx=0x7fc8ec007d60 comp rx=0 tx=0).stop 2023-09-30T02:11:41.716 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.714+0000 7fc901ddf700 10 monclient: shutdown 2023-09-30T02:11:41.716 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.714+0000 7fc901ddf700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:11:41.716 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.714+0000 7fc901ddf700 1 -- 172.21.15.59:0/343258870 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc8fc0a5730 msgr2=0x7fc8fc13a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:41.716 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.714+0000 7fc901ddf700 1 --2- 172.21.15.59:0/343258870 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fc8fc0a5730 0x7fc8fc13a5a0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fc8e4009f00 tx=0x7fc8e4017000 comp rx=0 tx=0).stop 2023-09-30T02:11:41.716 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.714+0000 7fc8fbfff700 1 -- 172.21.15.59:0/343258870 reap_dead start 2023-09-30T02:11:41.716 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.714+0000 7fc901ddf700 1 -- 172.21.15.59:0/343258870 shutdown_connections 2023-09-30T02:11:41.716 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.714+0000 7fc901ddf700 1 -- 172.21.15.59:0/343258870 >> 172.21.15.59:0/343258870 conn(0x7fc8fc0631f0 msgr2=0x7fc8fc141d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:11:41.716 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.714+0000 7fc901ddf700 1 -- 172.21.15.59:0/343258870 shutdown_connections 2023-09-30T02:11:41.716 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.714+0000 7fc901ddf700 1 -- 172.21.15.59:0/343258870 wait complete. 2023-09-30T02:11:41.716 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.714+0000 7fc901ddf700 1 librados: shutdown 2023-09-30T02:11:41.726 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2023-09-30T02:11:41.863 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.864+0000 7f39c7737700 10 monclient: get_monmap_and_config 2023-09-30T02:11:41.864 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.864+0000 7f39c7737700 10 monclient: build_initial_monmap 2023-09-30T02:11:41.864 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.864+0000 7f39c7737700 10 monclient: monmap: 2023-09-30T02:11:41.864 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:11:41.864 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:41.864 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:11:41.864683+0000 2023-09-30T02:11:41.865 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:11:41.864683+0000 2023-09-30T02:11:41.865 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:11:41.865 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:41.865 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:11:41.865 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:11:41.865 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:11:41.865 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:11:41.865 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:11:41.865 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:41.867 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.864+0000 7f39c7737700 1 Processor -- start 2023-09-30T02:11:41.867 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.865+0000 7f39c7737700 1 -- start start 2023-09-30T02:11:41.867 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.865+0000 7f39c7737700 10 monclient: init 2023-09-30T02:11:41.868 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.865+0000 7f39c7737700 10 monclient: _reopen_session rank -1 2023-09-30T02:11:41.868 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.865+0000 7f39c7737700 10 monclient: _add_conns ranks=[2,0,1,4,3] 2023-09-30T02:11:41.868 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.865+0000 7f39c7737700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f39c00a5b70 0x7f39c00b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:41.868 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.865+0000 7f39c7737700 10 monclient(hunting): picked mon.noname-b con 0x7f39c00a5b70 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:11:41.868 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.865+0000 7f39c7737700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f39c00a5f40 addr v1:172.21.15.59:6789/0 2023-09-30T02:11:41.868 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.865+0000 7f39c7737700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f39c0142290 addr v1:172.21.15.110:6789/0 2023-09-30T02:11:41.868 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.865+0000 7f39c7737700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39c00a63b0 con 0x7f39c00a5f40 2023-09-30T02:11:41.868 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.865+0000 7f39c7737700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39c00a75a0 con 0x7f39c0142290 2023-09-30T02:11:41.868 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.865+0000 7f39c7737700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:41.868 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.865+0000 7f39c7737700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39c01426d0 con 0x7f39c00a5b70 2023-09-30T02:11:41.868 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.865+0000 7f39c7737700 10 monclient(hunting): _renew_subs 2023-09-30T02:11:41.869 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.865+0000 7f39c7737700 10 monclient(hunting): authenticate will time out at 2147.167877s 2023-09-30T02:11:41.869 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.866+0000 7f39c54d3700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f39c00a5b70 0x7f39c00b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:41.869 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.866+0000 7f39c5cd4700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f39c0142290 0x7f39c013af30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.59:50296/0 (socket says 172.21.15.59:50296) 2023-09-30T02:11:41.869 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.866+0000 7f39c5cd4700 1 -- 172.21.15.59:0/2718642980 learned_addr learned my addr 172.21.15.59:0/2718642980 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:11:41.869 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.866+0000 7f39c54d3700 10 monclient(hunting): get_auth_request con 0x7f39c00a5b70 auth_method 0 2023-09-30T02:11:41.869 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.866+0000 7f39c54d3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:41.869 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.866+0000 7f39c54d3700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:41.869 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.866+0000 7f39c54d3700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:41.869 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.867+0000 7f39c54d3700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:41.869 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.867+0000 7f39c54d3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:41.870 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.867+0000 7f39c54d3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:41.870 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.869+0000 7f39b7fff700 1 -- 172.21.15.59:0/2718642980 <== mon.1 v1:172.21.15.110:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 969838435 0 0) 0x7f39c00a75a0 con 0x7f39c0142290 2023-09-30T02:11:41.871 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.869+0000 7f39b7fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:41.871 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.869+0000 7f39b7fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:41.871 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.869+0000 7f39b7fff700 10 monclient(hunting): my global_id is 4957 2023-09-30T02:11:41.871 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.869+0000 7f39b7fff700 1 -- 172.21.15.59:0/2718642980 --> v1:172.21.15.110:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39ac003570 con 0x7f39c0142290 2023-09-30T02:11:41.871 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.869+0000 7f39c54d3700 10 monclient(hunting): handle_auth_done global_id 4823 payload 274 2023-09-30T02:11:41.871 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.869+0000 7f39c54d3700 1 -- 172.21.15.59:0/2718642980 >> v1:172.21.15.110:6789/0 conn(0x7f39c0142290 legacy=0x7f39c013af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:41.871 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.869+0000 7f39c54d3700 1 -- 172.21.15.59:0/2718642980 >> v1:172.21.15.59:6789/0 conn(0x7f39c00a5f40 legacy=0x7f39c0141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:41.871 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.869+0000 7f39c54d3700 10 monclient: _finish_hunting 0 2023-09-30T02:11:41.871 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.869+0000 7f39c54d3700 1 monclient: found mon.noname-b 2023-09-30T02:11:41.871 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.869+0000 7f39c54d3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:11:41.871 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.869+0000 7f39c54d3700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.59:3301/0 2023-09-30T02:11:41.872 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.869+0000 7f39c54d3700 1 -- 172.21.15.59:0/2718642980 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39c00af3f0 con 0x7f39c00a5b70 2023-09-30T02:11:41.872 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.871+0000 7f39c54d3700 1 --2- 172.21.15.59:0/2718642980 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f39c00a5b70 0x7f39c00b1500 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f39bc009910 tx=0x7f39bc0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5fe2a142b0475e4 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:41.876 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.875+0000 7f39b7fff700 1 -- 172.21.15.59:0/2718642980 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39bc00b680 con 0x7f39c00a5b70 2023-09-30T02:11:41.877 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.875+0000 7f39b7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:41.877 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.875+0000 7f39b7fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:11:41.877 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.875+0000 7f39b7fff700 10 monclient: dump: 2023-09-30T02:11:41.877 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:41.877 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:41.877 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:41.877 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:41.877 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:41.877 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:41.877 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:41.878 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:41.878 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:41.878 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:41.878 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.875+0000 7f39b7fff700 10 monclient: _finish_auth 0 2023-09-30T02:11:41.878 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.875+0000 7f39b7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:11:41.878 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.875+0000 7f39b7fff700 1 -- 172.21.15.59:0/2718642980 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39bc00b7e0 con 0x7f39c00a5b70 2023-09-30T02:11:41.878 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.875+0000 7f39b7fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:11:41.878 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.875+0000 7f39b7fff700 1 -- 172.21.15.59:0/2718642980 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39bc00bba0 con 0x7f39c00a5b70 2023-09-30T02:11:41.878 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.875+0000 7f39b7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:41.878 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.875+0000 7f39b7fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:11:41.878 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.875+0000 7f39b7fff700 10 monclient: dump: 2023-09-30T02:11:41.878 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:41.879 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:41.879 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:41.879 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:41.879 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:41.879 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:41.879 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:41.879 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:41.879 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:41.879 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:41.879 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.875+0000 7f39c7737700 5 monclient: authenticate success, global_id 4823 2023-09-30T02:11:41.879 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.875+0000 7f39c7737700 10 monclient: get_monmap_and_config success 2023-09-30T02:11:41.879 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.875+0000 7f39c7737700 4 set_mon_vals no callback set 2023-09-30T02:11:41.880 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.876+0000 7f39c7737700 10 monclient: shutdown 2023-09-30T02:11:41.880 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.876+0000 7f39c7737700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:11:41.880 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.876+0000 7f39c7737700 1 -- 172.21.15.59:0/2718642980 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f39c00a5b70 msgr2=0x7f39c00b1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:41.880 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.876+0000 7f39c7737700 1 --2- 172.21.15.59:0/2718642980 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f39c00a5b70 0x7f39c00b1500 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f39bc009910 tx=0x7f39bc0091a0 comp rx=0 tx=0).stop 2023-09-30T02:11:41.880 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.876+0000 7f39c7737700 1 -- 172.21.15.59:0/2718642980 shutdown_connections 2023-09-30T02:11:41.880 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.876+0000 7f39c7737700 1 --2- 172.21.15.59:0/2718642980 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f39c00a5b70 0x7f39c00b1500 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:41.880 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.876+0000 7f39c7737700 1 -- 172.21.15.59:0/2718642980 >> 172.21.15.59:0/2718642980 conn(0x7f39c00631f0 msgr2=0x7f39c00635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:11:41.880 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.876+0000 7f39c7737700 1 -- 172.21.15.59:0/2718642980 shutdown_connections 2023-09-30T02:11:41.880 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.876+0000 7f39c7737700 1 -- 172.21.15.59:0/2718642980 wait complete. 2023-09-30T02:11:41.880 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.877+0000 7f39c7737700 10 monclient: build_initial_monmap 2023-09-30T02:11:41.880 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.877+0000 7f39c7737700 10 monclient: monmap: 2023-09-30T02:11:41.881 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:11:41.881 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:11:41.881 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:11:41.881 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:11:41.881 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:11:41.881 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:11:41.881 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:11:41.881 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:11:41.881 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:11:41.881 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:41.881 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.877+0000 7f39c7737700 1 librados: starting msgr at 2023-09-30T02:11:41.881 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.877+0000 7f39c7737700 1 librados: starting objecter 2023-09-30T02:11:41.882 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.877+0000 7f39c7737700 1 Processor -- start 2023-09-30T02:11:41.882 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.877+0000 7f39c7737700 1 -- start start 2023-09-30T02:11:41.882 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.877+0000 7f39c7737700 1 librados: setting wanted keys 2023-09-30T02:11:41.882 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.877+0000 7f39c7737700 1 librados: calling monclient init 2023-09-30T02:11:41.882 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.877+0000 7f39c7737700 10 monclient: init 2023-09-30T02:11:41.884 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.878+0000 7f39c7737700 10 monclient: _reopen_session rank -1 2023-09-30T02:11:41.884 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.878+0000 7f39c7737700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T02:11:41.885 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.878+0000 7f39c7737700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39c00a5b70 0x7f39c0144e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:41.885 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.878+0000 7f39c7737700 10 monclient(hunting): picked mon.noname-a con 0x7f39c00a5b70 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:11:41.885 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.878+0000 7f39c7737700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f39c00a5f40 0x7f39c013daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:41.885 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.878+0000 7f39c7737700 10 monclient(hunting): picked mon.noname-c con 0x7f39c00a5f40 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:11:41.885 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.878+0000 7f39c7737700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f39c0142290 0x7f39c013e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:41.885 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.878+0000 7f39c7737700 10 monclient(hunting): picked mon.noname-b con 0x7f39c0142290 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:11:41.885 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.878+0000 7f39c7737700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:41.885 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.878+0000 7f39c7737700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39c013e030 con 0x7f39c00a5b70 2023-09-30T02:11:41.885 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.878+0000 7f39c7737700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:41.885 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.878+0000 7f39c7737700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39c00a8650 con 0x7f39c0142290 2023-09-30T02:11:41.885 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.878+0000 7f39c7737700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:41.886 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.878+0000 7f39c7737700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39c00a8930 con 0x7f39c00a5f40 2023-09-30T02:11:41.886 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.878+0000 7f39c7737700 10 monclient(hunting): _renew_subs 2023-09-30T02:11:41.886 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.878+0000 7f39c7737700 10 monclient(hunting): authenticate will time out at 2147.180658s 2023-09-30T02:11:41.886 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.879+0000 7f39c4cd2700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f39c00a5f40 0x7f39c013daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:41.886 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.879+0000 7f39c5cd4700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f39c0142290 0x7f39c013e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:41.886 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.879+0000 7f39c4cd2700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f39c00a5f40 0x7f39c013daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:46272/0 (socket says 172.21.15.59:46272) 2023-09-30T02:11:41.886 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.879+0000 7f39c4cd2700 1 -- 172.21.15.59:0/1222668338 learned_addr learned my addr 172.21.15.59:0/1222668338 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:11:41.886 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.879+0000 7f39c4cd2700 10 monclient(hunting): get_auth_request con 0x7f39c00a5f40 auth_method 0 2023-09-30T02:11:41.886 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.879+0000 7f39c4cd2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:41.886 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.879+0000 7f39c4cd2700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:41.886 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.879+0000 7f39c4cd2700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:41.887 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.879+0000 7f39c5cd4700 10 monclient(hunting): get_auth_request con 0x7f39c0142290 auth_method 0 2023-09-30T02:11:41.887 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.879+0000 7f39c5cd4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:41.887 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.879+0000 7f39c5cd4700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:41.887 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.879+0000 7f39c5cd4700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:41.887 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.880+0000 7f39c4cd2700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:41.887 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.880+0000 7f39c4cd2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:41.887 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.880+0000 7f39c4cd2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:41.887 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.880+0000 7f39c5cd4700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:41.887 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.880+0000 7f39c5cd4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:41.887 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.880+0000 7f39c5cd4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:41.887 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.880+0000 7f39c54d3700 1 --2- 172.21.15.59:0/1222668338 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39c00a5b70 0x7f39c0144e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:41.887 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.881+0000 7f39c54d3700 10 monclient(hunting): get_auth_request con 0x7f39c00a5b70 auth_method 0 2023-09-30T02:11:41.888 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.881+0000 7f39c54d3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:41.888 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.881+0000 7f39c54d3700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:41.888 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.881+0000 7f39c54d3700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:41.888 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.883+0000 7f39c5cd4700 10 monclient(hunting): handle_auth_done global_id 4960 payload 824 2023-09-30T02:11:41.888 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.883+0000 7f39c5cd4700 1 -- 172.21.15.59:0/1222668338 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f39c00a5f40 msgr2=0x7f39c013daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:41.888 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.883+0000 7f39c5cd4700 1 --2- 172.21.15.59:0/1222668338 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f39c00a5f40 0x7f39c013daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:41.888 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.883+0000 7f39c5cd4700 1 -- 172.21.15.59:0/1222668338 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39c00a5b70 msgr2=0x7f39c0144e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:41.888 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.883+0000 7f39c5cd4700 1 --2- 172.21.15.59:0/1222668338 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39c00a5b70 0x7f39c0144e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:41.888 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.883+0000 7f39c5cd4700 10 monclient: _finish_hunting 0 2023-09-30T02:11:41.888 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.883+0000 7f39c5cd4700 1 monclient: found mon.noname-b 2023-09-30T02:11:41.888 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.883+0000 7f39c5cd4700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:11:41.888 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.883+0000 7f39c5cd4700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:11:41.889 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.883+0000 7f39c5cd4700 1 -- 172.21.15.59:0/1222668338 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39bc009420 con 0x7f39c0142290 2023-09-30T02:11:41.889 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.883+0000 7f39c54d3700 1 --2- 172.21.15.59:0/1222668338 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39c00a5b70 0x7f39c0144e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T02:11:41.889 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.885+0000 7f39c5cd4700 1 --2- 172.21.15.59:0/1222668338 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f39c0142290 0x7f39c013e7c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f39b800cb90 tx=0x7f39b8001980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:41.889 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.885+0000 7f39b57fa700 10 client.?.objecter ms_handle_connect 0x7f39c0142290 2023-09-30T02:11:41.889 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.885+0000 7f39b57fa700 10 client.?.objecter resend_mon_ops 2023-09-30T02:11:41.890 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.889+0000 7f39b57fa700 1 -- 172.21.15.59:0/1222668338 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39b800a8b0 con 0x7f39c0142290 2023-09-30T02:11:41.890 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.889+0000 7f39b57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:41.890 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.889+0000 7f39b57fa700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:11:41.890 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.889+0000 7f39b57fa700 10 monclient: dump: 2023-09-30T02:11:41.890 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:41.891 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:41.891 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:41.891 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:41.891 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:41.891 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:41.891 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:41.891 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:41.891 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:41.891 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:41.891 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.889+0000 7f39b57fa700 10 monclient: _finish_auth 0 2023-09-30T02:11:41.891 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.889+0000 7f39b57fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:11:41.891 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.889+0000 7f39b57fa700 1 -- 172.21.15.59:0/1222668338 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39b800aa10 con 0x7f39c0142290 2023-09-30T02:11:41.892 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.889+0000 7f39c7737700 5 monclient: authenticate success, global_id 4960 2023-09-30T02:11:41.892 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.889+0000 7f39c7737700 10 monclient: _renew_subs 2023-09-30T02:11:41.892 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.889+0000 7f39c7737700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:11:41.892 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.889+0000 7f39c7737700 1 -- 172.21.15.59:0/1222668338 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39c013c030 con 0x7f39c0142290 2023-09-30T02:11:41.892 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.889+0000 7f39b57fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:11:41.892 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.889+0000 7f39b57fa700 1 -- 172.21.15.59:0/1222668338 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39b800ad90 con 0x7f39c0142290 2023-09-30T02:11:41.892 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.889+0000 7f39b57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:41.892 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.889+0000 7f39b57fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:11:41.892 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.889+0000 7f39b57fa700 10 monclient: dump: 2023-09-30T02:11:41.892 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:41.892 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:41.892 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:41.893 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:41.893 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:41.893 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:41.893 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:41.893 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:41.893 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:41.893 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:41.893 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.889+0000 7f39b77fe700 4 set_mon_vals no callback set 2023-09-30T02:11:41.893 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.889+0000 7f39c7737700 10 client.4960.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:11:41.893 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.889+0000 7f39c7737700 10 monclient: _renew_subs 2023-09-30T02:11:41.893 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.889+0000 7f39c7737700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:11:41.893 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.889+0000 7f39c7737700 1 -- 172.21.15.59:0/1222668338 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39c013ef40 con 0x7f39c0142290 2023-09-30T02:11:41.894 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.889+0000 7f39c7737700 1 librados: init done 2023-09-30T02:11:41.894 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.890+0000 7f39c7737700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:11:41.894 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.890+0000 7f39c7737700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:11:41.894 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.890+0000 7f39c7737700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:11:41.894 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:41.890+0000 7f39c7737700 1 -- 172.21.15.59:0/1222668338 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39c0064a00 con 0x7f39c0142290 2023-09-30T02:11:42.157 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.157+0000 7f39b57fa700 1 -- 172.21.15.59:0/1222668338 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f39b8007300 con 0x7f39c0142290 2023-09-30T02:11:42.157 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.158+0000 7f39b57fa700 1 --2- 172.21.15.59:0/1222668338 >> v2:172.21.15.110:6800/115870 conn(0x7f39a4068990 0x7f39a406ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:42.158 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.158+0000 7f39c54d3700 1 --2- 172.21.15.59:0/1222668338 >> v2:172.21.15.110:6800/115870 conn(0x7f39a4068990 0x7f39a406ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:42.158 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.158+0000 7f39c54d3700 10 monclient: get_auth_request con 0x7f39a4068990 auth_method 0 2023-09-30T02:11:42.158 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.158+0000 7f39c54d3700 1 --2- 172.21.15.59:0/1222668338 >> v2:172.21.15.110:6800/115870 conn(0x7f39a4068990 0x7f39a406ae40 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f39bc0093f0 tx=0x7f39bc017000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:42.158 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.158+0000 7f39b57fa700 10 client.4960.objecter ms_handle_connect 0x7f39a4068990 2023-09-30T02:11:42.162 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.162+0000 7f39b57fa700 1 -- 172.21.15.59:0/1222668338 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f39b80555f0 con 0x7f39c0142290 2023-09-30T02:11:42.162 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.163+0000 7f39b57fa700 10 client.4960.objecter ms_dispatch 0x7f39c0001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:11:42.162 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.163+0000 7f39b57fa700 3 client.4960.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:11:42.162 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.163+0000 7f39b57fa700 3 client.4960.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:11:42.162 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.163+0000 7f39b57fa700 20 client.4960.objecter dump_active .. 0 homeless 2023-09-30T02:11:42.171 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.172+0000 7f39b57fa700 1 -- 172.21.15.59:0/1222668338 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f39b8051930 con 0x7f39c0142290 2023-09-30T02:11:42.172 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.172+0000 7f39b57fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:11:42.172 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.172+0000 7f39b57fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:11:42.423 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.423+0000 7f39c7737700 1 -- 172.21.15.59:0/1222668338 --> v2:172.21.15.110:6800/115870 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7f39c00a8790 con 0x7f39a4068990 2023-09-30T02:11:42.423 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.424+0000 7f39b57fa700 1 -- 172.21.15.59:0/1222668338 <== mgr.4108 v2:172.21.15.110:6800/115870 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f39c00a8790 con 0x7f39a4068990 2023-09-30T02:11:42.426 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.426+0000 7f39c7737700 10 librados: watch_flush enter 2023-09-30T02:11:42.426 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.426+0000 7f39c7737700 10 librados: watch_flush exit 2023-09-30T02:11:42.426 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.426+0000 7f39c7737700 20 client.4960.objecter shutdown clearing up homeless session... 2023-09-30T02:11:42.427 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.426+0000 7f39c7737700 10 client.4960.objecter successfully canceled tick 2023-09-30T02:11:42.427 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.426+0000 7f39c7737700 1 -- 172.21.15.59:0/1222668338 >> v2:172.21.15.110:6800/115870 conn(0x7f39a4068990 msgr2=0x7f39a406ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:42.427 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.426+0000 7f39c7737700 1 --2- 172.21.15.59:0/1222668338 >> v2:172.21.15.110:6800/115870 conn(0x7f39a4068990 0x7f39a406ae40 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f39bc0093f0 tx=0x7f39bc017000 comp rx=0 tx=0).stop 2023-09-30T02:11:42.427 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.426+0000 7f39c7737700 10 monclient: shutdown 2023-09-30T02:11:42.427 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.426+0000 7f39c7737700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:11:42.427 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.426+0000 7f39c7737700 1 -- 172.21.15.59:0/1222668338 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f39c0142290 msgr2=0x7f39c013e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:42.427 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.426+0000 7f39c7737700 1 --2- 172.21.15.59:0/1222668338 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f39c0142290 0x7f39c013e7c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f39b800cb90 tx=0x7f39b8001980 comp rx=0 tx=0).stop 2023-09-30T02:11:42.427 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.427+0000 7f39c7737700 1 -- 172.21.15.59:0/1222668338 shutdown_connections 2023-09-30T02:11:42.427 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.427+0000 7f39c7737700 1 --2- 172.21.15.59:0/1222668338 >> v2:172.21.15.110:6800/115870 conn(0x7f39a4068990 0x7f39a406ae40 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:42.427 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.427+0000 7f39c7737700 1 --2- 172.21.15.59:0/1222668338 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f39c0142290 0x7f39c013e7c0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:42.427 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.427+0000 7f39c7737700 1 --2- 172.21.15.59:0/1222668338 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39c00a5b70 0x7f39c0144e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:42.427 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.427+0000 7f39c7737700 1 --2- 172.21.15.59:0/1222668338 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f39c00a5f40 0x7f39c013daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:42.428 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.427+0000 7f39c7737700 1 -- 172.21.15.59:0/1222668338 >> 172.21.15.59:0/1222668338 conn(0x7f39c00631f0 msgr2=0x7f39c00b5340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:11:42.428 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.427+0000 7f39c7737700 1 -- 172.21.15.59:0/1222668338 shutdown_connections 2023-09-30T02:11:42.428 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.427+0000 7f39c7737700 1 -- 172.21.15.59:0/1222668338 wait complete. 2023-09-30T02:11:42.428 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.427+0000 7f39c7737700 1 librados: shutdown 2023-09-30T02:11:42.428 INFO:teuthology.orchestra.run.smithi059.stderr:instructed osd(s) 7 to deep-scrub 2023-09-30T02:11:42.441 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T02:11:42.573 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.574+0000 7f5750123700 10 monclient: get_monmap_and_config 2023-09-30T02:11:42.573 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.574+0000 7f5750123700 10 monclient: build_initial_monmap 2023-09-30T02:11:42.573 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.574+0000 7f5750123700 10 monclient: monmap: 2023-09-30T02:11:42.574 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:11:42.574 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:42.574 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:11:42.574550+0000 2023-09-30T02:11:42.574 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:11:42.574550+0000 2023-09-30T02:11:42.574 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:11:42.574 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:42.574 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:11:42.574 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:11:42.574 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:11:42.574 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:11:42.574 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:11:42.574 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:42.576 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.574+0000 7f5750123700 1 Processor -- start 2023-09-30T02:11:42.576 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.575+0000 7f5750123700 1 -- start start 2023-09-30T02:11:42.576 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.575+0000 7f5750123700 10 monclient: init 2023-09-30T02:11:42.576 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.575+0000 7f5750123700 10 monclient: _reopen_session rank -1 2023-09-30T02:11:42.576 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.575+0000 7f5750123700 10 monclient: _add_conns ranks=[4,1,2,0,3] 2023-09-30T02:11:42.576 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.575+0000 7f5750123700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f57480a5b70 0x7f57480b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:42.576 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.575+0000 7f5750123700 10 monclient(hunting): picked mon.noname-c con 0x7f57480a5b70 addr v2:172.21.15.110:3300/0 2023-09-30T02:11:42.577 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.575+0000 7f5750123700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f57480a5f40 addr v1:172.21.15.110:6789/0 2023-09-30T02:11:42.577 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.575+0000 7f5750123700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5748142290 0x7f574813af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:42.577 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.575+0000 7f5750123700 10 monclient(hunting): picked mon.noname-b con 0x7f5748142290 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:11:42.577 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.575+0000 7f5750123700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57480a75a0 con 0x7f57480a5f40 2023-09-30T02:11:42.577 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.575+0000 7f5750123700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:42.577 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.575+0000 7f5750123700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57481426d0 con 0x7f57480a5b70 2023-09-30T02:11:42.577 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.575+0000 7f5750123700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:42.577 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.575+0000 7f5750123700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57480a63e0 con 0x7f5748142290 2023-09-30T02:11:42.577 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.575+0000 7f5750123700 10 monclient(hunting): _renew_subs 2023-09-30T02:11:42.577 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.575+0000 7f5750123700 10 monclient(hunting): authenticate will time out at 2147.877734s 2023-09-30T02:11:42.577 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.576+0000 7f574debf700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f57480a5b70 0x7f57480b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:42.577 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.576+0000 7f574e6c0700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5748142290 0x7f574813af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:42.578 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.576+0000 7f574e6c0700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5748142290 0x7f574813af30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:46274/0 (socket says 172.21.15.59:46274) 2023-09-30T02:11:42.578 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.576+0000 7f574e6c0700 1 -- 172.21.15.59:0/3710691278 learned_addr learned my addr 172.21.15.59:0/3710691278 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:11:42.578 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.576+0000 7f574e6c0700 10 monclient(hunting): get_auth_request con 0x7f5748142290 auth_method 0 2023-09-30T02:11:42.578 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.576+0000 7f574e6c0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:42.578 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.576+0000 7f574e6c0700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:42.578 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.576+0000 7f574e6c0700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:42.578 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.576+0000 7f574debf700 10 monclient(hunting): get_auth_request con 0x7f57480a5b70 auth_method 0 2023-09-30T02:11:42.578 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.576+0000 7f574debf700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:42.578 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.576+0000 7f574debf700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:42.578 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.576+0000 7f574debf700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:42.579 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.577+0000 7f574e6c0700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:42.579 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.577+0000 7f574e6c0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:42.579 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.577+0000 7f574e6c0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:42.579 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.577+0000 7f574debf700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:42.580 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.577+0000 7f574debf700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:42.580 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.577+0000 7f574debf700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:42.580 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.579+0000 7f574e6c0700 10 monclient(hunting): handle_auth_done global_id 4829 payload 274 2023-09-30T02:11:42.580 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.579+0000 7f574e6c0700 1 -- 172.21.15.59:0/3710691278 >> v2:172.21.15.110:3300/0 conn(0x7f57480a5b70 msgr2=0x7f57480b1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:11:42.580 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.579+0000 7f574e6c0700 1 --2- 172.21.15.59:0/3710691278 >> v2:172.21.15.110:3300/0 conn(0x7f57480a5b70 0x7f57480b1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:42.580 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.579+0000 7f574e6c0700 1 -- 172.21.15.59:0/3710691278 >> v1:172.21.15.110:6789/0 conn(0x7f57480a5f40 legacy=0x7f5748141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:42.580 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.579+0000 7f574e6c0700 10 monclient: _finish_hunting 0 2023-09-30T02:11:42.580 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.579+0000 7f574e6c0700 1 monclient: found mon.noname-b 2023-09-30T02:11:42.580 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.579+0000 7f574e6c0700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:11:42.580 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.579+0000 7f574e6c0700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.59:3301/0 2023-09-30T02:11:42.580 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.579+0000 7f574e6c0700 1 -- 172.21.15.59:0/3710691278 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57480af3f0 con 0x7f5748142290 2023-09-30T02:11:42.581 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.581+0000 7f574e6c0700 1 --2- 172.21.15.59:0/3710691278 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5748142290 0x7f574813af30 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f5740009f20 tx=0x7f5740009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b7d5997d71674da server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:42.585 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.586+0000 7f574cebd700 1 -- 172.21.15.59:0/3710691278 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f574000e070 con 0x7f5748142290 2023-09-30T02:11:42.586 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.586+0000 7f574cebd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:42.586 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.586+0000 7f574cebd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:11:42.586 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.586+0000 7f574cebd700 10 monclient: dump: 2023-09-30T02:11:42.586 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:42.586 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:42.586 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:42.586 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:42.587 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:42.587 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:42.587 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:42.587 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:42.587 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:42.587 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:42.587 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.586+0000 7f574cebd700 10 monclient: _finish_auth 0 2023-09-30T02:11:42.587 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.586+0000 7f574cebd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:11:42.587 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.586+0000 7f574cebd700 1 -- 172.21.15.59:0/3710691278 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f574000b9d0 con 0x7f5748142290 2023-09-30T02:11:42.587 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.586+0000 7f5750123700 5 monclient: authenticate success, global_id 4829 2023-09-30T02:11:42.587 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.586+0000 7f5750123700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:11:42.587 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.586+0000 7f574cebd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:11:42.588 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.586+0000 7f5750123700 10 monclient: get_monmap_and_config success 2023-09-30T02:11:42.588 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.586+0000 7f5750123700 4 set_mon_vals no callback set 2023-09-30T02:11:42.588 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.586+0000 7f574cebd700 1 -- 172.21.15.59:0/3710691278 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57400092e0 con 0x7f5748142290 2023-09-30T02:11:42.588 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.586+0000 7f574cebd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:42.588 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.586+0000 7f574cebd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:11:42.588 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.586+0000 7f574cebd700 10 monclient: dump: 2023-09-30T02:11:42.588 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:42.588 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:42.588 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:42.588 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:42.588 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:42.589 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:42.589 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:42.589 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:42.589 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:42.589 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:42.593 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.587+0000 7f5750123700 10 monclient: shutdown 2023-09-30T02:11:42.593 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.587+0000 7f5750123700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:11:42.593 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.587+0000 7f5750123700 1 -- 172.21.15.59:0/3710691278 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5748142290 msgr2=0x7f574813af30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:42.593 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.587+0000 7f5750123700 1 --2- 172.21.15.59:0/3710691278 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5748142290 0x7f574813af30 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f5740009f20 tx=0x7f5740009840 comp rx=0 tx=0).stop 2023-09-30T02:11:42.593 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.587+0000 7f5750123700 1 -- 172.21.15.59:0/3710691278 shutdown_connections 2023-09-30T02:11:42.594 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.587+0000 7f5750123700 1 --2- 172.21.15.59:0/3710691278 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f5748142290 0x7f574813af30 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:42.594 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.587+0000 7f5750123700 1 --2- 172.21.15.59:0/3710691278 >> v2:172.21.15.110:3300/0 conn(0x7f57480a5b70 0x7f57480b1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:42.594 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.587+0000 7f5750123700 1 -- 172.21.15.59:0/3710691278 >> 172.21.15.59:0/3710691278 conn(0x7f57480631f0 msgr2=0x7f57480635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:11:42.594 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.587+0000 7f5750123700 1 -- 172.21.15.59:0/3710691278 shutdown_connections 2023-09-30T02:11:42.594 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.587+0000 7f5750123700 1 -- 172.21.15.59:0/3710691278 wait complete. 2023-09-30T02:11:42.594 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.588+0000 7f5750123700 10 monclient: build_initial_monmap 2023-09-30T02:11:42.594 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.588+0000 7f5750123700 10 monclient: monmap: 2023-09-30T02:11:42.594 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:11:42.594 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:11:42.595 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:11:42.595 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:11:42.595 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:11:42.595 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:11:42.595 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:11:42.595 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:11:42.595 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:11:42.595 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:42.595 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.588+0000 7f5750123700 1 librados: starting msgr at 2023-09-30T02:11:42.595 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.588+0000 7f5750123700 1 librados: starting objecter 2023-09-30T02:11:42.596 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.588+0000 7f5750123700 1 Processor -- start 2023-09-30T02:11:42.596 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.588+0000 7f5750123700 1 -- start start 2023-09-30T02:11:42.596 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.588+0000 7f5750123700 1 librados: setting wanted keys 2023-09-30T02:11:42.596 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.588+0000 7f5750123700 1 librados: calling monclient init 2023-09-30T02:11:42.596 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.588+0000 7f5750123700 10 monclient: init 2023-09-30T02:11:42.596 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.589+0000 7f5750123700 10 monclient: _reopen_session rank -1 2023-09-30T02:11:42.597 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.589+0000 7f5750123700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T02:11:42.597 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.589+0000 7f5750123700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f57480a5b70 0x7f57480aabd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:42.597 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.589+0000 7f5750123700 10 monclient(hunting): picked mon.noname-b con 0x7f57480a5b70 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:11:42.597 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.589+0000 7f5750123700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f57480a5f40 0x7f574813e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:42.597 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.589+0000 7f5750123700 10 monclient(hunting): picked mon.noname-c con 0x7f57480a5f40 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:11:42.597 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.589+0000 7f5750123700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5748142290 0x7f574814dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:42.597 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.589+0000 7f5750123700 10 monclient(hunting): picked mon.noname-a con 0x7f5748142290 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:11:42.597 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.589+0000 7f5750123700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:42.597 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.589+0000 7f5750123700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57480abdd0 con 0x7f5748142290 2023-09-30T02:11:42.597 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.589+0000 7f5750123700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:42.598 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.589+0000 7f5750123700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57480ac040 con 0x7f57480a5b70 2023-09-30T02:11:42.598 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.589+0000 7f5750123700 10 monclient(hunting): start opening mon connection 2023-09-30T02:11:42.598 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.589+0000 7f5750123700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57480aa6b0 con 0x7f57480a5f40 2023-09-30T02:11:42.598 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.589+0000 7f5750123700 10 monclient(hunting): _renew_subs 2023-09-30T02:11:42.598 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.589+0000 7f5750123700 10 monclient(hunting): authenticate will time out at 2147.891735s 2023-09-30T02:11:42.598 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.590+0000 7f574d6be700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f57480a5f40 0x7f574813e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:42.598 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.590+0000 7f574debf700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f57480a5b70 0x7f57480aabd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:42.598 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.590+0000 7f574d6be700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f57480a5f40 0x7f574813e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:46288/0 (socket says 172.21.15.59:46288) 2023-09-30T02:11:42.598 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.590+0000 7f574d6be700 1 -- 172.21.15.59:0/2937394862 learned_addr learned my addr 172.21.15.59:0/2937394862 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:11:42.599 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.590+0000 7f574d6be700 10 monclient(hunting): get_auth_request con 0x7f57480a5f40 auth_method 0 2023-09-30T02:11:42.599 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.590+0000 7f574d6be700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:42.599 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.590+0000 7f574d6be700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:42.599 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.590+0000 7f574d6be700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:42.599 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.590+0000 7f574e6c0700 1 --2- 172.21.15.59:0/2937394862 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5748142290 0x7f574814dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:42.599 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.590+0000 7f574debf700 10 monclient(hunting): get_auth_request con 0x7f57480a5b70 auth_method 0 2023-09-30T02:11:42.599 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.590+0000 7f574debf700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:42.599 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.590+0000 7f574debf700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:42.599 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.590+0000 7f574debf700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:42.599 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.591+0000 7f574d6be700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:42.600 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.591+0000 7f574d6be700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:42.600 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.591+0000 7f574d6be700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:42.600 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.591+0000 7f574e6c0700 10 monclient(hunting): get_auth_request con 0x7f5748142290 auth_method 0 2023-09-30T02:11:42.600 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.591+0000 7f574e6c0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:11:42.600 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.591+0000 7f574e6c0700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:11:42.600 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.591+0000 7f574e6c0700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:11:42.600 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.591+0000 7f574debf700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:42.600 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.591+0000 7f574debf700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:42.600 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.591+0000 7f574debf700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:42.600 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.591+0000 7f574e6c0700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:11:42.601 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.591+0000 7f574e6c0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:11:42.601 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.591+0000 7f574e6c0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:11:42.603 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.594+0000 7f574debf700 10 monclient(hunting): handle_auth_done global_id 4969 payload 824 2023-09-30T02:11:42.603 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.594+0000 7f574debf700 1 -- 172.21.15.59:0/2937394862 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f57480a5f40 msgr2=0x7f574813e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:42.603 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.594+0000 7f574debf700 1 --2- 172.21.15.59:0/2937394862 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f57480a5f40 0x7f574813e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:42.603 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.594+0000 7f574debf700 1 -- 172.21.15.59:0/2937394862 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5748142290 msgr2=0x7f574814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:42.603 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.594+0000 7f574debf700 1 --2- 172.21.15.59:0/2937394862 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5748142290 0x7f574814dc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:42.603 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.594+0000 7f574debf700 10 monclient: _finish_hunting 0 2023-09-30T02:11:42.603 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.594+0000 7f574debf700 1 monclient: found mon.noname-b 2023-09-30T02:11:42.604 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.594+0000 7f574debf700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:11:42.604 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.594+0000 7f574debf700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:11:42.604 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.594+0000 7f574debf700 1 -- 172.21.15.59:0/2937394862 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5740009ae0 con 0x7f57480a5b70 2023-09-30T02:11:42.604 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.596+0000 7f574debf700 1 --2- 172.21.15.59:0/2937394862 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f57480a5b70 0x7f57480aabd0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f574400ea50 tx=0x7f57440091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:42.604 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.596+0000 7f5739ffb700 10 client.?.objecter ms_handle_connect 0x7f57480a5b70 2023-09-30T02:11:42.604 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.596+0000 7f5739ffb700 10 client.?.objecter resend_mon_ops 2023-09-30T02:11:42.604 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.600+0000 7f5739ffb700 1 -- 172.21.15.59:0/2937394862 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57440095d0 con 0x7f57480a5b70 2023-09-30T02:11:42.604 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.600+0000 7f5739ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:42.604 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.600+0000 7f5739ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:11:42.604 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.600+0000 7f5739ffb700 10 monclient: dump: 2023-09-30T02:11:42.604 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:42.605 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:42.605 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:42.605 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:42.605 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:42.605 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:42.605 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:42.605 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:42.605 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:42.605 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:42.605 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.600+0000 7f5739ffb700 10 monclient: _finish_auth 0 2023-09-30T02:11:42.606 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.600+0000 7f5739ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:11:42.606 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.600+0000 7f5739ffb700 1 -- 172.21.15.59:0/2937394862 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f574400c7f0 con 0x7f57480a5b70 2023-09-30T02:11:42.606 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.600+0000 7f5739ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:11:42.606 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.600+0000 7f5739ffb700 1 -- 172.21.15.59:0/2937394862 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f574400cbb0 con 0x7f57480a5b70 2023-09-30T02:11:42.606 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.600+0000 7f5750123700 5 monclient: authenticate success, global_id 4969 2023-09-30T02:11:42.606 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.600+0000 7f5750123700 10 monclient: _renew_subs 2023-09-30T02:11:42.606 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.600+0000 7f5750123700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:11:42.606 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.600+0000 7f5750123700 1 -- 172.21.15.59:0/2937394862 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57480a8dc0 con 0x7f57480a5b70 2023-09-30T02:11:42.606 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.600+0000 7f573bfff700 4 set_mon_vals no callback set 2023-09-30T02:11:42.606 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.600+0000 7f5739ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:11:42.607 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.600+0000 7f5739ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:11:42.607 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.600+0000 7f5739ffb700 10 monclient: dump: 2023-09-30T02:11:42.607 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:11:42.607 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:11:42.607 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:42.607 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:11:42.608 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:11:42.608 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:11:42.608 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:11:42.608 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:11:42.608 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:11:42.608 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:11:42.608 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.601+0000 7f5750123700 10 client.4969.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:11:42.608 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.601+0000 7f5750123700 10 monclient: _renew_subs 2023-09-30T02:11:42.608 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.601+0000 7f5750123700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:11:42.608 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.601+0000 7f5750123700 1 -- 172.21.15.59:0/2937394862 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57480aa130 con 0x7f57480a5b70 2023-09-30T02:11:42.608 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.601+0000 7f5750123700 1 librados: init done 2023-09-30T02:11:42.609 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.602+0000 7f5750123700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:11:42.609 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.602+0000 7f5750123700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:11:42.609 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.602+0000 7f5750123700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:11:42.609 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.602+0000 7f5750123700 1 -- 172.21.15.59:0/2937394862 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57480b1800 con 0x7f57480a5b70 2023-09-30T02:11:42.829 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.829+0000 7f5739ffb700 1 -- 172.21.15.59:0/2937394862 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f5744013470 con 0x7f57480a5b70 2023-09-30T02:11:42.829 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.830+0000 7f5739ffb700 1 --2- 172.21.15.59:0/2937394862 >> v2:172.21.15.110:6800/115870 conn(0x7f572c0689e0 0x7f572c06ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:11:42.830 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.830+0000 7f5739ffb700 1 -- 172.21.15.59:0/2937394862 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f5744084760 con 0x7f57480a5b70 2023-09-30T02:11:42.830 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.830+0000 7f5739ffb700 10 client.4969.objecter ms_dispatch 0x7f5748001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:11:42.830 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.830+0000 7f5739ffb700 3 client.4969.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:11:42.830 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.830+0000 7f5739ffb700 3 client.4969.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:11:42.830 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.830+0000 7f5739ffb700 20 client.4969.objecter dump_active .. 0 homeless 2023-09-30T02:11:42.830 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.830+0000 7f574d6be700 1 --2- 172.21.15.59:0/2937394862 >> v2:172.21.15.110:6800/115870 conn(0x7f572c0689e0 0x7f572c06ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:11:42.830 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.830+0000 7f574d6be700 10 monclient: get_auth_request con 0x7f572c0689e0 auth_method 0 2023-09-30T02:11:42.831 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.831+0000 7f574d6be700 1 --2- 172.21.15.59:0/2937394862 >> v2:172.21.15.110:6800/115870 conn(0x7f572c0689e0 0x7f572c06ae90 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f57480ab490 tx=0x7f573c000bc0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:11:42.831 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.831+0000 7f5739ffb700 10 client.4969.objecter ms_handle_connect 0x7f572c0689e0 2023-09-30T02:11:42.860 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.861+0000 7f5739ffb700 1 -- 172.21.15.59:0/2937394862 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5744052080 con 0x7f57480a5b70 2023-09-30T02:11:42.860 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.861+0000 7f5739ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:11:42.860 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:42.861+0000 7f5739ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:11:43.062 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:43.062+0000 7f5750123700 1 -- 172.21.15.59:0/2937394862 --> v2:172.21.15.110:6800/115870 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f574813b1b0 con 0x7f572c0689e0 2023-09-30T02:11:43.065 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:43.066+0000 7f5739ffb700 1 -- 172.21.15.59:0/2937394862 <== mgr.4108 v2:172.21.15.110:6800/115870 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+70167 (secure 0 0 0) 0x7f574813b1b0 con 0x7f572c0689e0 2023-09-30T02:11:43.066 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:11:43.068 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:43.068+0000 7f5750123700 10 librados: watch_flush enter 2023-09-30T02:11:43.068 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:43.068+0000 7f5750123700 10 librados: watch_flush exit 2023-09-30T02:11:43.069 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:43.068+0000 7f5750123700 20 client.4969.objecter shutdown clearing up homeless session... 2023-09-30T02:11:43.069 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:43.068+0000 7f5750123700 10 client.4969.objecter successfully canceled tick 2023-09-30T02:11:43.069 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:43.068+0000 7f5750123700 1 -- 172.21.15.59:0/2937394862 >> v2:172.21.15.110:6800/115870 conn(0x7f572c0689e0 msgr2=0x7f572c06ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:43.069 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:43.068+0000 7f5750123700 1 --2- 172.21.15.59:0/2937394862 >> v2:172.21.15.110:6800/115870 conn(0x7f572c0689e0 0x7f572c06ae90 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f57480ab490 tx=0x7f573c000bc0 comp rx=0 tx=0).stop 2023-09-30T02:11:43.069 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:43.068+0000 7f5750123700 10 monclient: shutdown 2023-09-30T02:11:43.069 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:43.068+0000 7f5750123700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:11:43.069 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:43.068+0000 7f5750123700 1 -- 172.21.15.59:0/2937394862 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f57480a5b70 msgr2=0x7f57480aabd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:11:43.069 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:43.068+0000 7f5750123700 1 --2- 172.21.15.59:0/2937394862 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f57480a5b70 0x7f57480aabd0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f574400ea50 tx=0x7f57440091a0 comp rx=0 tx=0).stop 2023-09-30T02:11:43.069 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:43.068+0000 7f5750123700 1 -- 172.21.15.59:0/2937394862 shutdown_connections 2023-09-30T02:11:43.069 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:43.068+0000 7f5750123700 1 --2- 172.21.15.59:0/2937394862 >> v2:172.21.15.110:6800/115870 conn(0x7f572c0689e0 0x7f572c06ae90 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:43.070 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:43.068+0000 7f5750123700 1 --2- 172.21.15.59:0/2937394862 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f57480a5b70 0x7f57480aabd0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:43.070 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:43.068+0000 7f5750123700 1 --2- 172.21.15.59:0/2937394862 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5748142290 0x7f574814dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:43.070 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:43.068+0000 7f5750123700 1 --2- 172.21.15.59:0/2937394862 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f57480a5f40 0x7f574813e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:11:43.070 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:43.068+0000 7f5750123700 1 -- 172.21.15.59:0/2937394862 >> 172.21.15.59:0/2937394862 conn(0x7f57480631f0 msgr2=0x7f57480b0be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:11:43.070 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:43.068+0000 7f5750123700 1 -- 172.21.15.59:0/2937394862 shutdown_connections 2023-09-30T02:11:43.070 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:43.068+0000 7f5750123700 1 -- 172.21.15.59:0/2937394862 wait complete. 2023-09-30T02:11:43.070 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:11:43.068+0000 7f5750123700 1 librados: shutdown 2023-09-30T02:11:43.070 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2023-09-30T02:11:43.081 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":199,"stamp":"2023-09-30T02:11:41.649731+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":43088,"kb_used_data":2448,"kb_used_omap":0,"kb_used_meta":40576,"kb_avail":754931632,"statfs":{"total":773094113280,"available":773049991168,"internally_reserved":0,"allocated":2506752,"data_stored":1396000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":41549824},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":73,"apply_latency_ms":73,"commit_latency_ns":73000000,"apply_latency_ns":73000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002543"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-09-30T02:06:19.722714+0000","last_change":"2023-09-30T02:06:18.210390+0000","last_active":"2023-09-30T02:06:19.722714+0000","last_peered":"2023-09-30T02:06:19.722714+0000","last_clean":"2023-09-30T02:06:19.722714+0000","last_became_active":"2023-09-30T02:06:18.198153+0000","last_became_peered":"2023-09-30T02:06:18.198153+0000","last_unstale":"2023-09-30T02:06:19.722714+0000","last_undegraded":"2023-09-30T02:06:19.722714+0000","last_fullsized":"2023-09-30T02:06:19.722714+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_clean_scrub_stamp":"2023-09-30T02:06:15.800912+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-09-30T02:06:19.701414+0000","last_change":"2023-09-30T02:06:18.262708+0000","last_active":"2023-09-30T02:06:19.701414+0000","last_peered":"2023-09-30T02:06:19.701414+0000","last_clean":"2023-09-30T02:06:19.701414+0000","last_became_active":"2023-09-30T02:06:18.224464+0000","last_became_peered":"2023-09-30T02:06:18.224464+0000","last_unstale":"2023-09-30T02:06:19.701414+0000","last_undegraded":"2023-09-30T02:06:19.701414+0000","last_fullsized":"2023-09-30T02:06:19.701414+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_clean_scrub_stamp":"2023-09-30T02:06:15.800912+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-09-30T02:06:19.712214+0000","last_change":"2023-09-30T02:06:17.910244+0000","last_active":"2023-09-30T02:06:19.712214+0000","last_peered":"2023-09-30T02:06:19.712214+0000","last_clean":"2023-09-30T02:06:19.712214+0000","last_became_active":"2023-09-30T02:06:17.855689+0000","last_became_peered":"2023-09-30T02:06:17.855689+0000","last_unstale":"2023-09-30T02:06:19.712214+0000","last_undegraded":"2023-09-30T02:06:19.712214+0000","last_fullsized":"2023-09-30T02:06:19.712214+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_clean_scrub_stamp":"2023-09-30T02:06:15.800912+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-09-30T02:06:19.697676+0000","last_change":"2023-09-30T02:06:17.806354+0000","last_active":"2023-09-30T02:06:19.697676+0000","last_peered":"2023-09-30T02:06:19.697676+0000","last_clean":"2023-09-30T02:06:19.697676+0000","last_became_active":"2023-09-30T02:06:17.706615+0000","last_became_peered":"2023-09-30T02:06:17.706615+0000","last_unstale":"2023-09-30T02:06:19.697676+0000","last_undegraded":"2023-09-30T02:06:19.697676+0000","last_fullsized":"2023-09-30T02:06:19.697676+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_clean_scrub_stamp":"2023-09-30T02:06:15.800912+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"25'78","reported_seq":130,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-09-30T02:11:38.810874+0000","last_change":"2023-09-30T02:11:38.802202+0000","last_active":"2023-09-30T02:11:38.810874+0000","last_peered":"2023-09-30T02:11:38.810874+0000","last_clean":"2023-09-30T02:11:38.810874+0000","last_became_active":"2023-09-30T02:06:11.592631+0000","last_became_peered":"2023-09-30T02:06:11.592631+0000","last_unstale":"2023-09-30T02:11:38.810874+0000","last_undegraded":"2023-09-30T02:11:38.810874+0000","last_fullsized":"2023-09-30T02:11:38.810874+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"25'78","last_scrub_stamp":"2023-09-30T02:11:38.789901+0000","last_deep_scrub":"25'78","last_deep_scrub_stamp":"2023-09-30T02:11:38.789901+0000","last_clean_scrub_stamp":"2023-09-30T02:11:38.789901+0000","objects_scrubbed":2,"log_size":78,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T13:15:01.381693+0000","scrub_duration":0.75921421200000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-09-30T02:06:19.716778+0000","last_change":"2023-09-30T02:06:18.102959+0000","last_active":"2023-09-30T02:06:19.716778+0000","last_peered":"2023-09-30T02:06:19.716778+0000","last_clean":"2023-09-30T02:06:19.716778+0000","last_became_active":"2023-09-30T02:06:18.091407+0000","last_became_peered":"2023-09-30T02:06:18.091407+0000","last_unstale":"2023-09-30T02:06:19.716778+0000","last_undegraded":"2023-09-30T02:06:19.716778+0000","last_fullsized":"2023-09-30T02:06:19.716778+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_clean_scrub_stamp":"2023-09-30T02:06:15.800912+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":10,"reported_epoch":24,"state":"active+clean","last_fresh":"2023-09-30T02:06:18.250861+0000","last_change":"2023-09-30T02:06:17.465018+0000","last_active":"2023-09-30T02:06:18.250861+0000","last_peered":"2023-09-30T02:06:18.250861+0000","last_clean":"2023-09-30T02:06:18.250861+0000","last_became_active":"2023-09-30T02:06:17.420574+0000","last_became_peered":"2023-09-30T02:06:17.420574+0000","last_unstale":"2023-09-30T02:06:18.250861+0000","last_undegraded":"2023-09-30T02:06:18.250861+0000","last_fullsized":"2023-09-30T02:06:18.250861+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_clean_scrub_stamp":"2023-09-30T02:06:15.800912+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:46:28.414555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-09-30T02:06:19.707317+0000","last_change":"2023-09-30T02:06:17.896913+0000","last_active":"2023-09-30T02:06:19.707317+0000","last_peered":"2023-09-30T02:06:19.707317+0000","last_clean":"2023-09-30T02:06:19.707317+0000","last_became_active":"2023-09-30T02:06:17.890282+0000","last_became_peered":"2023-09-30T02:06:17.890282+0000","last_unstale":"2023-09-30T02:06:19.707317+0000","last_undegraded":"2023-09-30T02:06:19.707317+0000","last_fullsized":"2023-09-30T02:06:19.707317+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_clean_scrub_stamp":"2023-09-30T02:06:15.800912+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":10,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-09-30T02:06:19.304924+0000","last_change":"2023-09-30T02:06:17.877329+0000","last_active":"2023-09-30T02:06:19.304924+0000","last_peered":"2023-09-30T02:06:19.304924+0000","last_clean":"2023-09-30T02:06:19.304924+0000","last_became_active":"2023-09-30T02:06:17.836124+0000","last_became_peered":"2023-09-30T02:06:17.836124+0000","last_unstale":"2023-09-30T02:06:19.304924+0000","last_undegraded":"2023-09-30T02:06:19.304924+0000","last_fullsized":"2023-09-30T02:06:19.304924+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_clean_scrub_stamp":"2023-09-30T02:06:15.800912+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":19,"seq":81604378694,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5208,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366632,"statfs":{"total":96636764160,"available":96631431168,"internally_reserved":0,"allocated":212992,"data_stored":75040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":12.859999999999999,"5min":16.204000000000001,"15min":26.364000000000001},"min":{"1min":8.8520000000000003,"5min":7.1429999999999998,"15min":7.1429999999999998},"max":{"1min":17.263000000000002,"5min":414.25400000000002,"15min":414.25400000000002},"last":16.280999999999999},{"interface":"front","average":{"1min":12.984,"5min":17.393999999999998,"15min":29.042999999999999},"min":{"1min":7.1289999999999996,"5min":7.1289999999999996,"15min":7.1289999999999996},"max":{"1min":21.555,"5min":421.178,"15min":421.178},"last":12.614000000000001}]},{"osd":1,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":13.353,"5min":15.817,"15min":24.888999999999999},"min":{"1min":8.9610000000000003,"5min":6.8200000000000003,"15min":6.8200000000000003},"max":{"1min":16.489999999999998,"5min":326.60899999999998,"15min":326.60899999999998},"last":12.226000000000001},{"interface":"front","average":{"1min":14.826000000000001,"5min":19.149999999999999,"15min":31.222999999999999},"min":{"1min":8.5099999999999998,"5min":8.5079999999999991,"15min":8.5079999999999991},"max":{"1min":24.559999999999999,"5min":418.35500000000002,"15min":418.35500000000002},"last":9.1069999999999993}]},{"osd":2,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":16.120000000000001,"5min":21.582000000000001,"15min":33.603999999999999},"min":{"1min":9.2850000000000001,"5min":8.3960000000000008,"15min":8.3960000000000008},"max":{"1min":23.693999999999999,"5min":423.46699999999998,"15min":423.46699999999998},"last":11.455},{"interface":"front","average":{"1min":14.952,"5min":19.960999999999999,"15min":33.359999999999999},"min":{"1min":9.0749999999999993,"5min":8.4000000000000004,"15min":8.4000000000000004},"max":{"1min":23.123000000000001,"5min":438.697,"15min":438.697},"last":17.873999999999999}]},{"osd":4,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":15.225,"5min":19.898,"15min":32.472000000000001},"min":{"1min":9.5370000000000008,"5min":8.4030000000000005,"15min":8.4030000000000005},"max":{"1min":26.390000000000001,"5min":443.36399999999998,"15min":443.36399999999998},"last":17.149999999999999},{"interface":"front","average":{"1min":17.562999999999999,"5min":22.780999999999999,"15min":35.773000000000003},"min":{"1min":10.282,"5min":10.282,"15min":10.282},"max":{"1min":23.670000000000002,"5min":453.20699999999999,"15min":453.20699999999999},"last":19.344999999999999}]},{"osd":5,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":14.875,"5min":20.896999999999998,"15min":33.405000000000001},"min":{"1min":8.2430000000000003,"5min":8.141,"15min":8.141},"max":{"1min":22.359000000000002,"5min":443.94600000000003,"15min":443.94600000000003},"last":15.193},{"interface":"front","average":{"1min":19.251000000000001,"5min":22.654,"15min":32.534999999999997},"min":{"1min":11.301,"5min":10.746,"15min":10.746},"max":{"1min":25.305,"5min":335.11799999999999,"15min":335.11799999999999},"last":21.353000000000002}]},{"osd":6,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":18.507000000000001,"5min":22.931999999999999,"15min":36.216999999999999},"min":{"1min":12.955,"5min":9.4009999999999998,"15min":9.4009999999999998},"max":{"1min":28.02,"5min":432.34699999999998,"15min":432.34699999999998},"last":13.403},{"interface":"front","average":{"1min":21.754000000000001,"5min":24.712,"15min":34.631999999999998},"min":{"1min":18.292999999999999,"5min":13.018000000000001,"15min":13.018000000000001},"max":{"1min":27.356999999999999,"5min":334.178,"15min":334.178},"last":22.361999999999998}]},{"osd":7,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":17.149000000000001,"5min":23.225000000000001,"15min":35.642000000000003},"min":{"1min":10.933999999999999,"5min":10.928000000000001,"15min":10.928000000000001},"max":{"1min":23.986999999999998,"5min":461.42000000000002,"15min":461.42000000000002},"last":20.103999999999999},{"interface":"front","average":{"1min":22.254000000000001,"5min":26.744,"15min":39.024000000000001},"min":{"1min":17.257000000000001,"5min":15.686,"15min":15.686},"max":{"1min":25.774000000000001,"5min":431.27800000000002,"15min":431.27800000000002},"last":23.481000000000002}]}]},{"osd":1,"up_from":18,"seq":77309411397,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5856,"kb_used_data":600,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":94365984,"statfs":{"total":96636764160,"available":96630767616,"internally_reserved":0,"allocated":614400,"data_stored":472880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":65,"apply_latency_ms":65,"commit_latency_ns":65000000,"apply_latency_ns":65000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":13.101000000000001,"5min":13.603999999999999,"15min":13.84},"min":{"1min":8.7560000000000002,"5min":7.1639999999999997,"15min":7.1639999999999997},"max":{"1min":19.596,"5min":24.626999999999999,"15min":24.626999999999999},"last":15.388},{"interface":"front","average":{"1min":11.407,"5min":12.699,"15min":14.554},"min":{"1min":6.4859999999999998,"5min":6.4859999999999998,"15min":6.4859999999999998},"max":{"1min":17.785,"5min":38.213999999999999,"15min":38.213999999999999},"last":19.498999999999999}]},{"osd":2,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":12.917999999999999,"5min":13.146000000000001,"15min":13.359999999999999},"min":{"1min":8.4949999999999992,"5min":6.9509999999999996,"15min":6.9509999999999996},"max":{"1min":20.673999999999999,"5min":39.070999999999998,"15min":39.070999999999998},"last":13.785},{"interface":"front","average":{"1min":14.161,"5min":15.342000000000001,"15min":16.507999999999999},"min":{"1min":8.6050000000000004,"5min":8.6050000000000004,"15min":8.6050000000000004},"max":{"1min":20.436,"5min":28.981999999999999,"15min":28.981999999999999},"last":12.920999999999999}]},{"osd":3,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":18.742999999999999,"5min":28.812999999999999,"15min":56.773000000000003},"min":{"1min":13.654999999999999,"5min":9.2110000000000003,"15min":9.2110000000000003},"max":{"1min":23.431999999999999,"5min":789.26099999999997,"15min":789.26099999999997},"last":21.184999999999999},{"interface":"front","average":{"1min":16.210999999999999,"5min":26.545999999999999,"15min":55.965000000000003},"min":{"1min":7.8869999999999996,"5min":7.8869999999999996,"15min":7.8869999999999996},"max":{"1min":24.469000000000001,"5min":827.53899999999999,"15min":827.53899999999999},"last":24.033999999999999}]},{"osd":4,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":14.420999999999999,"5min":21.696999999999999,"15min":34.310000000000002},"min":{"1min":11.004,"5min":8.7029999999999994,"15min":8.7029999999999994},"max":{"1min":20.788,"5min":390.31700000000001,"15min":390.31700000000001},"last":12.039},{"interface":"front","average":{"1min":17.638000000000002,"5min":21.739000000000001,"15min":34.366999999999997},"min":{"1min":10.234,"5min":10.234,"15min":10.234},"max":{"1min":24.907,"5min":381.255,"15min":381.255},"last":18.641999999999999}]},{"osd":5,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":13.234999999999999,"5min":14.538,"15min":15.808999999999999},"min":{"1min":9.2680000000000007,"5min":7.6849999999999996,"15min":7.6849999999999996},"max":{"1min":21.724,"5min":31.677,"15min":31.677},"last":12.603999999999999},{"interface":"front","average":{"1min":20.417000000000002,"5min":20.757000000000001,"15min":21.004999999999999},"min":{"1min":9.9860000000000007,"5min":9.9860000000000007,"15min":9.9860000000000007},"max":{"1min":26.379000000000001,"5min":31.449999999999999,"15min":31.449999999999999},"last":25.015999999999998}]},{"osd":6,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":18.373999999999999,"5min":19.23,"15min":21.132000000000001},"min":{"1min":12.045,"5min":9.7560000000000002,"15min":9.7560000000000002},"max":{"1min":24.495999999999999,"5min":41.316000000000003,"15min":41.316000000000003},"last":17.035},{"interface":"front","average":{"1min":16.523,"5min":17.629999999999999,"15min":19.443999999999999},"min":{"1min":10.407,"5min":9.2699999999999996,"15min":9.2699999999999996},"max":{"1min":22.161999999999999,"5min":53.012,"15min":53.012},"last":17.867999999999999}]},{"osd":7,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":19.762,"5min":24.777999999999999,"15min":37.265999999999998},"min":{"1min":14.295999999999999,"5min":9.5310000000000006,"15min":9.5310000000000006},"max":{"1min":23.609000000000002,"5min":332.072,"15min":332.072},"last":14.691000000000001},{"interface":"front","average":{"1min":17.829000000000001,"5min":20.178999999999998,"15min":24.689},"min":{"1min":11.316000000000001,"5min":8.141,"15min":8.141},"max":{"1min":23.631,"5min":108.00700000000001,"15min":108.00700000000001},"last":20.366}]}]},{"osd":7,"up_from":14,"seq":60129542215,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5272,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":94366568,"statfs":{"total":96636764160,"available":96631365632,"internally_reserved":0,"allocated":212992,"data_stored":75040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:11:21 2023","interfaces":[{"interface":"back","average":{"1min":8.6750000000000007,"5min":8.9369999999999994,"15min":8.9199999999999999},"min":{"1min":6.2140000000000004,"5min":5.3120000000000003,"15min":5.3120000000000003},"max":{"1min":16.916,"5min":19.491,"15min":19.491},"last":6.274},{"interface":"front","average":{"1min":10.153,"5min":9.9879999999999995,"15min":9.7560000000000002},"min":{"1min":7.3099999999999996,"5min":6.2850000000000001,"15min":6.2850000000000001},"max":{"1min":19.263000000000002,"5min":20.952000000000002,"15min":20.952000000000002},"last":13.173}]},{"osd":1,"last update":"Sat Sep 30 02:11:21 2023","interfaces":[{"interface":"back","average":{"1min":10.753,"5min":11.709,"15min":14.192},"min":{"1min":6.6630000000000003,"5min":6.6630000000000003,"15min":6.6630000000000003},"max":{"1min":18.591000000000001,"5min":89.454999999999998,"15min":89.454999999999998},"last":11.272},{"interface":"front","average":{"1min":11.548999999999999,"5min":12.834,"15min":14.573},"min":{"1min":8.0909999999999993,"5min":7.0700000000000003,"15min":7.0700000000000003},"max":{"1min":19.093,"5min":83.695999999999998,"15min":83.695999999999998},"last":10.196999999999999}]},{"osd":2,"last update":"Sat Sep 30 02:11:21 2023","interfaces":[{"interface":"back","average":{"1min":13.856,"5min":13.919,"15min":13.823},"min":{"1min":8.1240000000000006,"5min":7.4669999999999996,"15min":7.4669999999999996},"max":{"1min":20.259,"5min":21.370999999999999,"15min":21.370999999999999},"last":16.166},{"interface":"front","average":{"1min":14.368,"5min":14.385,"15min":15.127000000000001},"min":{"1min":10.176,"5min":7.8920000000000003,"15min":7.8920000000000003},"max":{"1min":19.542999999999999,"5min":21.712,"15min":21.712},"last":12.458}]},{"osd":3,"last update":"Sat Sep 30 02:11:21 2023","interfaces":[{"interface":"back","average":{"1min":15.417999999999999,"5min":15.926,"15min":15.646000000000001},"min":{"1min":11.319000000000001,"5min":7.1710000000000003,"15min":7.1710000000000003},"max":{"1min":21.238,"5min":24.276,"15min":24.276},"last":19.545999999999999},{"interface":"front","average":{"1min":16.335999999999999,"5min":16.087,"15min":16.295000000000002},"min":{"1min":12.477,"5min":7.9630000000000001,"15min":7.9630000000000001},"max":{"1min":22.021999999999998,"5min":24.716000000000001,"15min":24.716000000000001},"last":20.483000000000001}]},{"osd":4,"last update":"Sat Sep 30 02:11:12 2023","interfaces":[{"interface":"back","average":{"1min":16.693999999999999,"5min":17.527000000000001,"15min":19.007999999999999},"min":{"1min":9.4380000000000006,"5min":5.7039999999999997,"15min":5.7039999999999997},"max":{"1min":23.076000000000001,"5min":99.298000000000002,"15min":99.298000000000002},"last":16.765999999999998},{"interface":"front","average":{"1min":12.294,"5min":15.105,"15min":17.399000000000001},"min":{"1min":7.1509999999999998,"5min":7.1509999999999998,"15min":7.1509999999999998},"max":{"1min":17.899000000000001,"5min":86.938999999999993,"15min":86.938999999999993},"last":15.055}]},{"osd":5,"last update":"Sat Sep 30 02:11:12 2023","interfaces":[{"interface":"back","average":{"1min":15.673999999999999,"5min":17.282,"15min":19.91},"min":{"1min":8.6509999999999998,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":21.527000000000001,"5min":118.675,"15min":118.675},"last":17.835999999999999},{"interface":"front","average":{"1min":18.716999999999999,"5min":19.459,"15min":21.693999999999999},"min":{"1min":16.239999999999998,"5min":8.7330000000000005,"15min":8.7330000000000005},"max":{"1min":23.338999999999999,"5min":114.14400000000001,"15min":114.14400000000001},"last":15.699}]},{"osd":6,"last update":"Sat Sep 30 02:11:12 2023","interfaces":[{"interface":"back","average":{"1min":19.045000000000002,"5min":20.443999999999999,"15min":23.024000000000001},"min":{"1min":8.4359999999999999,"5min":8.4359999999999999,"15min":8.4359999999999999},"max":{"1min":24.417000000000002,"5min":111.048,"15min":111.048},"last":18.709},{"interface":"front","average":{"1min":18.745000000000001,"5min":19.427,"15min":22.183},"min":{"1min":12.542,"5min":9.8140000000000001,"15min":9.8140000000000001},"max":{"1min":23.036000000000001,"5min":115.241,"15min":115.241},"last":21.466000000000001}]}]},{"osd":0,"up_from":18,"seq":77309411398,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5208,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366632,"statfs":{"total":96636764160,"available":96631431168,"internally_reserved":0,"allocated":212992,"data_stored":75040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 30 02:11:19 2023","interfaces":[{"interface":"back","average":{"1min":10.754,"5min":16.196000000000002,"15min":29.155000000000001},"min":{"1min":7.2089999999999996,"5min":7.2089999999999996,"15min":7.2089999999999996},"max":{"1min":19.498999999999999,"5min":425.49599999999998,"15min":425.49599999999998},"last":15.128},{"interface":"front","average":{"1min":13.093,"5min":20.338000000000001,"15min":39.884},"min":{"1min":8.1210000000000004,"5min":7.9269999999999996,"15min":7.9269999999999996},"max":{"1min":18.349,"5min":667.81100000000004,"15min":667.81100000000004},"last":14.298}]},{"osd":2,"last update":"Sat Sep 30 02:11:19 2023","interfaces":[{"interface":"back","average":{"1min":12.025,"5min":14.532999999999999,"15min":18.928999999999998},"min":{"1min":8.2110000000000003,"5min":7.718,"15min":7.718},"max":{"1min":16.954000000000001,"5min":162.50700000000001,"15min":162.50700000000001},"last":21.158999999999999},{"interface":"front","average":{"1min":17.457000000000001,"5min":17.991,"15min":20.966999999999999},"min":{"1min":10.747,"5min":8.5079999999999991,"15min":8.5079999999999991},"max":{"1min":23.617000000000001,"5min":131.863,"15min":131.863},"last":10.91}]},{"osd":3,"last update":"Sat Sep 30 02:11:19 2023","interfaces":[{"interface":"back","average":{"1min":13.837,"5min":14.763,"15min":15.271000000000001},"min":{"1min":8.1020000000000003,"5min":7.4859999999999998,"15min":7.4859999999999998},"max":{"1min":22.571999999999999,"5min":24.763000000000002,"15min":24.763000000000002},"last":11.733000000000001},{"interface":"front","average":{"1min":18.533999999999999,"5min":18.309999999999999,"15min":18.783000000000001},"min":{"1min":10.459,"5min":9.8569999999999993,"15min":9.8569999999999993},"max":{"1min":23.57,"5min":27.309000000000001,"15min":27.309000000000001},"last":17.803999999999998}]},{"osd":4,"last update":"Sat Sep 30 02:11:19 2023","interfaces":[{"interface":"back","average":{"1min":16.027999999999999,"5min":16.099,"15min":19.271000000000001},"min":{"1min":9.9179999999999993,"5min":9.0700000000000003,"15min":9.0700000000000003},"max":{"1min":22.122,"5min":102.456,"15min":102.456},"last":16.757000000000001},{"interface":"front","average":{"1min":13.881,"5min":17.271000000000001,"15min":20.933},"min":{"1min":8.7929999999999993,"5min":8.7929999999999993,"15min":8.7929999999999993},"max":{"1min":19.463999999999999,"5min":98.293999999999997,"15min":98.293999999999997},"last":13.191000000000001}]},{"osd":5,"last update":"Sat Sep 30 02:11:19 2023","interfaces":[{"interface":"back","average":{"1min":20.268999999999998,"5min":20.559999999999999,"15min":22.850000000000001},"min":{"1min":14.006,"5min":10.726000000000001,"15min":10.726000000000001},"max":{"1min":28.995000000000001,"5min":79.566999999999993,"15min":79.566999999999993},"last":24.268999999999998},{"interface":"front","average":{"1min":16.812999999999999,"5min":18.582999999999998,"15min":21.466999999999999},"min":{"1min":9.7599999999999998,"5min":9.7599999999999998,"15min":9.7599999999999998},"max":{"1min":24.213000000000001,"5min":109.80500000000001,"15min":109.80500000000001},"last":9.5820000000000007}]},{"osd":6,"last update":"Sat Sep 30 02:11:19 2023","interfaces":[{"interface":"back","average":{"1min":18.077999999999999,"5min":20.367000000000001,"15min":26.062999999999999},"min":{"1min":11.140000000000001,"5min":9.2829999999999995,"15min":9.2829999999999995},"max":{"1min":24.276,"5min":161.01499999999999,"15min":161.01499999999999},"last":19.298999999999999},{"interface":"front","average":{"1min":22.562000000000001,"5min":22.975000000000001,"15min":26.202999999999999},"min":{"1min":13.904,"5min":13.163,"15min":13.163},"max":{"1min":27.786000000000001,"5min":118.71599999999999,"15min":118.71599999999999},"last":22.602}]},{"osd":7,"last update":"Sat Sep 30 02:11:19 2023","interfaces":[{"interface":"back","average":{"1min":20.117999999999999,"5min":20.393000000000001,"15min":23.483000000000001},"min":{"1min":11.741,"5min":9.8810000000000002,"15min":9.8810000000000002},"max":{"1min":26.491,"5min":111.86,"15min":111.86},"last":18.484999999999999},{"interface":"front","average":{"1min":21.664000000000001,"5min":22.388999999999999,"15min":26.010999999999999},"min":{"1min":15.186999999999999,"5min":10.237,"15min":10.237},"max":{"1min":27.012,"5min":126.889,"15min":126.889},"last":23.600000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574918,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5208,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366632,"statfs":{"total":96636764160,"available":96631431168,"internally_reserved":0,"allocated":212992,"data_stored":75040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:11:20 2023","interfaces":[{"interface":"back","average":{"1min":9.6969999999999992,"5min":9.8569999999999993,"15min":9.7870000000000008},"min":{"1min":6.5830000000000002,"5min":5.774,"15min":5.774},"max":{"1min":16.530999999999999,"5min":19.709,"15min":19.709},"last":14.106999999999999},{"interface":"front","average":{"1min":10.919,"5min":10.390000000000001,"15min":11.138999999999999},"min":{"1min":6.1239999999999997,"5min":5.0419999999999998,"15min":5.0419999999999998},"max":{"1min":19.603000000000002,"5min":23.134,"15min":23.134},"last":7.9160000000000004}]},{"osd":1,"last update":"Sat Sep 30 02:11:20 2023","interfaces":[{"interface":"back","average":{"1min":11.829000000000001,"5min":12.981999999999999,"15min":14.561999999999999},"min":{"1min":7.351,"5min":6.3010000000000002,"15min":6.3010000000000002},"max":{"1min":17.946999999999999,"5min":27.242999999999999,"15min":27.242999999999999},"last":9.9120000000000008},{"interface":"front","average":{"1min":11.532999999999999,"5min":11.537000000000001,"15min":12.59},"min":{"1min":7.9770000000000003,"5min":7.4989999999999997,"15min":7.4989999999999997},"max":{"1min":18.957000000000001,"5min":24.388999999999999,"15min":24.388999999999999},"last":11.996}]},{"osd":2,"last update":"Sat Sep 30 02:11:20 2023","interfaces":[{"interface":"back","average":{"1min":14.006,"5min":14.630000000000001,"15min":15.132},"min":{"1min":9.6690000000000005,"5min":7.7039999999999997,"15min":7.7039999999999997},"max":{"1min":19.521000000000001,"5min":24.510999999999999,"15min":24.510999999999999},"last":19.265999999999998},{"interface":"front","average":{"1min":12.595000000000001,"5min":13.534000000000001,"15min":14.086},"min":{"1min":7.9699999999999998,"5min":7.9699999999999998,"15min":7.9699999999999998},"max":{"1min":20.085000000000001,"5min":25.033999999999999,"15min":25.033999999999999},"last":9.202}]},{"osd":3,"last update":"Sat Sep 30 02:11:20 2023","interfaces":[{"interface":"back","average":{"1min":14.676,"5min":19.765999999999998,"15min":31.925000000000001},"min":{"1min":8.3770000000000007,"5min":8.2089999999999996,"15min":8.2089999999999996},"max":{"1min":20.157,"5min":540.52099999999996,"15min":540.52099999999996},"last":16.553000000000001},{"interface":"front","average":{"1min":15.218,"5min":21.224,"15min":35.917999999999999},"min":{"1min":9.4489999999999998,"5min":7.9649999999999999,"15min":7.9649999999999999},"max":{"1min":19.452999999999999,"5min":644.31600000000003,"15min":644.31600000000003},"last":14.73}]},{"osd":4,"last update":"Sat Sep 30 02:11:15 2023","interfaces":[{"interface":"back","average":{"1min":17.922000000000001,"5min":18.664999999999999,"15min":22.556999999999999},"min":{"1min":13.959,"5min":9.0969999999999995,"15min":9.0969999999999995},"max":{"1min":21.425999999999998,"5min":179.749,"15min":179.749},"last":17.558},{"interface":"front","average":{"1min":15.129,"5min":15.641999999999999,"15min":19.817},"min":{"1min":8.7550000000000008,"5min":7.6070000000000002,"15min":7.6070000000000002},"max":{"1min":19.663,"5min":174.52799999999999,"15min":174.52799999999999},"last":11.07}]},{"osd":5,"last update":"Sat Sep 30 02:11:15 2023","interfaces":[{"interface":"back","average":{"1min":16.963000000000001,"5min":16.859000000000002,"15min":16.969000000000001},"min":{"1min":9.6210000000000004,"5min":8.2379999999999995,"15min":8.2379999999999995},"max":{"1min":21.975999999999999,"5min":36.709000000000003,"15min":36.709000000000003},"last":13.23},{"interface":"front","average":{"1min":15.141999999999999,"5min":16.25,"15min":16.556000000000001},"min":{"1min":9.0809999999999995,"5min":5.0279999999999996,"15min":5.0279999999999996},"max":{"1min":21.393999999999998,"5min":27.364999999999998,"15min":27.364999999999998},"last":12.619}]},{"osd":7,"last update":"Sat Sep 30 02:11:09 2023","interfaces":[{"interface":"back","average":{"1min":19.030000000000001,"5min":19.597000000000001,"15min":22.396999999999998},"min":{"1min":13.644,"5min":10.367000000000001,"15min":10.367000000000001},"max":{"1min":22.771000000000001,"5min":105.801,"15min":105.801},"last":20.428000000000001},{"interface":"front","average":{"1min":17.553000000000001,"5min":19.193999999999999,"15min":20.960000000000001},"min":{"1min":7.3120000000000003,"5min":7.3120000000000003,"15min":7.3120000000000003},"max":{"1min":23.283000000000001,"5min":90.156000000000006,"15min":90.156000000000006},"last":18.437000000000001}]}]},{"osd":4,"up_from":15,"seq":64424509511,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5208,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366632,"statfs":{"total":96636764160,"available":96631431168,"internally_reserved":0,"allocated":212992,"data_stored":75040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:11:17 2023","interfaces":[{"interface":"back","average":{"1min":11.865,"5min":10.567,"15min":9.4399999999999995},"min":{"1min":6.3970000000000002,"5min":5.2699999999999996,"15min":5.2699999999999996},"max":{"1min":20.655000000000001,"5min":24.532,"15min":24.532},"last":8.3659999999999997},{"interface":"front","average":{"1min":11.137,"5min":10.974,"15min":11.926},"min":{"1min":7.0190000000000001,"5min":5.4530000000000003,"15min":5.4530000000000003},"max":{"1min":21.841000000000001,"5min":25.225000000000001,"15min":25.225000000000001},"last":10.884}]},{"osd":1,"last update":"Sat Sep 30 02:11:17 2023","interfaces":[{"interface":"back","average":{"1min":12.827,"5min":13.271000000000001,"15min":13.897},"min":{"1min":8.3960000000000008,"5min":6.2830000000000004,"15min":6.2830000000000004},"max":{"1min":17.864000000000001,"5min":28.917000000000002,"15min":28.917000000000002},"last":8.077},{"interface":"front","average":{"1min":10.146000000000001,"5min":11.148,"15min":11.789999999999999},"min":{"1min":7.4059999999999997,"5min":7.0250000000000004,"15min":7.0250000000000004},"max":{"1min":19.279,"5min":30.922999999999998,"15min":30.922999999999998},"last":9.8870000000000005}]},{"osd":2,"last update":"Sat Sep 30 02:11:17 2023","interfaces":[{"interface":"back","average":{"1min":14.563000000000001,"5min":13.641999999999999,"15min":14.407999999999999},"min":{"1min":7.8300000000000001,"5min":7.1609999999999996,"15min":7.1609999999999996},"max":{"1min":19.428000000000001,"5min":21.175000000000001,"15min":21.175000000000001},"last":16.468},{"interface":"front","average":{"1min":12.637,"5min":13.872,"15min":14.563000000000001},"min":{"1min":6.7060000000000004,"5min":6.7060000000000004,"15min":6.7060000000000004},"max":{"1min":18.747,"5min":27.533999999999999,"15min":27.533999999999999},"last":9.125}]},{"osd":3,"last update":"Sat Sep 30 02:11:23 2023","interfaces":[{"interface":"back","average":{"1min":16.146000000000001,"5min":16.042999999999999,"15min":15.887},"min":{"1min":9.4719999999999995,"5min":9.4719999999999995,"15min":9.4719999999999995},"max":{"1min":23.526,"5min":27.632999999999999,"15min":27.632999999999999},"last":17.82},{"interface":"front","average":{"1min":15.352,"5min":16.143999999999998,"15min":17.184999999999999},"min":{"1min":8.2599999999999998,"5min":7.9269999999999996,"15min":7.9269999999999996},"max":{"1min":23.113,"5min":30.495000000000001,"15min":30.495000000000001},"last":16.196000000000002}]},{"osd":5,"last update":"Sat Sep 30 02:11:11 2023","interfaces":[{"interface":"back","average":{"1min":16.295999999999999,"5min":18.059000000000001,"15min":20.965},"min":{"1min":12.298,"5min":7.9080000000000004,"15min":7.9080000000000004},"max":{"1min":21.044,"5min":99.784000000000006,"15min":99.784000000000006},"last":13.747999999999999},{"interface":"front","average":{"1min":13.371,"5min":15.058,"15min":17.594999999999999},"min":{"1min":8.3030000000000008,"5min":7.609,"15min":7.609},"max":{"1min":19.103000000000002,"5min":98.634,"15min":98.634},"last":12.789999999999999}]},{"osd":6,"last update":"Sat Sep 30 02:11:11 2023","interfaces":[{"interface":"back","average":{"1min":16.995000000000001,"5min":16.553999999999998,"15min":16.952999999999999},"min":{"1min":7.8949999999999996,"5min":7.8949999999999996,"15min":7.8949999999999996},"max":{"1min":22.498999999999999,"5min":27.277999999999999,"15min":27.277999999999999},"last":18.670000000000002},{"interface":"front","average":{"1min":15.564,"5min":16.422000000000001,"15min":16.088999999999999},"min":{"1min":9.109,"5min":6.5810000000000004,"15min":6.5810000000000004},"max":{"1min":20.640999999999998,"5min":28.640000000000001,"15min":28.640000000000001},"last":11.715999999999999}]},{"osd":7,"last update":"Sat Sep 30 02:11:11 2023","interfaces":[{"interface":"back","average":{"1min":17.225000000000001,"5min":18.161000000000001,"15min":18.513000000000002},"min":{"1min":7.79,"5min":7.79,"15min":7.79},"max":{"1min":23.047000000000001,"5min":31.143000000000001,"15min":31.143000000000001},"last":25.411999999999999},{"interface":"front","average":{"1min":17.949000000000002,"5min":18.007000000000001,"15min":18.579000000000001},"min":{"1min":10.542999999999999,"5min":8.6509999999999998,"15min":8.6509999999999998},"max":{"1min":22.966999999999999,"5min":30.529,"15min":30.529},"last":15.587999999999999}]}]},{"osd":5,"up_from":15,"seq":64424509511,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5920,"kb_used_data":600,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":94365920,"statfs":{"total":96636764160,"available":96630702080,"internally_reserved":0,"allocated":614400,"data_stored":472880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:11:18 2023","interfaces":[{"interface":"back","average":{"1min":10.455,"5min":10.214,"15min":11.779999999999999},"min":{"1min":6.1040000000000001,"5min":5.7350000000000003,"15min":5.7350000000000003},"max":{"1min":19.498999999999999,"5min":22.452999999999999,"15min":22.452999999999999},"last":10.345000000000001},{"interface":"front","average":{"1min":10.673999999999999,"5min":10.722,"15min":11.548},"min":{"1min":6.6929999999999996,"5min":5.8879999999999999,"15min":5.8879999999999999},"max":{"1min":20.34,"5min":22.821999999999999,"15min":22.821999999999999},"last":7.9859999999999998}]},{"osd":1,"last update":"Sat Sep 30 02:11:18 2023","interfaces":[{"interface":"back","average":{"1min":14.335000000000001,"5min":13.829000000000001,"15min":15.194000000000001},"min":{"1min":8.9589999999999996,"5min":7.29,"15min":7.29},"max":{"1min":22.402999999999999,"5min":47.451999999999998,"15min":47.451999999999998},"last":16.48},{"interface":"front","average":{"1min":11.304,"5min":11.545,"15min":12.558},"min":{"1min":7.774,"5min":6.6130000000000004,"15min":6.6130000000000004},"max":{"1min":15.734999999999999,"5min":48.841999999999999,"15min":48.841999999999999},"last":8.7469999999999999}]},{"osd":2,"last update":"Sat Sep 30 02:11:18 2023","interfaces":[{"interface":"back","average":{"1min":15.868,"5min":13.461,"15min":11.835000000000001},"min":{"1min":8.2829999999999995,"5min":6.8250000000000002,"15min":6.8250000000000002},"max":{"1min":27.501000000000001,"5min":27.501000000000001,"15min":27.501000000000001},"last":14.368},{"interface":"front","average":{"1min":12.898999999999999,"5min":14.695,"15min":15.513},"min":{"1min":7.8150000000000004,"5min":7.5369999999999999,"15min":7.5369999999999999},"max":{"1min":19.411000000000001,"5min":26.576000000000001,"15min":26.576000000000001},"last":9.4489999999999998}]},{"osd":3,"last update":"Sat Sep 30 02:11:30 2023","interfaces":[{"interface":"back","average":{"1min":14.194000000000001,"5min":15.137,"15min":15.063000000000001},"min":{"1min":7.6980000000000004,"5min":7.2619999999999996,"15min":7.2619999999999996},"max":{"1min":21.891999999999999,"5min":26.895,"15min":26.895},"last":11.933},{"interface":"front","average":{"1min":15.656000000000001,"5min":17.158999999999999,"15min":17.841999999999999},"min":{"1min":11.603,"5min":11.140000000000001,"15min":11.140000000000001},"max":{"1min":19.952000000000002,"5min":24.702999999999999,"15min":24.702999999999999},"last":15.664}]},{"osd":4,"last update":"Sat Sep 30 02:11:18 2023","interfaces":[{"interface":"back","average":{"1min":15.975,"5min":17.355,"15min":18.113},"min":{"1min":10.063000000000001,"5min":8.8160000000000007,"15min":8.8160000000000007},"max":{"1min":22.620999999999999,"5min":26.710000000000001,"15min":26.710000000000001},"last":13.494},{"interface":"front","average":{"1min":16.137,"5min":16.138999999999999,"15min":15.467000000000001},"min":{"1min":10.363,"5min":6.0270000000000001,"15min":6.0270000000000001},"max":{"1min":23.376000000000001,"5min":28.344999999999999,"15min":28.344999999999999},"last":13.728}]},{"osd":6,"last update":"Sat Sep 30 02:11:18 2023","interfaces":[{"interface":"back","average":{"1min":17.373000000000001,"5min":16.178999999999998,"15min":14.978999999999999},"min":{"1min":10.907999999999999,"5min":7.0679999999999996,"15min":7.0679999999999996},"max":{"1min":22.890999999999998,"5min":23.573,"15min":23.573},"last":21.486000000000001},{"interface":"front","average":{"1min":14.722,"5min":16.218,"15min":16.617000000000001},"min":{"1min":7.8550000000000004,"5min":7.5540000000000003,"15min":7.5540000000000003},"max":{"1min":20.41,"5min":27.763999999999999,"15min":27.763999999999999},"last":22.797000000000001}]},{"osd":7,"last update":"Sat Sep 30 02:11:18 2023","interfaces":[{"interface":"back","average":{"1min":17.943999999999999,"5min":18.265999999999998,"15min":19.606000000000002},"min":{"1min":12.4,"5min":10.584,"15min":10.584},"max":{"1min":22.768000000000001,"5min":28.951000000000001,"15min":28.951000000000001},"last":17.518999999999998},{"interface":"front","average":{"1min":17.033999999999999,"5min":16.844000000000001,"15min":16.053999999999998},"min":{"1min":11.109,"5min":7.0490000000000004,"15min":7.0490000000000004},"max":{"1min":21.635999999999999,"5min":25.151,"15min":25.151},"last":11.602}]}]},{"osd":2,"up_from":18,"seq":77309411397,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5208,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366632,"statfs":{"total":96636764160,"available":96631431168,"internally_reserved":0,"allocated":212992,"data_stored":75040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:11:21 2023","interfaces":[{"interface":"back","average":{"1min":12.927,"5min":13.202999999999999,"15min":13.711},"min":{"1min":7.1929999999999996,"5min":6.665,"15min":6.665},"max":{"1min":20.434000000000001,"5min":69.003,"15min":69.003},"last":11.837},{"interface":"front","average":{"1min":13.492000000000001,"5min":14.566000000000001,"15min":16.558},"min":{"1min":8.6880000000000006,"5min":8.1329999999999991,"15min":8.1329999999999991},"max":{"1min":25.254000000000001,"5min":91.331999999999994,"15min":91.331999999999994},"last":7.3710000000000004}]},{"osd":1,"last update":"Sat Sep 30 02:11:21 2023","interfaces":[{"interface":"back","average":{"1min":14.648999999999999,"5min":16.484999999999999,"15min":20.882000000000001},"min":{"1min":9.4800000000000004,"5min":7.9630000000000001,"15min":7.9630000000000001},"max":{"1min":21.640999999999998,"5min":145.952,"15min":145.952},"last":20.225999999999999},{"interface":"front","average":{"1min":15.581,"5min":17.215,"15min":21.452000000000002},"min":{"1min":9.6679999999999993,"5min":8.7469999999999999,"15min":8.7469999999999999},"max":{"1min":25.678000000000001,"5min":155.72900000000001,"15min":155.72900000000001},"last":13.598000000000001}]},{"osd":3,"last update":"Sat Sep 30 02:11:21 2023","interfaces":[{"interface":"back","average":{"1min":16.968,"5min":16.724,"15min":17.524000000000001},"min":{"1min":10.601000000000001,"5min":10.124000000000001,"15min":10.124000000000001},"max":{"1min":32.027000000000001,"5min":39.378999999999998,"15min":39.378999999999998},"last":14.952},{"interface":"front","average":{"1min":17.798999999999999,"5min":18.437000000000001,"15min":18.405000000000001},"min":{"1min":8.9250000000000007,"5min":8.9250000000000007,"15min":8.9250000000000007},"max":{"1min":24.704999999999998,"5min":36.649000000000001,"15min":36.649000000000001},"last":15.609}]},{"osd":4,"last update":"Sat Sep 30 02:11:21 2023","interfaces":[{"interface":"back","average":{"1min":18.395,"5min":18.151,"15min":18.896999999999998},"min":{"1min":11.792999999999999,"5min":8.9359999999999999,"15min":8.9359999999999999},"max":{"1min":29.699000000000002,"5min":39.186999999999998,"15min":39.186999999999998},"last":17.797000000000001},{"interface":"front","average":{"1min":11.427,"5min":11.253,"15min":12.537000000000001},"min":{"1min":6.7329999999999997,"5min":6.7329999999999997,"15min":6.7329999999999997},"max":{"1min":23.036999999999999,"5min":31.32,"15min":31.32},"last":9.5760000000000005}]},{"osd":5,"last update":"Sat Sep 30 02:11:21 2023","interfaces":[{"interface":"back","average":{"1min":19.495000000000001,"5min":18.518999999999998,"15min":19.271000000000001},"min":{"1min":12.682,"5min":8.3499999999999996,"15min":8.3499999999999996},"max":{"1min":26.695,"5min":43.409999999999997,"15min":43.409999999999997},"last":17.169},{"interface":"front","average":{"1min":20.788,"5min":21.66,"15min":22.609999999999999},"min":{"1min":11.974,"5min":11.974,"15min":11.974},"max":{"1min":33.695999999999998,"5min":40.646000000000001,"15min":40.646000000000001},"last":21.327999999999999}]},{"osd":6,"last update":"Sat Sep 30 02:11:21 2023","interfaces":[{"interface":"back","average":{"1min":13.474,"5min":14.634,"15min":15.917999999999999},"min":{"1min":9.8520000000000003,"5min":7.4829999999999997,"15min":7.4829999999999997},"max":{"1min":21.18,"5min":34.045000000000002,"15min":34.045000000000002},"last":10.582000000000001},{"interface":"front","average":{"1min":19.893999999999998,"5min":19.949000000000002,"15min":21.067},"min":{"1min":12.353999999999999,"5min":9.1600000000000001,"15min":9.1600000000000001},"max":{"1min":24.425000000000001,"5min":47.997999999999998,"15min":47.997999999999998},"last":19.106000000000002}]},{"osd":7,"last update":"Sat Sep 30 02:11:21 2023","interfaces":[{"interface":"back","average":{"1min":21.699999999999999,"5min":22.599,"15min":24.295000000000002},"min":{"1min":15.781000000000001,"5min":13.542999999999999,"15min":13.542999999999999},"max":{"1min":31.061,"5min":44.585000000000001,"15min":44.585000000000001},"last":22.02},{"interface":"front","average":{"1min":15.755000000000001,"5min":16.690999999999999,"15min":17.437000000000001},"min":{"1min":8.8230000000000004,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":26.234000000000002,"5min":38.295000000000002,"15min":38.295000000000002},"last":12.65}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T02:11:43.086 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-09-30T02:06:15.800912+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=2, tm_min=6, tm_sec=15, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=2, tm_min=11, tm_sec=33, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T02:11:43.086 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-09-30T02:06:15.800912+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=2, tm_min=6, tm_sec=15, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=2, tm_min=11, tm_sec=33, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T02:11:43.086 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-09-30T02:06:15.800912+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=2, tm_min=6, tm_sec=15, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=2, tm_min=11, tm_sec=33, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T02:11:43.086 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-09-30T02:06:15.800912+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=2, tm_min=6, tm_sec=15, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=2, tm_min=11, tm_sec=33, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T02:11:43.086 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-09-30T02:06:15.800912+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=2, tm_min=6, tm_sec=15, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=2, tm_min=11, tm_sec=33, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T02:11:43.087 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-09-30T02:06:15.800912+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=2, tm_min=6, tm_sec=15, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=2, tm_min=11, tm_sec=33, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T02:11:43.087 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-09-30T02:06:15.800912+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=2, tm_min=6, tm_sec=15, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=2, tm_min=11, tm_sec=33, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T02:11:43.087 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-09-30T02:06:15.800912+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=2, tm_min=6, tm_sec=15, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=2, tm_min=11, tm_sec=33, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T02:11:43.087 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-30T02:11:54.589 INFO:tasks.ceph.mon.a.smithi059.stderr:==00:00:07:04.454 116256== Warning: unimplemented fcntl command: 1036 2023-09-30T02:11:54.591 INFO:tasks.ceph.mon.a.smithi059.stderr:==00:00:07:04.457 116256== Warning: unimplemented fcntl command: 1036 2023-09-30T02:11:54.617 INFO:tasks.ceph.mon.a.smithi059.stderr:==00:00:07:04.483 116256== Warning: unimplemented fcntl command: 1036 2023-09-30T02:11:54.648 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:07:04.512 116257== Warning: unimplemented fcntl command: 1036 2023-09-30T02:11:54.648 INFO:tasks.ceph.mon.b.smithi110.stderr:==00:00:07:04.495 115869== Warning: unimplemented fcntl command: 1036 2023-09-30T02:11:54.649 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:07:04.514 116257== Warning: unimplemented fcntl command: 1036 2023-09-30T02:11:54.650 INFO:tasks.ceph.mon.b.smithi110.stderr:==00:00:07:04.498 115869== Warning: unimplemented fcntl command: 1036 2023-09-30T02:11:54.675 INFO:tasks.ceph.mon.b.smithi110.stderr:==00:00:07:04.522 115869== Warning: unimplemented fcntl command: 1036 2023-09-30T02:11:54.676 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:07:04.541 116257== Warning: unimplemented fcntl command: 1036 2023-09-30T02:12:03.088 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T02:12:03.232 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.231+0000 7f8a7ebc1700 10 monclient: get_monmap_and_config 2023-09-30T02:12:03.232 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.231+0000 7f8a7ebc1700 10 monclient: build_initial_monmap 2023-09-30T02:12:03.232 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.231+0000 7f8a7ebc1700 10 monclient: monmap: 2023-09-30T02:12:03.232 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:12:03.232 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:12:03.232 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:12:03.232980+0000 2023-09-30T02:12:03.232 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:12:03.232980+0000 2023-09-30T02:12:03.232 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:12:03.232 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:12:03.233 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:12:03.233 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:12:03.233 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:12:03.233 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:12:03.233 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:12:03.233 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:12:03.236 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.232+0000 7f8a7ebc1700 1 Processor -- start 2023-09-30T02:12:03.236 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.232+0000 7f8a7ebc1700 1 -- start start 2023-09-30T02:12:03.236 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.232+0000 7f8a7ebc1700 10 monclient: init 2023-09-30T02:12:03.236 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.233+0000 7f8a7ebc1700 10 monclient: _reopen_session rank -1 2023-09-30T02:12:03.236 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.233+0000 7f8a7ebc1700 10 monclient: _add_conns ranks=[2,4,0,3,1] 2023-09-30T02:12:03.236 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.233+0000 7f8a7ebc1700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8a780a5b70 0x7f8a780b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:12:03.236 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.233+0000 7f8a7ebc1700 10 monclient(hunting): picked mon.noname-b con 0x7f8a780a5b70 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:12:03.237 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.233+0000 7f8a7ebc1700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f8a780a5f40 0x7f8a78141b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:12:03.237 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.233+0000 7f8a7ebc1700 10 monclient(hunting): picked mon.noname-c con 0x7f8a780a5f40 addr v2:172.21.15.110:3300/0 2023-09-30T02:12:03.237 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.233+0000 7f8a7ebc1700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8a781420c0 addr v1:172.21.15.59:6789/0 2023-09-30T02:12:03.237 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.233+0000 7f8a7ebc1700 1 -- --> v1:172.21.15.59:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a780a63b0 con 0x7f8a781420c0 2023-09-30T02:12:03.237 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.233+0000 7f8a7ebc1700 10 monclient(hunting): start opening mon connection 2023-09-30T02:12:03.237 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.233+0000 7f8a7ebc1700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a780a75a0 con 0x7f8a780a5f40 2023-09-30T02:12:03.237 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.233+0000 7f8a7ebc1700 10 monclient(hunting): start opening mon connection 2023-09-30T02:12:03.237 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.233+0000 7f8a7ebc1700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a780af3f0 con 0x7f8a780a5b70 2023-09-30T02:12:03.237 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.233+0000 7f8a7ebc1700 10 monclient(hunting): _renew_subs 2023-09-30T02:12:03.237 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.233+0000 7f8a7ebc1700 10 monclient(hunting): authenticate will time out at 2168.536282s 2023-09-30T02:12:03.237 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.233+0000 7f8a7d15e700 1 --1- >> v1:172.21.15.59:6789/0 conn(0x7f8a781420c0 0x7f8a7813e230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.59:6789/0 says I am v1:172.21.15.59:35882/0 (socket says 172.21.15.59:35882) 2023-09-30T02:12:03.237 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.233+0000 7f8a7d15e700 1 -- 172.21.15.59:0/111599280 learned_addr learned my addr 172.21.15.59:0/111599280 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:12:03.238 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.233+0000 7f8a7c95d700 1 --2- 172.21.15.59:0/111599280 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8a780a5b70 0x7f8a780b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:12:03.238 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.234+0000 7f8a77fff700 1 --2- 172.21.15.59:0/111599280 >> v2:172.21.15.110:3300/0 conn(0x7f8a780a5f40 0x7f8a78141b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:12:03.238 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.234+0000 7f8a7c95d700 10 monclient(hunting): get_auth_request con 0x7f8a780a5b70 auth_method 0 2023-09-30T02:12:03.238 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.234+0000 7f8a7c95d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:12:03.238 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.234+0000 7f8a7c95d700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:12:03.238 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.234+0000 7f8a7c95d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:12:03.238 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.234+0000 7f8a77fff700 10 monclient(hunting): get_auth_request con 0x7f8a780a5f40 auth_method 0 2023-09-30T02:12:03.238 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.234+0000 7f8a77fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:12:03.238 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.234+0000 7f8a77fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:12:03.238 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.234+0000 7f8a77fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:12:03.238 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.235+0000 7f8a7c95d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:12:03.238 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.235+0000 7f8a7c95d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:12:03.239 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.235+0000 7f8a7c95d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:12:03.239 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.235+0000 7f8a77fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:12:03.239 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.235+0000 7f8a77fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:12:03.239 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.235+0000 7f8a77fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:12:03.240 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.237+0000 7f8a777fe700 1 -- 172.21.15.59:0/111599280 <== mon.0 v1:172.21.15.59:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1939793349 0 0) 0x7f8a780a63b0 con 0x7f8a781420c0 2023-09-30T02:12:03.240 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.237+0000 7f8a777fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:12:03.240 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.237+0000 7f8a777fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:12:03.240 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.237+0000 7f8a777fe700 10 monclient(hunting): my global_id is 4935 2023-09-30T02:12:03.240 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.237+0000 7f8a777fe700 1 -- 172.21.15.59:0/111599280 --> v1:172.21.15.59:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a68003570 con 0x7f8a781420c0 2023-09-30T02:12:03.240 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.238+0000 7f8a7c95d700 10 monclient(hunting): handle_auth_done global_id 4835 payload 274 2023-09-30T02:12:03.240 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.238+0000 7f8a7c95d700 1 -- 172.21.15.59:0/111599280 >> v2:172.21.15.110:3300/0 conn(0x7f8a780a5f40 msgr2=0x7f8a78141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:12:03.240 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.238+0000 7f8a7c95d700 1 --2- 172.21.15.59:0/111599280 >> v2:172.21.15.110:3300/0 conn(0x7f8a780a5f40 0x7f8a78141b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:12:03.240 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.238+0000 7f8a7c95d700 1 -- 172.21.15.59:0/111599280 >> v1:172.21.15.59:6789/0 conn(0x7f8a781420c0 legacy=0x7f8a7813e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:12:03.240 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.238+0000 7f8a7c95d700 10 monclient: _finish_hunting 0 2023-09-30T02:12:03.241 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.238+0000 7f8a7c95d700 1 monclient: found mon.noname-b 2023-09-30T02:12:03.241 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.238+0000 7f8a7c95d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:12:03.241 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.238+0000 7f8a7c95d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.59:3301/0 2023-09-30T02:12:03.241 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.238+0000 7f8a7c95d700 1 -- 172.21.15.59:0/111599280 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a78142500 con 0x7f8a780a5b70 2023-09-30T02:12:03.241 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.240+0000 7f8a7c95d700 1 --2- 172.21.15.59:0/111599280 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8a780a5b70 0x7f8a780b1500 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f8a6c0098e0 tx=0x7f8a6c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3284272aba3f0a8c server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:12:03.245 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.244+0000 7f8a777fe700 1 -- 172.21.15.59:0/111599280 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a6c011630 con 0x7f8a780a5b70 2023-09-30T02:12:03.245 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.245+0000 7f8a777fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:12:03.245 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.245+0000 7f8a777fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:12:03.245 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.245+0000 7f8a777fe700 10 monclient: dump: 2023-09-30T02:12:03.245 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:12:03.245 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:12:03.245 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:12:03.246 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:12:03.246 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:12:03.246 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:12:03.246 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:12:03.246 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:12:03.246 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:12:03.246 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:12:03.246 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.245+0000 7f8a777fe700 10 monclient: _finish_auth 0 2023-09-30T02:12:03.246 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.245+0000 7f8a777fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:12:03.246 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.245+0000 7f8a777fe700 1 -- 172.21.15.59:0/111599280 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a6c011790 con 0x7f8a780a5b70 2023-09-30T02:12:03.246 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.245+0000 7f8a7ebc1700 5 monclient: authenticate success, global_id 4835 2023-09-30T02:12:03.246 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.245+0000 7f8a7ebc1700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:12:03.247 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.245+0000 7f8a777fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:12:03.247 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.245+0000 7f8a7ebc1700 10 monclient: get_monmap_and_config success 2023-09-30T02:12:03.247 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.245+0000 7f8a777fe700 1 -- 172.21.15.59:0/111599280 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a6c011b30 con 0x7f8a780a5b70 2023-09-30T02:12:03.247 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.245+0000 7f8a7ebc1700 4 set_mon_vals no callback set 2023-09-30T02:12:03.247 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.245+0000 7f8a777fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:12:03.247 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.245+0000 7f8a777fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:12:03.247 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.245+0000 7f8a777fe700 10 monclient: dump: 2023-09-30T02:12:03.247 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:12:03.247 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:12:03.247 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:12:03.247 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:12:03.247 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:12:03.248 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:12:03.248 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:12:03.248 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:12:03.248 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:12:03.248 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:12:03.251 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.245+0000 7f8a7ebc1700 10 monclient: shutdown 2023-09-30T02:12:03.251 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.245+0000 7f8a7ebc1700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:12:03.251 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.245+0000 7f8a7ebc1700 1 -- 172.21.15.59:0/111599280 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8a780a5b70 msgr2=0x7f8a780b1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:12:03.251 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.245+0000 7f8a7ebc1700 1 --2- 172.21.15.59:0/111599280 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8a780a5b70 0x7f8a780b1500 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f8a6c0098e0 tx=0x7f8a6c0091a0 comp rx=0 tx=0).stop 2023-09-30T02:12:03.251 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.245+0000 7f8a7ebc1700 1 -- 172.21.15.59:0/111599280 shutdown_connections 2023-09-30T02:12:03.251 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.245+0000 7f8a7ebc1700 1 --2- 172.21.15.59:0/111599280 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8a780a5b70 0x7f8a780b1500 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:12:03.251 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.245+0000 7f8a7ebc1700 1 --2- 172.21.15.59:0/111599280 >> v2:172.21.15.110:3300/0 conn(0x7f8a780a5f40 0x7f8a78141b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:12:03.252 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.245+0000 7f8a7ebc1700 1 -- 172.21.15.59:0/111599280 >> 172.21.15.59:0/111599280 conn(0x7f8a780631f0 msgr2=0x7f8a780635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:12:03.252 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.245+0000 7f8a7ebc1700 1 -- 172.21.15.59:0/111599280 shutdown_connections 2023-09-30T02:12:03.252 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.245+0000 7f8a7ebc1700 1 -- 172.21.15.59:0/111599280 wait complete. 2023-09-30T02:12:03.252 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.246+0000 7f8a7ebc1700 10 monclient: build_initial_monmap 2023-09-30T02:12:03.252 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.246+0000 7f8a7ebc1700 10 monclient: monmap: 2023-09-30T02:12:03.252 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:12:03.252 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:12:03.252 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:12:03.252 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:12:03.252 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:12:03.252 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:12:03.252 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:12:03.252 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:12:03.253 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:12:03.253 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:12:03.253 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.247+0000 7f8a7ebc1700 1 librados: starting msgr at 2023-09-30T02:12:03.253 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.247+0000 7f8a7ebc1700 1 librados: starting objecter 2023-09-30T02:12:03.253 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.247+0000 7f8a7ebc1700 1 Processor -- start 2023-09-30T02:12:03.253 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.247+0000 7f8a7ebc1700 1 -- start start 2023-09-30T02:12:03.253 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.247+0000 7f8a7ebc1700 1 librados: setting wanted keys 2023-09-30T02:12:03.253 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.247+0000 7f8a7ebc1700 1 librados: calling monclient init 2023-09-30T02:12:03.253 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.247+0000 7f8a7ebc1700 10 monclient: init 2023-09-30T02:12:03.253 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.248+0000 7f8a7ebc1700 10 monclient: _reopen_session rank -1 2023-09-30T02:12:03.253 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.248+0000 7f8a7ebc1700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T02:12:03.253 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.248+0000 7f8a7ebc1700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8a780a5b70 0x7f8a780a9e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:12:03.254 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.248+0000 7f8a7ebc1700 10 monclient(hunting): picked mon.noname-b con 0x7f8a780a5b70 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:12:03.254 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.248+0000 7f8a7ebc1700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8a780a5f40 0x7f8a7814de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:12:03.254 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.248+0000 7f8a7ebc1700 10 monclient(hunting): picked mon.noname-c con 0x7f8a780a5f40 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:12:03.254 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.248+0000 7f8a7ebc1700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a781420c0 0x7f8a780ae080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:12:03.254 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.248+0000 7f8a7ebc1700 10 monclient(hunting): picked mon.noname-a con 0x7f8a781420c0 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:12:03.254 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.248+0000 7f8a7ebc1700 10 monclient(hunting): start opening mon connection 2023-09-30T02:12:03.254 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.248+0000 7f8a7ebc1700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a780aa540 con 0x7f8a781420c0 2023-09-30T02:12:03.254 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.248+0000 7f8a7ebc1700 10 monclient(hunting): start opening mon connection 2023-09-30T02:12:03.254 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.248+0000 7f8a7ebc1700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a78137ea0 con 0x7f8a780a5b70 2023-09-30T02:12:03.254 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.248+0000 7f8a7ebc1700 10 monclient(hunting): start opening mon connection 2023-09-30T02:12:03.254 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.248+0000 7f8a7ebc1700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a7814e380 con 0x7f8a780a5f40 2023-09-30T02:12:03.254 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.248+0000 7f8a7ebc1700 10 monclient(hunting): _renew_subs 2023-09-30T02:12:03.255 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.248+0000 7f8a7ebc1700 10 monclient(hunting): authenticate will time out at 2168.551548s 2023-09-30T02:12:03.255 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.249+0000 7f8a77fff700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8a780a5f40 0x7f8a7814de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:12:03.255 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.249+0000 7f8a7d15e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a781420c0 0x7f8a780ae080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:12:03.255 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.249+0000 7f8a7c95d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8a780a5b70 0x7f8a780a9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:12:03.255 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.249+0000 7f8a77fff700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8a780a5f40 0x7f8a7814de40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3301/0 says I am v2:172.21.15.59:60774/0 (socket says 172.21.15.59:60774) 2023-09-30T02:12:03.255 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.249+0000 7f8a77fff700 1 -- 172.21.15.59:0/4024408918 learned_addr learned my addr 172.21.15.59:0/4024408918 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:12:03.255 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.249+0000 7f8a77fff700 10 monclient(hunting): get_auth_request con 0x7f8a780a5f40 auth_method 0 2023-09-30T02:12:03.255 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.249+0000 7f8a77fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:12:03.255 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.249+0000 7f8a77fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:12:03.255 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.249+0000 7f8a77fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:12:03.255 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.249+0000 7f8a7d15e700 10 monclient(hunting): get_auth_request con 0x7f8a781420c0 auth_method 0 2023-09-30T02:12:03.255 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.249+0000 7f8a7d15e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:12:03.255 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.249+0000 7f8a7d15e700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:12:03.256 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.249+0000 7f8a7d15e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:12:03.256 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.249+0000 7f8a7c95d700 10 monclient(hunting): get_auth_request con 0x7f8a780a5b70 auth_method 0 2023-09-30T02:12:03.256 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.249+0000 7f8a7c95d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:12:03.256 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.249+0000 7f8a7c95d700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:12:03.256 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.249+0000 7f8a7c95d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:12:03.256 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.250+0000 7f8a77fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:12:03.256 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.250+0000 7f8a77fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:12:03.256 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.250+0000 7f8a77fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:12:03.256 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.250+0000 7f8a7d15e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:12:03.256 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.250+0000 7f8a7d15e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:12:03.256 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.250+0000 7f8a7d15e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:12:03.256 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.250+0000 7f8a7c95d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:12:03.256 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.250+0000 7f8a7c95d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:12:03.257 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.250+0000 7f8a7c95d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:12:03.257 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.254+0000 7f8a77fff700 10 monclient(hunting): handle_auth_done global_id 4838 payload 824 2023-09-30T02:12:03.257 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.254+0000 7f8a77fff700 1 -- 172.21.15.59:0/4024408918 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8a780a5b70 msgr2=0x7f8a780a9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:12:03.257 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.254+0000 7f8a77fff700 1 --2- 172.21.15.59:0/4024408918 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8a780a5b70 0x7f8a780a9e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:12:03.258 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.254+0000 7f8a77fff700 1 -- 172.21.15.59:0/4024408918 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a781420c0 msgr2=0x7f8a780ae080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:12:03.258 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.254+0000 7f8a77fff700 1 --2- 172.21.15.59:0/4024408918 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a781420c0 0x7f8a780ae080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:12:03.258 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.254+0000 7f8a77fff700 10 monclient: _finish_hunting 0 2023-09-30T02:12:03.258 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.254+0000 7f8a77fff700 1 monclient: found mon.noname-c 2023-09-30T02:12:03.258 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.254+0000 7f8a77fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:12:03.258 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.254+0000 7f8a77fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.59:3301/0 2023-09-30T02:12:03.258 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.254+0000 7f8a77fff700 1 -- 172.21.15.59:0/4024408918 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a6c009420 con 0x7f8a780a5f40 2023-09-30T02:12:03.258 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.257+0000 7f8a77fff700 1 --2- 172.21.15.59:0/4024408918 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8a780a5f40 0x7f8a7814de40 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f8a6400ec80 tx=0x7f8a640091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:12:03.258 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.257+0000 7f8a74ff9700 10 client.?.objecter ms_handle_connect 0x7f8a780a5f40 2023-09-30T02:12:03.258 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.257+0000 7f8a74ff9700 10 client.?.objecter resend_mon_ops 2023-09-30T02:12:03.261 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.261+0000 7f8a74ff9700 1 -- 172.21.15.59:0/4024408918 <== mon.2 v2:172.21.15.59:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a6400c850 con 0x7f8a780a5f40 2023-09-30T02:12:03.262 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.261+0000 7f8a74ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:12:03.262 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.261+0000 7f8a74ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:12:03.262 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.261+0000 7f8a74ff9700 10 monclient: dump: 2023-09-30T02:12:03.262 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:12:03.262 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:12:03.262 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:12:03.262 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:12:03.262 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:12:03.262 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:12:03.262 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:12:03.263 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:12:03.263 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:12:03.263 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:12:03.263 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.261+0000 7f8a74ff9700 10 monclient: _finish_auth 0 2023-09-30T02:12:03.263 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.261+0000 7f8a74ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:12:03.263 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.261+0000 7f8a74ff9700 1 -- 172.21.15.59:0/4024408918 <== mon.2 v2:172.21.15.59:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a6400c9b0 con 0x7f8a780a5f40 2023-09-30T02:12:03.263 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.261+0000 7f8a74ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:12:03.263 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.261+0000 7f8a74ff9700 1 -- 172.21.15.59:0/4024408918 <== mon.2 v2:172.21.15.59:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a6400cd50 con 0x7f8a780a5f40 2023-09-30T02:12:03.263 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.261+0000 7f8a7ebc1700 5 monclient: authenticate success, global_id 4838 2023-09-30T02:12:03.263 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.261+0000 7f8a7ebc1700 10 monclient: _renew_subs 2023-09-30T02:12:03.263 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.261+0000 7f8a7ebc1700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:12:03.263 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.261+0000 7f8a7ebc1700 1 -- 172.21.15.59:0/4024408918 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a780a9250 con 0x7f8a780a5f40 2023-09-30T02:12:03.264 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.261+0000 7f8a74ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:12:03.264 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.261+0000 7f8a74ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T02:12:03.264 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.261+0000 7f8a76ffd700 4 set_mon_vals no callback set 2023-09-30T02:12:03.264 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.261+0000 7f8a74ff9700 10 monclient: dump: 2023-09-30T02:12:03.264 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:12:03.264 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:12:03.264 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:12:03.264 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:12:03.264 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:12:03.264 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:12:03.264 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:12:03.264 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:12:03.264 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:12:03.265 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:12:03.265 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.261+0000 7f8a7ebc1700 10 client.4838.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:12:03.265 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.261+0000 7f8a7ebc1700 10 monclient: _renew_subs 2023-09-30T02:12:03.265 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.261+0000 7f8a7ebc1700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:12:03.265 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.261+0000 7f8a7ebc1700 1 -- 172.21.15.59:0/4024408918 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a7814eaf0 con 0x7f8a780a5f40 2023-09-30T02:12:03.265 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.261+0000 7f8a7ebc1700 1 librados: init done 2023-09-30T02:12:03.265 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.262+0000 7f8a7ebc1700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:12:03.266 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.262+0000 7f8a7ebc1700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:12:03.266 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.262+0000 7f8a7ebc1700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.59:3301/0 2023-09-30T02:12:03.266 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.262+0000 7f8a7ebc1700 1 -- 172.21.15.59:0/4024408918 --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a780a7280 con 0x7f8a780a5f40 2023-09-30T02:12:03.282 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.282+0000 7f8a74ff9700 1 -- 172.21.15.59:0/4024408918 <== mon.2 v2:172.21.15.59:3301/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f8a64013680 con 0x7f8a780a5f40 2023-09-30T02:12:03.283 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.283+0000 7f8a74ff9700 1 --2- 172.21.15.59:0/4024408918 >> v2:172.21.15.110:6800/115870 conn(0x7f8a5c068960 0x7f8a5c06ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:12:03.283 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.283+0000 7f8a7c95d700 1 --2- 172.21.15.59:0/4024408918 >> v2:172.21.15.110:6800/115870 conn(0x7f8a5c068960 0x7f8a5c06ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:12:03.283 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.283+0000 7f8a7c95d700 10 monclient: get_auth_request con 0x7f8a5c068960 auth_method 0 2023-09-30T02:12:03.284 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.283+0000 7f8a7c95d700 1 --2- 172.21.15.59:0/4024408918 >> v2:172.21.15.110:6800/115870 conn(0x7f8a5c068960 0x7f8a5c06ae10 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f8a6c0093f0 tx=0x7f8a6c017000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:12:03.284 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.284+0000 7f8a74ff9700 10 client.4838.objecter ms_handle_connect 0x7f8a5c068960 2023-09-30T02:12:03.447 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.446+0000 7f8a74ff9700 1 -- 172.21.15.59:0/4024408918 <== mon.2 v2:172.21.15.59:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f8a64056580 con 0x7f8a780a5f40 2023-09-30T02:12:03.447 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.446+0000 7f8a74ff9700 10 client.4838.objecter ms_dispatch 0x7f8a78001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:12:03.447 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.446+0000 7f8a74ff9700 3 client.4838.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:12:03.447 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.446+0000 7f8a74ff9700 3 client.4838.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:12:03.447 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.446+0000 7f8a74ff9700 20 client.4838.objecter dump_active .. 0 homeless 2023-09-30T02:12:03.573 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.573+0000 7f8a74ff9700 1 -- 172.21.15.59:0/4024408918 <== mon.2 v2:172.21.15.59:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8a640521c0 con 0x7f8a780a5f40 2023-09-30T02:12:03.573 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.573+0000 7f8a74ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:12:03.573 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.573+0000 7f8a74ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:12:03.781 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.780+0000 7f8a7ebc1700 1 -- 172.21.15.59:0/4024408918 --> v2:172.21.15.110:6800/115870 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8a7806f530 con 0x7f8a5c068960 2023-09-30T02:12:03.784 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.784+0000 7f8a74ff9700 1 -- 172.21.15.59:0/4024408918 <== mgr.4108 v2:172.21.15.110:6800/115870 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+70518 (secure 0 0 0) 0x7f8a7806f530 con 0x7f8a5c068960 2023-09-30T02:12:03.785 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:12:03.787 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.786+0000 7f8a7ebc1700 10 librados: watch_flush enter 2023-09-30T02:12:03.787 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.786+0000 7f8a7ebc1700 10 librados: watch_flush exit 2023-09-30T02:12:03.787 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.786+0000 7f8a7ebc1700 20 client.4838.objecter shutdown clearing up homeless session... 2023-09-30T02:12:03.788 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.786+0000 7f8a7ebc1700 10 client.4838.objecter successfully canceled tick 2023-09-30T02:12:03.788 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.786+0000 7f8a7ebc1700 1 -- 172.21.15.59:0/4024408918 >> v2:172.21.15.110:6800/115870 conn(0x7f8a5c068960 msgr2=0x7f8a5c06ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:12:03.788 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.786+0000 7f8a7ebc1700 1 --2- 172.21.15.59:0/4024408918 >> v2:172.21.15.110:6800/115870 conn(0x7f8a5c068960 0x7f8a5c06ae10 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f8a6c0093f0 tx=0x7f8a6c017000 comp rx=0 tx=0).stop 2023-09-30T02:12:03.788 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.786+0000 7f8a7ebc1700 10 monclient: shutdown 2023-09-30T02:12:03.788 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.786+0000 7f8a7ebc1700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:12:03.788 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.786+0000 7f8a7ebc1700 1 -- 172.21.15.59:0/4024408918 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8a780a5f40 msgr2=0x7f8a7814de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:12:03.788 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.786+0000 7f8a7ebc1700 1 --2- 172.21.15.59:0/4024408918 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8a780a5f40 0x7f8a7814de40 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f8a6400ec80 tx=0x7f8a640091a0 comp rx=0 tx=0).stop 2023-09-30T02:12:03.788 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.786+0000 7f8a7ebc1700 1 -- 172.21.15.59:0/4024408918 shutdown_connections 2023-09-30T02:12:03.788 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.786+0000 7f8a7ebc1700 1 --2- 172.21.15.59:0/4024408918 >> v2:172.21.15.110:6800/115870 conn(0x7f8a5c068960 0x7f8a5c06ae10 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:12:03.788 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.786+0000 7f8a7ebc1700 1 --2- 172.21.15.59:0/4024408918 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f8a780a5b70 0x7f8a780a9e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:12:03.789 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.786+0000 7f8a7ebc1700 1 --2- 172.21.15.59:0/4024408918 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8a781420c0 0x7f8a780ae080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:12:03.789 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.786+0000 7f8a7ebc1700 1 --2- 172.21.15.59:0/4024408918 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f8a780a5f40 0x7f8a7814de40 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:12:03.789 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.786+0000 7f8a7ebc1700 1 -- 172.21.15.59:0/4024408918 >> 172.21.15.59:0/4024408918 conn(0x7f8a780631f0 msgr2=0x7f8a780635c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:12:03.789 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.787+0000 7f8a7ebc1700 1 -- 172.21.15.59:0/4024408918 shutdown_connections 2023-09-30T02:12:03.789 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.787+0000 7f8a7ebc1700 1 -- 172.21.15.59:0/4024408918 wait complete. 2023-09-30T02:12:03.789 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:03.787+0000 7f8a7ebc1700 1 librados: shutdown 2023-09-30T02:12:03.789 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2023-09-30T02:12:03.804 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":210,"stamp":"2023-09-30T02:12:03.654247+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":43088,"kb_used_data":2448,"kb_used_omap":0,"kb_used_meta":40576,"kb_avail":754931632,"statfs":{"total":773094113280,"available":773049991168,"internally_reserved":0,"allocated":2506752,"data_stored":1396000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":41549824},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":48,"apply_latency_ms":48,"commit_latency_ns":48000000,"apply_latency_ns":48000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002533"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-09-30T02:06:19.722714+0000","last_change":"2023-09-30T02:06:18.210390+0000","last_active":"2023-09-30T02:06:19.722714+0000","last_peered":"2023-09-30T02:06:19.722714+0000","last_clean":"2023-09-30T02:06:19.722714+0000","last_became_active":"2023-09-30T02:06:18.198153+0000","last_became_peered":"2023-09-30T02:06:18.198153+0000","last_unstale":"2023-09-30T02:06:19.722714+0000","last_undegraded":"2023-09-30T02:06:19.722714+0000","last_fullsized":"2023-09-30T02:06:19.722714+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:06:15.800912+0000","last_clean_scrub_stamp":"2023-09-30T02:06:15.800912+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-09-30T02:11:42.499140+0000","last_change":"2023-09-30T02:11:42.483742+0000","last_active":"2023-09-30T02:11:42.499140+0000","last_peered":"2023-09-30T02:11:42.499140+0000","last_clean":"2023-09-30T02:11:42.499140+0000","last_became_active":"2023-09-30T02:06:18.224464+0000","last_became_peered":"2023-09-30T02:06:18.224464+0000","last_unstale":"2023-09-30T02:11:42.499140+0000","last_undegraded":"2023-09-30T02:11:42.499140+0000","last_fullsized":"2023-09-30T02:11:42.499140+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:11:42.476968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:11:42.476968+0000","last_clean_scrub_stamp":"2023-09-30T02:11:42.476968+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:41:19.177320+0000","scrub_duration":0.50626253899999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-09-30T02:11:57.649362+0000","last_change":"2023-09-30T02:11:57.648560+0000","last_active":"2023-09-30T02:11:57.649362+0000","last_peered":"2023-09-30T02:11:57.649362+0000","last_clean":"2023-09-30T02:11:57.649362+0000","last_became_active":"2023-09-30T02:06:17.855689+0000","last_became_peered":"2023-09-30T02:06:17.855689+0000","last_unstale":"2023-09-30T02:11:57.649362+0000","last_undegraded":"2023-09-30T02:11:57.649362+0000","last_fullsized":"2023-09-30T02:11:57.649362+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:11:57.645782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:11:57.645782+0000","last_clean_scrub_stamp":"2023-09-30T02:11:57.645782+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:31:54.616937+0000","scrub_duration":0.091109921999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-09-30T02:11:45.008944+0000","last_change":"2023-09-30T02:11:44.999892+0000","last_active":"2023-09-30T02:11:45.008944+0000","last_peered":"2023-09-30T02:11:45.008944+0000","last_clean":"2023-09-30T02:11:45.008944+0000","last_became_active":"2023-09-30T02:06:17.706615+0000","last_became_peered":"2023-09-30T02:06:17.706615+0000","last_unstale":"2023-09-30T02:11:45.008944+0000","last_undegraded":"2023-09-30T02:11:45.008944+0000","last_fullsized":"2023-09-30T02:11:45.008944+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:11:44.982640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:11:44.982640+0000","last_clean_scrub_stamp":"2023-09-30T02:11:44.982640+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:07:32.712429+0000","scrub_duration":0.32725594099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"25'78","reported_seq":130,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-09-30T02:11:38.810874+0000","last_change":"2023-09-30T02:11:38.802202+0000","last_active":"2023-09-30T02:11:38.810874+0000","last_peered":"2023-09-30T02:11:38.810874+0000","last_clean":"2023-09-30T02:11:38.810874+0000","last_became_active":"2023-09-30T02:06:11.592631+0000","last_became_peered":"2023-09-30T02:06:11.592631+0000","last_unstale":"2023-09-30T02:11:38.810874+0000","last_undegraded":"2023-09-30T02:11:38.810874+0000","last_fullsized":"2023-09-30T02:11:38.810874+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"25'78","last_scrub_stamp":"2023-09-30T02:11:38.789901+0000","last_deep_scrub":"25'78","last_deep_scrub_stamp":"2023-09-30T02:11:38.789901+0000","last_clean_scrub_stamp":"2023-09-30T02:11:38.789901+0000","objects_scrubbed":2,"log_size":78,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T13:15:01.381693+0000","scrub_duration":0.75921421200000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-09-30T02:11:55.862695+0000","last_change":"2023-09-30T02:11:55.861465+0000","last_active":"2023-09-30T02:11:55.862695+0000","last_peered":"2023-09-30T02:11:55.862695+0000","last_clean":"2023-09-30T02:11:55.862695+0000","last_became_active":"2023-09-30T02:06:18.091407+0000","last_became_peered":"2023-09-30T02:06:18.091407+0000","last_unstale":"2023-09-30T02:11:55.862695+0000","last_undegraded":"2023-09-30T02:11:55.862695+0000","last_fullsized":"2023-09-30T02:11:55.862695+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:11:55.852135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:11:55.852135+0000","last_clean_scrub_stamp":"2023-09-30T02:11:55.852135+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:50:23.066683+0000","scrub_duration":0.207301711,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-09-30T02:11:43.188136+0000","last_change":"2023-09-30T02:11:43.181007+0000","last_active":"2023-09-30T02:11:43.188136+0000","last_peered":"2023-09-30T02:11:43.188136+0000","last_clean":"2023-09-30T02:11:43.188136+0000","last_became_active":"2023-09-30T02:06:17.420574+0000","last_became_peered":"2023-09-30T02:06:17.420574+0000","last_unstale":"2023-09-30T02:11:43.188136+0000","last_undegraded":"2023-09-30T02:11:43.188136+0000","last_fullsized":"2023-09-30T02:11:43.188136+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:11:43.169695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:11:43.169695+0000","last_clean_scrub_stamp":"2023-09-30T02:11:43.169695+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:10:10.065495+0000","scrub_duration":0.29302578699999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-09-30T02:11:47.937481+0000","last_change":"2023-09-30T02:11:47.936716+0000","last_active":"2023-09-30T02:11:47.937481+0000","last_peered":"2023-09-30T02:11:47.937481+0000","last_clean":"2023-09-30T02:11:47.937481+0000","last_became_active":"2023-09-30T02:06:17.890282+0000","last_became_peered":"2023-09-30T02:06:17.890282+0000","last_unstale":"2023-09-30T02:11:47.937481+0000","last_undegraded":"2023-09-30T02:11:47.937481+0000","last_fullsized":"2023-09-30T02:11:47.937481+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:11:47.929243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:11:47.929243+0000","last_clean_scrub_stamp":"2023-09-30T02:11:47.929243+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T13:34:17.770214+0000","scrub_duration":0.21181633,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-09-30T02:11:56.282358+0000","last_change":"2023-09-30T02:11:56.270837+0000","last_active":"2023-09-30T02:11:56.282358+0000","last_peered":"2023-09-30T02:11:56.282358+0000","last_clean":"2023-09-30T02:11:56.282358+0000","last_became_active":"2023-09-30T02:06:17.836124+0000","last_became_peered":"2023-09-30T02:06:17.836124+0000","last_unstale":"2023-09-30T02:11:56.282358+0000","last_undegraded":"2023-09-30T02:11:56.282358+0000","last_fullsized":"2023-09-30T02:11:56.282358+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:11:56.261674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:11:56.261674+0000","last_clean_scrub_stamp":"2023-09-30T02:11:56.261674+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:22:17.480004+0000","scrub_duration":0.22799536000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":19,"seq":81604378698,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5208,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366632,"statfs":{"total":96636764160,"available":96631431168,"internally_reserved":0,"allocated":212992,"data_stored":75040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":12.859999999999999,"5min":16.204000000000001,"15min":26.364000000000001},"min":{"1min":8.8520000000000003,"5min":7.1429999999999998,"15min":7.1429999999999998},"max":{"1min":17.263000000000002,"5min":414.25400000000002,"15min":414.25400000000002},"last":9.391},{"interface":"front","average":{"1min":12.984,"5min":17.393999999999998,"15min":29.042999999999999},"min":{"1min":7.1289999999999996,"5min":7.1289999999999996,"15min":7.1289999999999996},"max":{"1min":21.555,"5min":421.178,"15min":421.178},"last":9.8190000000000008}]},{"osd":1,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":13.353,"5min":15.817,"15min":24.888999999999999},"min":{"1min":8.9610000000000003,"5min":6.8200000000000003,"15min":6.8200000000000003},"max":{"1min":16.489999999999998,"5min":326.60899999999998,"15min":326.60899999999998},"last":11.096},{"interface":"front","average":{"1min":14.826000000000001,"5min":19.149999999999999,"15min":31.222999999999999},"min":{"1min":8.5099999999999998,"5min":8.5079999999999991,"15min":8.5079999999999991},"max":{"1min":24.559999999999999,"5min":418.35500000000002,"15min":418.35500000000002},"last":16.184000000000001}]},{"osd":2,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":16.120000000000001,"5min":21.582000000000001,"15min":33.603999999999999},"min":{"1min":9.2850000000000001,"5min":8.3960000000000008,"15min":8.3960000000000008},"max":{"1min":23.693999999999999,"5min":423.46699999999998,"15min":423.46699999999998},"last":14.988},{"interface":"front","average":{"1min":14.952,"5min":19.960999999999999,"15min":33.359999999999999},"min":{"1min":9.0749999999999993,"5min":8.4000000000000004,"15min":8.4000000000000004},"max":{"1min":23.123000000000001,"5min":438.697,"15min":438.697},"last":12.864000000000001}]},{"osd":4,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":15.225,"5min":19.898,"15min":32.472000000000001},"min":{"1min":9.5370000000000008,"5min":8.4030000000000005,"15min":8.4030000000000005},"max":{"1min":26.390000000000001,"5min":443.36399999999998,"15min":443.36399999999998},"last":17.286000000000001},{"interface":"front","average":{"1min":17.562999999999999,"5min":22.780999999999999,"15min":35.773000000000003},"min":{"1min":10.282,"5min":10.282,"15min":10.282},"max":{"1min":23.670000000000002,"5min":453.20699999999999,"15min":453.20699999999999},"last":24.088999999999999}]},{"osd":5,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":14.875,"5min":20.896999999999998,"15min":33.405000000000001},"min":{"1min":8.2430000000000003,"5min":8.141,"15min":8.141},"max":{"1min":22.359000000000002,"5min":443.94600000000003,"15min":443.94600000000003},"last":14.208},{"interface":"front","average":{"1min":19.251000000000001,"5min":22.654,"15min":32.534999999999997},"min":{"1min":11.301,"5min":10.746,"15min":10.746},"max":{"1min":25.305,"5min":335.11799999999999,"15min":335.11799999999999},"last":20.98}]},{"osd":6,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":18.507000000000001,"5min":22.931999999999999,"15min":36.216999999999999},"min":{"1min":12.955,"5min":9.4009999999999998,"15min":9.4009999999999998},"max":{"1min":28.02,"5min":432.34699999999998,"15min":432.34699999999998},"last":18.783000000000001},{"interface":"front","average":{"1min":21.754000000000001,"5min":24.712,"15min":34.631999999999998},"min":{"1min":18.292999999999999,"5min":13.018000000000001,"15min":13.018000000000001},"max":{"1min":27.356999999999999,"5min":334.178,"15min":334.178},"last":22.123999999999999}]},{"osd":7,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":17.149000000000001,"5min":23.225000000000001,"15min":35.642000000000003},"min":{"1min":10.933999999999999,"5min":10.928000000000001,"15min":10.928000000000001},"max":{"1min":23.986999999999998,"5min":461.42000000000002,"15min":461.42000000000002},"last":18.003},{"interface":"front","average":{"1min":22.254000000000001,"5min":26.744,"15min":39.024000000000001},"min":{"1min":17.257000000000001,"5min":15.686,"15min":15.686},"max":{"1min":25.774000000000001,"5min":431.27800000000002,"15min":431.27800000000002},"last":22.853999999999999}]}]},{"osd":1,"up_from":18,"seq":77309411401,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5856,"kb_used_data":600,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":94365984,"statfs":{"total":96636764160,"available":96630767616,"internally_reserved":0,"allocated":614400,"data_stored":472880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":13.101000000000001,"5min":13.603999999999999,"15min":13.84},"min":{"1min":8.7560000000000002,"5min":7.1639999999999997,"15min":7.1639999999999997},"max":{"1min":19.596,"5min":24.626999999999999,"15min":24.626999999999999},"last":10.374000000000001},{"interface":"front","average":{"1min":11.407,"5min":12.699,"15min":14.554},"min":{"1min":6.4859999999999998,"5min":6.4859999999999998,"15min":6.4859999999999998},"max":{"1min":17.785,"5min":38.213999999999999,"15min":38.213999999999999},"last":8.3079999999999998}]},{"osd":2,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":12.917999999999999,"5min":13.146000000000001,"15min":13.359999999999999},"min":{"1min":8.4949999999999992,"5min":6.9509999999999996,"15min":6.9509999999999996},"max":{"1min":20.673999999999999,"5min":39.070999999999998,"15min":39.070999999999998},"last":12.686},{"interface":"front","average":{"1min":14.161,"5min":15.342000000000001,"15min":16.507999999999999},"min":{"1min":8.6050000000000004,"5min":8.6050000000000004,"15min":8.6050000000000004},"max":{"1min":20.436,"5min":28.981999999999999,"15min":28.981999999999999},"last":9.2059999999999995}]},{"osd":3,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":18.742999999999999,"5min":28.812999999999999,"15min":56.773000000000003},"min":{"1min":13.654999999999999,"5min":9.2110000000000003,"15min":9.2110000000000003},"max":{"1min":23.431999999999999,"5min":789.26099999999997,"15min":789.26099999999997},"last":22.16},{"interface":"front","average":{"1min":16.210999999999999,"5min":26.545999999999999,"15min":55.965000000000003},"min":{"1min":7.8869999999999996,"5min":7.8869999999999996,"15min":7.8869999999999996},"max":{"1min":24.469000000000001,"5min":827.53899999999999,"15min":827.53899999999999},"last":11.186}]},{"osd":4,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":14.420999999999999,"5min":21.696999999999999,"15min":34.310000000000002},"min":{"1min":11.004,"5min":8.7029999999999994,"15min":8.7029999999999994},"max":{"1min":20.788,"5min":390.31700000000001,"15min":390.31700000000001},"last":13.439},{"interface":"front","average":{"1min":17.638000000000002,"5min":21.739000000000001,"15min":34.366999999999997},"min":{"1min":10.234,"5min":10.234,"15min":10.234},"max":{"1min":24.907,"5min":381.255,"15min":381.255},"last":17.974}]},{"osd":5,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":13.234999999999999,"5min":14.538,"15min":15.808999999999999},"min":{"1min":9.2680000000000007,"5min":7.6849999999999996,"15min":7.6849999999999996},"max":{"1min":21.724,"5min":31.677,"15min":31.677},"last":24.463999999999999},{"interface":"front","average":{"1min":20.417000000000002,"5min":20.757000000000001,"15min":21.004999999999999},"min":{"1min":9.9860000000000007,"5min":9.9860000000000007,"15min":9.9860000000000007},"max":{"1min":26.379000000000001,"5min":31.449999999999999,"15min":31.449999999999999},"last":25.138000000000002}]},{"osd":6,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":18.373999999999999,"5min":19.23,"15min":21.132000000000001},"min":{"1min":12.045,"5min":9.7560000000000002,"15min":9.7560000000000002},"max":{"1min":24.495999999999999,"5min":41.316000000000003,"15min":41.316000000000003},"last":17.449999999999999},{"interface":"front","average":{"1min":16.523,"5min":17.629999999999999,"15min":19.443999999999999},"min":{"1min":10.407,"5min":9.2699999999999996,"15min":9.2699999999999996},"max":{"1min":22.161999999999999,"5min":53.012,"15min":53.012},"last":16.812999999999999}]},{"osd":7,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":19.762,"5min":24.777999999999999,"15min":37.265999999999998},"min":{"1min":14.295999999999999,"5min":9.5310000000000006,"15min":9.5310000000000006},"max":{"1min":23.609000000000002,"5min":332.072,"15min":332.072},"last":20.041},{"interface":"front","average":{"1min":17.829000000000001,"5min":20.178999999999998,"15min":24.689},"min":{"1min":11.316000000000001,"5min":8.141,"15min":8.141},"max":{"1min":23.631,"5min":108.00700000000001,"15min":108.00700000000001},"last":14.682}]}]},{"osd":7,"up_from":14,"seq":60129542219,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5272,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":94366568,"statfs":{"total":96636764160,"available":96631365632,"internally_reserved":0,"allocated":212992,"data_stored":75040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:11:21 2023","interfaces":[{"interface":"back","average":{"1min":8.6750000000000007,"5min":8.9369999999999994,"15min":8.9199999999999999},"min":{"1min":6.2140000000000004,"5min":5.3120000000000003,"15min":5.3120000000000003},"max":{"1min":16.916,"5min":19.491,"15min":19.491},"last":7.9850000000000003},{"interface":"front","average":{"1min":10.153,"5min":9.9879999999999995,"15min":9.7560000000000002},"min":{"1min":7.3099999999999996,"5min":6.2850000000000001,"15min":6.2850000000000001},"max":{"1min":19.263000000000002,"5min":20.952000000000002,"15min":20.952000000000002},"last":7.6790000000000003}]},{"osd":1,"last update":"Sat Sep 30 02:11:21 2023","interfaces":[{"interface":"back","average":{"1min":10.753,"5min":11.709,"15min":14.192},"min":{"1min":6.6630000000000003,"5min":6.6630000000000003,"15min":6.6630000000000003},"max":{"1min":18.591000000000001,"5min":89.454999999999998,"15min":89.454999999999998},"last":10.827},{"interface":"front","average":{"1min":11.548999999999999,"5min":12.834,"15min":14.573},"min":{"1min":8.0909999999999993,"5min":7.0700000000000003,"15min":7.0700000000000003},"max":{"1min":19.093,"5min":83.695999999999998,"15min":83.695999999999998},"last":10.151999999999999}]},{"osd":2,"last update":"Sat Sep 30 02:11:21 2023","interfaces":[{"interface":"back","average":{"1min":13.856,"5min":13.919,"15min":13.823},"min":{"1min":8.1240000000000006,"5min":7.4669999999999996,"15min":7.4669999999999996},"max":{"1min":20.259,"5min":21.370999999999999,"15min":21.370999999999999},"last":11.164},{"interface":"front","average":{"1min":14.368,"5min":14.385,"15min":15.127000000000001},"min":{"1min":10.176,"5min":7.8920000000000003,"15min":7.8920000000000003},"max":{"1min":19.542999999999999,"5min":21.712,"15min":21.712},"last":16.035}]},{"osd":3,"last update":"Sat Sep 30 02:11:21 2023","interfaces":[{"interface":"back","average":{"1min":15.417999999999999,"5min":15.926,"15min":15.646000000000001},"min":{"1min":11.319000000000001,"5min":7.1710000000000003,"15min":7.1710000000000003},"max":{"1min":21.238,"5min":24.276,"15min":24.276},"last":14.792999999999999},{"interface":"front","average":{"1min":16.335999999999999,"5min":16.087,"15min":16.295000000000002},"min":{"1min":12.477,"5min":7.9630000000000001,"15min":7.9630000000000001},"max":{"1min":22.021999999999998,"5min":24.716000000000001,"15min":24.716000000000001},"last":18.710000000000001}]},{"osd":4,"last update":"Sat Sep 30 02:11:12 2023","interfaces":[{"interface":"back","average":{"1min":16.693999999999999,"5min":17.527000000000001,"15min":19.007999999999999},"min":{"1min":9.4380000000000006,"5min":5.7039999999999997,"15min":5.7039999999999997},"max":{"1min":23.076000000000001,"5min":99.298000000000002,"15min":99.298000000000002},"last":19.934000000000001},{"interface":"front","average":{"1min":12.294,"5min":15.105,"15min":17.399000000000001},"min":{"1min":7.1509999999999998,"5min":7.1509999999999998,"15min":7.1509999999999998},"max":{"1min":17.899000000000001,"5min":86.938999999999993,"15min":86.938999999999993},"last":15.176}]},{"osd":5,"last update":"Sat Sep 30 02:11:12 2023","interfaces":[{"interface":"back","average":{"1min":15.673999999999999,"5min":17.282,"15min":19.91},"min":{"1min":8.6509999999999998,"5min":7.6050000000000004,"15min":7.6050000000000004},"max":{"1min":21.527000000000001,"5min":118.675,"15min":118.675},"last":16.812000000000001},{"interface":"front","average":{"1min":18.716999999999999,"5min":19.459,"15min":21.693999999999999},"min":{"1min":16.239999999999998,"5min":8.7330000000000005,"15min":8.7330000000000005},"max":{"1min":23.338999999999999,"5min":114.14400000000001,"15min":114.14400000000001},"last":20.959}]},{"osd":6,"last update":"Sat Sep 30 02:11:12 2023","interfaces":[{"interface":"back","average":{"1min":19.045000000000002,"5min":20.443999999999999,"15min":23.024000000000001},"min":{"1min":8.4359999999999999,"5min":8.4359999999999999,"15min":8.4359999999999999},"max":{"1min":24.417000000000002,"5min":111.048,"15min":111.048},"last":21.850000000000001},{"interface":"front","average":{"1min":18.745000000000001,"5min":19.427,"15min":22.183},"min":{"1min":12.542,"5min":9.8140000000000001,"15min":9.8140000000000001},"max":{"1min":23.036000000000001,"5min":115.241,"15min":115.241},"last":18.050999999999998}]}]},{"osd":0,"up_from":18,"seq":77309411402,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5208,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366632,"statfs":{"total":96636764160,"available":96631431168,"internally_reserved":0,"allocated":212992,"data_stored":75040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 30 02:11:19 2023","interfaces":[{"interface":"back","average":{"1min":10.754,"5min":16.196000000000002,"15min":29.155000000000001},"min":{"1min":7.2089999999999996,"5min":7.2089999999999996,"15min":7.2089999999999996},"max":{"1min":19.498999999999999,"5min":425.49599999999998,"15min":425.49599999999998},"last":8.6029999999999998},{"interface":"front","average":{"1min":13.093,"5min":20.338000000000001,"15min":39.884},"min":{"1min":8.1210000000000004,"5min":7.9269999999999996,"15min":7.9269999999999996},"max":{"1min":18.349,"5min":667.81100000000004,"15min":667.81100000000004},"last":13.682}]},{"osd":2,"last update":"Sat Sep 30 02:11:19 2023","interfaces":[{"interface":"back","average":{"1min":12.025,"5min":14.532999999999999,"15min":18.928999999999998},"min":{"1min":8.2110000000000003,"5min":7.718,"15min":7.718},"max":{"1min":16.954000000000001,"5min":162.50700000000001,"15min":162.50700000000001},"last":12.188000000000001},{"interface":"front","average":{"1min":17.457000000000001,"5min":17.991,"15min":20.966999999999999},"min":{"1min":10.747,"5min":8.5079999999999991,"15min":8.5079999999999991},"max":{"1min":23.617000000000001,"5min":131.863,"15min":131.863},"last":19.422999999999998}]},{"osd":3,"last update":"Sat Sep 30 02:11:19 2023","interfaces":[{"interface":"back","average":{"1min":13.837,"5min":14.763,"15min":15.271000000000001},"min":{"1min":8.1020000000000003,"5min":7.4859999999999998,"15min":7.4859999999999998},"max":{"1min":22.571999999999999,"5min":24.763000000000002,"15min":24.763000000000002},"last":16.965},{"interface":"front","average":{"1min":18.533999999999999,"5min":18.309999999999999,"15min":18.783000000000001},"min":{"1min":10.459,"5min":9.8569999999999993,"15min":9.8569999999999993},"max":{"1min":23.57,"5min":27.309000000000001,"15min":27.309000000000001},"last":22.440000000000001}]},{"osd":4,"last update":"Sat Sep 30 02:11:19 2023","interfaces":[{"interface":"back","average":{"1min":16.027999999999999,"5min":16.099,"15min":19.271000000000001},"min":{"1min":9.9179999999999993,"5min":9.0700000000000003,"15min":9.0700000000000003},"max":{"1min":22.122,"5min":102.456,"15min":102.456},"last":21.806999999999999},{"interface":"front","average":{"1min":13.881,"5min":17.271000000000001,"15min":20.933},"min":{"1min":8.7929999999999993,"5min":8.7929999999999993,"15min":8.7929999999999993},"max":{"1min":19.463999999999999,"5min":98.293999999999997,"15min":98.293999999999997},"last":14.119999999999999}]},{"osd":5,"last update":"Sat Sep 30 02:11:19 2023","interfaces":[{"interface":"back","average":{"1min":20.268999999999998,"5min":20.559999999999999,"15min":22.850000000000001},"min":{"1min":14.006,"5min":10.726000000000001,"15min":10.726000000000001},"max":{"1min":28.995000000000001,"5min":79.566999999999993,"15min":79.566999999999993},"last":25.850999999999999},{"interface":"front","average":{"1min":16.812999999999999,"5min":18.582999999999998,"15min":21.466999999999999},"min":{"1min":9.7599999999999998,"5min":9.7599999999999998,"15min":9.7599999999999998},"max":{"1min":24.213000000000001,"5min":109.80500000000001,"15min":109.80500000000001},"last":16.300999999999998}]},{"osd":6,"last update":"Sat Sep 30 02:11:19 2023","interfaces":[{"interface":"back","average":{"1min":18.077999999999999,"5min":20.367000000000001,"15min":26.062999999999999},"min":{"1min":11.140000000000001,"5min":9.2829999999999995,"15min":9.2829999999999995},"max":{"1min":24.276,"5min":161.01499999999999,"15min":161.01499999999999},"last":23.396000000000001},{"interface":"front","average":{"1min":22.562000000000001,"5min":22.975000000000001,"15min":26.202999999999999},"min":{"1min":13.904,"5min":13.163,"15min":13.163},"max":{"1min":27.786000000000001,"5min":118.71599999999999,"15min":118.71599999999999},"last":25.111000000000001}]},{"osd":7,"last update":"Sat Sep 30 02:11:19 2023","interfaces":[{"interface":"back","average":{"1min":20.117999999999999,"5min":20.393000000000001,"15min":23.483000000000001},"min":{"1min":11.741,"5min":9.8810000000000002,"15min":9.8810000000000002},"max":{"1min":26.491,"5min":111.86,"15min":111.86},"last":20.891999999999999},{"interface":"front","average":{"1min":21.664000000000001,"5min":22.388999999999999,"15min":26.010999999999999},"min":{"1min":15.186999999999999,"5min":10.237,"15min":10.237},"max":{"1min":27.012,"5min":126.889,"15min":126.889},"last":18.373999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574922,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5208,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366632,"statfs":{"total":96636764160,"available":96631431168,"internally_reserved":0,"allocated":212992,"data_stored":75040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:11:20 2023","interfaces":[{"interface":"back","average":{"1min":9.6969999999999992,"5min":9.8569999999999993,"15min":9.7870000000000008},"min":{"1min":6.5830000000000002,"5min":5.774,"15min":5.774},"max":{"1min":16.530999999999999,"5min":19.709,"15min":19.709},"last":16.937000000000001},{"interface":"front","average":{"1min":10.919,"5min":10.390000000000001,"15min":11.138999999999999},"min":{"1min":6.1239999999999997,"5min":5.0419999999999998,"15min":5.0419999999999998},"max":{"1min":19.603000000000002,"5min":23.134,"15min":23.134},"last":7.218}]},{"osd":1,"last update":"Sat Sep 30 02:11:20 2023","interfaces":[{"interface":"back","average":{"1min":11.829000000000001,"5min":12.981999999999999,"15min":14.561999999999999},"min":{"1min":7.351,"5min":6.3010000000000002,"15min":6.3010000000000002},"max":{"1min":17.946999999999999,"5min":27.242999999999999,"15min":27.242999999999999},"last":17.564},{"interface":"front","average":{"1min":11.532999999999999,"5min":11.537000000000001,"15min":12.59},"min":{"1min":7.9770000000000003,"5min":7.4989999999999997,"15min":7.4989999999999997},"max":{"1min":18.957000000000001,"5min":24.388999999999999,"15min":24.388999999999999},"last":9.3849999999999998}]},{"osd":2,"last update":"Sat Sep 30 02:11:20 2023","interfaces":[{"interface":"back","average":{"1min":14.006,"5min":14.630000000000001,"15min":15.132},"min":{"1min":9.6690000000000005,"5min":7.7039999999999997,"15min":7.7039999999999997},"max":{"1min":19.521000000000001,"5min":24.510999999999999,"15min":24.510999999999999},"last":12.991},{"interface":"front","average":{"1min":12.595000000000001,"5min":13.534000000000001,"15min":14.086},"min":{"1min":7.9699999999999998,"5min":7.9699999999999998,"15min":7.9699999999999998},"max":{"1min":20.085000000000001,"5min":25.033999999999999,"15min":25.033999999999999},"last":11.234999999999999}]},{"osd":3,"last update":"Sat Sep 30 02:11:20 2023","interfaces":[{"interface":"back","average":{"1min":14.676,"5min":19.765999999999998,"15min":31.925000000000001},"min":{"1min":8.3770000000000007,"5min":8.2089999999999996,"15min":8.2089999999999996},"max":{"1min":20.157,"5min":540.52099999999996,"15min":540.52099999999996},"last":7.9100000000000001},{"interface":"front","average":{"1min":15.218,"5min":21.224,"15min":35.917999999999999},"min":{"1min":9.4489999999999998,"5min":7.9649999999999999,"15min":7.9649999999999999},"max":{"1min":19.452999999999999,"5min":644.31600000000003,"15min":644.31600000000003},"last":15.465999999999999}]},{"osd":4,"last update":"Sat Sep 30 02:11:15 2023","interfaces":[{"interface":"back","average":{"1min":17.922000000000001,"5min":18.664999999999999,"15min":22.556999999999999},"min":{"1min":13.959,"5min":9.0969999999999995,"15min":9.0969999999999995},"max":{"1min":21.425999999999998,"5min":179.749,"15min":179.749},"last":11.852},{"interface":"front","average":{"1min":15.129,"5min":15.641999999999999,"15min":19.817},"min":{"1min":8.7550000000000008,"5min":7.6070000000000002,"15min":7.6070000000000002},"max":{"1min":19.663,"5min":174.52799999999999,"15min":174.52799999999999},"last":10.372}]},{"osd":5,"last update":"Sat Sep 30 02:11:15 2023","interfaces":[{"interface":"back","average":{"1min":16.963000000000001,"5min":16.859000000000002,"15min":16.969000000000001},"min":{"1min":9.6210000000000004,"5min":8.2379999999999995,"15min":8.2379999999999995},"max":{"1min":21.975999999999999,"5min":36.709000000000003,"15min":36.709000000000003},"last":19.686},{"interface":"front","average":{"1min":15.141999999999999,"5min":16.25,"15min":16.556000000000001},"min":{"1min":9.0809999999999995,"5min":5.0279999999999996,"15min":5.0279999999999996},"max":{"1min":21.393999999999998,"5min":27.364999999999998,"15min":27.364999999999998},"last":14.728}]},{"osd":7,"last update":"Sat Sep 30 02:11:09 2023","interfaces":[{"interface":"back","average":{"1min":19.030000000000001,"5min":19.597000000000001,"15min":22.396999999999998},"min":{"1min":13.644,"5min":10.367000000000001,"15min":10.367000000000001},"max":{"1min":22.771000000000001,"5min":105.801,"15min":105.801},"last":18.77},{"interface":"front","average":{"1min":17.553000000000001,"5min":19.193999999999999,"15min":20.960000000000001},"min":{"1min":7.3120000000000003,"5min":7.3120000000000003,"15min":7.3120000000000003},"max":{"1min":23.283000000000001,"5min":90.156000000000006,"15min":90.156000000000006},"last":13.851000000000001}]}]},{"osd":4,"up_from":15,"seq":64424509515,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5208,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366632,"statfs":{"total":96636764160,"available":96631431168,"internally_reserved":0,"allocated":212992,"data_stored":75040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:11:17 2023","interfaces":[{"interface":"back","average":{"1min":11.865,"5min":10.567,"15min":9.4399999999999995},"min":{"1min":6.3970000000000002,"5min":5.2699999999999996,"15min":5.2699999999999996},"max":{"1min":20.655000000000001,"5min":24.532,"15min":24.532},"last":9.2720000000000002},{"interface":"front","average":{"1min":11.137,"5min":10.974,"15min":11.926},"min":{"1min":7.0190000000000001,"5min":5.4530000000000003,"15min":5.4530000000000003},"max":{"1min":21.841000000000001,"5min":25.225000000000001,"15min":25.225000000000001},"last":11.529999999999999}]},{"osd":1,"last update":"Sat Sep 30 02:11:17 2023","interfaces":[{"interface":"back","average":{"1min":12.827,"5min":13.271000000000001,"15min":13.897},"min":{"1min":8.3960000000000008,"5min":6.2830000000000004,"15min":6.2830000000000004},"max":{"1min":17.864000000000001,"5min":28.917000000000002,"15min":28.917000000000002},"last":9.7170000000000005},{"interface":"front","average":{"1min":10.146000000000001,"5min":11.148,"15min":11.789999999999999},"min":{"1min":7.4059999999999997,"5min":7.0250000000000004,"15min":7.0250000000000004},"max":{"1min":19.279,"5min":30.922999999999998,"15min":30.922999999999998},"last":13.794}]},{"osd":2,"last update":"Sat Sep 30 02:11:17 2023","interfaces":[{"interface":"back","average":{"1min":14.563000000000001,"5min":13.641999999999999,"15min":14.407999999999999},"min":{"1min":7.8300000000000001,"5min":7.1609999999999996,"15min":7.1609999999999996},"max":{"1min":19.428000000000001,"5min":21.175000000000001,"15min":21.175000000000001},"last":12.664},{"interface":"front","average":{"1min":12.637,"5min":13.872,"15min":14.563000000000001},"min":{"1min":6.7060000000000004,"5min":6.7060000000000004,"15min":6.7060000000000004},"max":{"1min":18.747,"5min":27.533999999999999,"15min":27.533999999999999},"last":10.593999999999999}]},{"osd":3,"last update":"Sat Sep 30 02:11:23 2023","interfaces":[{"interface":"back","average":{"1min":16.146000000000001,"5min":16.042999999999999,"15min":15.887},"min":{"1min":9.4719999999999995,"5min":9.4719999999999995,"15min":9.4719999999999995},"max":{"1min":23.526,"5min":27.632999999999999,"15min":27.632999999999999},"last":18.475999999999999},{"interface":"front","average":{"1min":15.352,"5min":16.143999999999998,"15min":17.184999999999999},"min":{"1min":8.2599999999999998,"5min":7.9269999999999996,"15min":7.9269999999999996},"max":{"1min":23.113,"5min":30.495000000000001,"15min":30.495000000000001},"last":17.306000000000001}]},{"osd":5,"last update":"Sat Sep 30 02:11:11 2023","interfaces":[{"interface":"back","average":{"1min":16.295999999999999,"5min":18.059000000000001,"15min":20.965},"min":{"1min":12.298,"5min":7.9080000000000004,"15min":7.9080000000000004},"max":{"1min":21.044,"5min":99.784000000000006,"15min":99.784000000000006},"last":19.439},{"interface":"front","average":{"1min":13.371,"5min":15.058,"15min":17.594999999999999},"min":{"1min":8.3030000000000008,"5min":7.609,"15min":7.609},"max":{"1min":19.103000000000002,"5min":98.634,"15min":98.634},"last":14.579000000000001}]},{"osd":6,"last update":"Sat Sep 30 02:11:11 2023","interfaces":[{"interface":"back","average":{"1min":16.995000000000001,"5min":16.553999999999998,"15min":16.952999999999999},"min":{"1min":7.8949999999999996,"5min":7.8949999999999996,"15min":7.8949999999999996},"max":{"1min":22.498999999999999,"5min":27.277999999999999,"15min":27.277999999999999},"last":17.015999999999998},{"interface":"front","average":{"1min":15.564,"5min":16.422000000000001,"15min":16.088999999999999},"min":{"1min":9.109,"5min":6.5810000000000004,"15min":6.5810000000000004},"max":{"1min":20.640999999999998,"5min":28.640000000000001,"15min":28.640000000000001},"last":15.709}]},{"osd":7,"last update":"Sat Sep 30 02:11:11 2023","interfaces":[{"interface":"back","average":{"1min":17.225000000000001,"5min":18.161000000000001,"15min":18.513000000000002},"min":{"1min":7.79,"5min":7.79,"15min":7.79},"max":{"1min":23.047000000000001,"5min":31.143000000000001,"15min":31.143000000000001},"last":20.454999999999998},{"interface":"front","average":{"1min":17.949000000000002,"5min":18.007000000000001,"15min":18.579000000000001},"min":{"1min":10.542999999999999,"5min":8.6509999999999998,"15min":8.6509999999999998},"max":{"1min":22.966999999999999,"5min":30.529,"15min":30.529},"last":21.221}]}]},{"osd":5,"up_from":15,"seq":64424509515,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5920,"kb_used_data":600,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":94365920,"statfs":{"total":96636764160,"available":96630702080,"internally_reserved":0,"allocated":614400,"data_stored":472880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:11:18 2023","interfaces":[{"interface":"back","average":{"1min":10.455,"5min":10.214,"15min":11.779999999999999},"min":{"1min":6.1040000000000001,"5min":5.7350000000000003,"15min":5.7350000000000003},"max":{"1min":19.498999999999999,"5min":22.452999999999999,"15min":22.452999999999999},"last":13.135999999999999},{"interface":"front","average":{"1min":10.673999999999999,"5min":10.722,"15min":11.548},"min":{"1min":6.6929999999999996,"5min":5.8879999999999999,"15min":5.8879999999999999},"max":{"1min":20.34,"5min":22.821999999999999,"15min":22.821999999999999},"last":16.041}]},{"osd":1,"last update":"Sat Sep 30 02:11:18 2023","interfaces":[{"interface":"back","average":{"1min":14.335000000000001,"5min":13.829000000000001,"15min":15.194000000000001},"min":{"1min":8.9589999999999996,"5min":7.29,"15min":7.29},"max":{"1min":22.402999999999999,"5min":47.451999999999998,"15min":47.451999999999998},"last":19.192},{"interface":"front","average":{"1min":11.304,"5min":11.545,"15min":12.558},"min":{"1min":7.774,"5min":6.6130000000000004,"15min":6.6130000000000004},"max":{"1min":15.734999999999999,"5min":48.841999999999999,"15min":48.841999999999999},"last":22.173999999999999}]},{"osd":2,"last update":"Sat Sep 30 02:11:18 2023","interfaces":[{"interface":"back","average":{"1min":15.868,"5min":13.461,"15min":11.835000000000001},"min":{"1min":8.2829999999999995,"5min":6.8250000000000002,"15min":6.8250000000000002},"max":{"1min":27.501000000000001,"5min":27.501000000000001,"15min":27.501000000000001},"last":15.035},{"interface":"front","average":{"1min":12.898999999999999,"5min":14.695,"15min":15.513},"min":{"1min":7.8150000000000004,"5min":7.5369999999999999,"15min":7.5369999999999999},"max":{"1min":19.411000000000001,"5min":26.576000000000001,"15min":26.576000000000001},"last":10.305999999999999}]},{"osd":3,"last update":"Sat Sep 30 02:11:30 2023","interfaces":[{"interface":"back","average":{"1min":14.194000000000001,"5min":15.137,"15min":15.063000000000001},"min":{"1min":7.6980000000000004,"5min":7.2619999999999996,"15min":7.2619999999999996},"max":{"1min":21.891999999999999,"5min":26.895,"15min":26.895},"last":12.260999999999999},{"interface":"front","average":{"1min":15.656000000000001,"5min":17.158999999999999,"15min":17.841999999999999},"min":{"1min":11.603,"5min":11.140000000000001,"15min":11.140000000000001},"max":{"1min":19.952000000000002,"5min":24.702999999999999,"15min":24.702999999999999},"last":19.509}]},{"osd":4,"last update":"Sat Sep 30 02:11:18 2023","interfaces":[{"interface":"back","average":{"1min":15.975,"5min":17.355,"15min":18.113},"min":{"1min":10.063000000000001,"5min":8.8160000000000007,"15min":8.8160000000000007},"max":{"1min":22.620999999999999,"5min":26.710000000000001,"15min":26.710000000000001},"last":15.744},{"interface":"front","average":{"1min":16.137,"5min":16.138999999999999,"15min":15.467000000000001},"min":{"1min":10.363,"5min":6.0270000000000001,"15min":6.0270000000000001},"max":{"1min":23.376000000000001,"5min":28.344999999999999,"15min":28.344999999999999},"last":20.524000000000001}]},{"osd":6,"last update":"Sat Sep 30 02:11:18 2023","interfaces":[{"interface":"back","average":{"1min":17.373000000000001,"5min":16.178999999999998,"15min":14.978999999999999},"min":{"1min":10.907999999999999,"5min":7.0679999999999996,"15min":7.0679999999999996},"max":{"1min":22.890999999999998,"5min":23.573,"15min":23.573},"last":21.099},{"interface":"front","average":{"1min":14.722,"5min":16.218,"15min":16.617000000000001},"min":{"1min":7.8550000000000004,"5min":7.5540000000000003,"15min":7.5540000000000003},"max":{"1min":20.41,"5min":27.763999999999999,"15min":27.763999999999999},"last":11.077}]},{"osd":7,"last update":"Sat Sep 30 02:11:18 2023","interfaces":[{"interface":"back","average":{"1min":17.943999999999999,"5min":18.265999999999998,"15min":19.606000000000002},"min":{"1min":12.4,"5min":10.584,"15min":10.584},"max":{"1min":22.768000000000001,"5min":28.951000000000001,"15min":28.951000000000001},"last":16.748000000000001},{"interface":"front","average":{"1min":17.033999999999999,"5min":16.844000000000001,"15min":16.053999999999998},"min":{"1min":11.109,"5min":7.0490000000000004,"15min":7.0490000000000004},"max":{"1min":21.635999999999999,"5min":25.151,"15min":25.151},"last":17.436}]}]},{"osd":2,"up_from":18,"seq":77309411401,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5208,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366632,"statfs":{"total":96636764160,"available":96631431168,"internally_reserved":0,"allocated":212992,"data_stored":75040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:11:21 2023","interfaces":[{"interface":"back","average":{"1min":12.927,"5min":13.202999999999999,"15min":13.711},"min":{"1min":7.1929999999999996,"5min":6.665,"15min":6.665},"max":{"1min":20.434000000000001,"5min":69.003,"15min":69.003},"last":8.5220000000000002},{"interface":"front","average":{"1min":13.492000000000001,"5min":14.566000000000001,"15min":16.558},"min":{"1min":8.6880000000000006,"5min":8.1329999999999991,"15min":8.1329999999999991},"max":{"1min":25.254000000000001,"5min":91.331999999999994,"15min":91.331999999999994},"last":11.916}]},{"osd":1,"last update":"Sat Sep 30 02:11:21 2023","interfaces":[{"interface":"back","average":{"1min":14.648999999999999,"5min":16.484999999999999,"15min":20.882000000000001},"min":{"1min":9.4800000000000004,"5min":7.9630000000000001,"15min":7.9630000000000001},"max":{"1min":21.640999999999998,"5min":145.952,"15min":145.952},"last":15.074999999999999},{"interface":"front","average":{"1min":15.581,"5min":17.215,"15min":21.452000000000002},"min":{"1min":9.6679999999999993,"5min":8.7469999999999999,"15min":8.7469999999999999},"max":{"1min":25.678000000000001,"5min":155.72900000000001,"15min":155.72900000000001},"last":21.422000000000001}]},{"osd":3,"last update":"Sat Sep 30 02:11:21 2023","interfaces":[{"interface":"back","average":{"1min":16.968,"5min":16.724,"15min":17.524000000000001},"min":{"1min":10.601000000000001,"5min":10.124000000000001,"15min":10.124000000000001},"max":{"1min":32.027000000000001,"5min":39.378999999999998,"15min":39.378999999999998},"last":16.736000000000001},{"interface":"front","average":{"1min":17.798999999999999,"5min":18.437000000000001,"15min":18.405000000000001},"min":{"1min":8.9250000000000007,"5min":8.9250000000000007,"15min":8.9250000000000007},"max":{"1min":24.704999999999998,"5min":36.649000000000001,"15min":36.649000000000001},"last":13.365}]},{"osd":4,"last update":"Sat Sep 30 02:11:21 2023","interfaces":[{"interface":"back","average":{"1min":18.395,"5min":18.151,"15min":18.896999999999998},"min":{"1min":11.792999999999999,"5min":8.9359999999999999,"15min":8.9359999999999999},"max":{"1min":29.699000000000002,"5min":39.186999999999998,"15min":39.186999999999998},"last":16.016999999999999},{"interface":"front","average":{"1min":11.427,"5min":11.253,"15min":12.537000000000001},"min":{"1min":6.7329999999999997,"5min":6.7329999999999997,"15min":6.7329999999999997},"max":{"1min":23.036999999999999,"5min":31.32,"15min":31.32},"last":10.725}]},{"osd":5,"last update":"Sat Sep 30 02:11:21 2023","interfaces":[{"interface":"back","average":{"1min":19.495000000000001,"5min":18.518999999999998,"15min":19.271000000000001},"min":{"1min":12.682,"5min":8.3499999999999996,"15min":8.3499999999999996},"max":{"1min":26.695,"5min":43.409999999999997,"15min":43.409999999999997},"last":9.7379999999999995},{"interface":"front","average":{"1min":20.788,"5min":21.66,"15min":22.609999999999999},"min":{"1min":11.974,"5min":11.974,"15min":11.974},"max":{"1min":33.695999999999998,"5min":40.646000000000001,"15min":40.646000000000001},"last":18.666}]},{"osd":6,"last update":"Sat Sep 30 02:11:21 2023","interfaces":[{"interface":"back","average":{"1min":13.474,"5min":14.634,"15min":15.917999999999999},"min":{"1min":9.8520000000000003,"5min":7.4829999999999997,"15min":7.4829999999999997},"max":{"1min":21.18,"5min":34.045000000000002,"15min":34.045000000000002},"last":20.277000000000001},{"interface":"front","average":{"1min":19.893999999999998,"5min":19.949000000000002,"15min":21.067},"min":{"1min":12.353999999999999,"5min":9.1600000000000001,"15min":9.1600000000000001},"max":{"1min":24.425000000000001,"5min":47.997999999999998,"15min":47.997999999999998},"last":20.722000000000001}]},{"osd":7,"last update":"Sat Sep 30 02:11:21 2023","interfaces":[{"interface":"back","average":{"1min":21.699999999999999,"5min":22.599,"15min":24.295000000000002},"min":{"1min":15.781000000000001,"5min":13.542999999999999,"15min":13.542999999999999},"max":{"1min":31.061,"5min":44.585000000000001,"15min":44.585000000000001},"last":19.369},{"interface":"front","average":{"1min":15.755000000000001,"5min":16.690999999999999,"15min":17.437000000000001},"min":{"1min":8.8230000000000004,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":26.234000000000002,"5min":38.295000000000002,"15min":38.295000000000002},"last":12.651}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T02:12:03.807 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-09-30T02:06:15.800912+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=2, tm_min=6, tm_sec=15, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=2, tm_min=11, tm_sec=33, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T02:12:03.807 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-30T02:12:14.657 INFO:tasks.ceph.mon.a.smithi059.stderr:==00:00:07:24.522 116256== Warning: unimplemented fcntl command: 1036 2023-09-30T02:12:14.662 INFO:tasks.ceph.mon.a.smithi059.stderr:==00:00:07:24.527 116256== Warning: unimplemented fcntl command: 1036 2023-09-30T02:12:14.690 INFO:tasks.ceph.mon.a.smithi059.stderr:==00:00:07:24.556 116256== Warning: unimplemented fcntl command: 1036 2023-09-30T02:12:14.714 INFO:tasks.ceph.mon.b.smithi110.stderr:==00:00:07:24.561 115869== Warning: unimplemented fcntl command: 1036 2023-09-30T02:12:14.718 INFO:tasks.ceph.mon.b.smithi110.stderr:==00:00:07:24.565 115869== Warning: unimplemented fcntl command: 1036 2023-09-30T02:12:14.720 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:07:24.586 116257== Warning: unimplemented fcntl command: 1036 2023-09-30T02:12:14.729 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:07:24.594 116257== Warning: unimplemented fcntl command: 1036 2023-09-30T02:12:14.741 INFO:tasks.ceph.mon.b.smithi110.stderr:==00:00:07:24.589 115869== Warning: unimplemented fcntl command: 1036 2023-09-30T02:12:14.752 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:07:24.618 116257== Warning: unimplemented fcntl command: 1036 2023-09-30T02:12:23.808 DEBUG:teuthology.orchestra.run.smithi059:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T02:12:23.948 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.947+0000 7fe93c86c700 10 monclient: get_monmap_and_config 2023-09-30T02:12:23.948 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.947+0000 7fe93c86c700 10 monclient: build_initial_monmap 2023-09-30T02:12:23.948 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.947+0000 7fe93c86c700 10 monclient: monmap: 2023-09-30T02:12:23.948 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:12:23.948 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:12:23.948 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:12:23.949097+0000 2023-09-30T02:12:23.948 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:12:23.949097+0000 2023-09-30T02:12:23.948 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:12:23.949 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:12:23.949 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:12:23.949 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:12:23.949 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:12:23.949 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:12:23.949 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:12:23.949 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:12:23.951 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.948+0000 7fe93c86c700 1 Processor -- start 2023-09-30T02:12:23.951 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.948+0000 7fe93c86c700 1 -- start start 2023-09-30T02:12:23.951 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.948+0000 7fe93c86c700 10 monclient: init 2023-09-30T02:12:23.951 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.949+0000 7fe93c86c700 10 monclient: _reopen_session rank -1 2023-09-30T02:12:23.951 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.949+0000 7fe93c86c700 10 monclient: _add_conns ranks=[4,1,3,2,0] 2023-09-30T02:12:23.951 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.949+0000 7fe93c86c700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe9380a5b70 0x7fe9380b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:12:23.951 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.949+0000 7fe93c86c700 10 monclient(hunting): picked mon.noname-c con 0x7fe9380a5b70 addr v2:172.21.15.110:3300/0 2023-09-30T02:12:23.951 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.949+0000 7fe93c86c700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe9380a5f40 addr v1:172.21.15.110:6789/0 2023-09-30T02:12:23.951 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.949+0000 7fe93c86c700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fe938142290 0x7fe93813af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:12:23.951 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.949+0000 7fe93c86c700 10 monclient(hunting): picked mon.noname-a con 0x7fe938142290 addr v2:172.21.15.59:3300/0 2023-09-30T02:12:23.951 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.949+0000 7fe93c86c700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9380a75a0 con 0x7fe9380a5f40 2023-09-30T02:12:23.952 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.949+0000 7fe93c86c700 10 monclient(hunting): start opening mon connection 2023-09-30T02:12:23.952 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.949+0000 7fe93c86c700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9380a63e0 con 0x7fe938142290 2023-09-30T02:12:23.952 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.949+0000 7fe93c86c700 10 monclient(hunting): start opening mon connection 2023-09-30T02:12:23.952 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.949+0000 7fe93c86c700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9381426b0 con 0x7fe9380a5b70 2023-09-30T02:12:23.952 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.949+0000 7fe93c86c700 10 monclient(hunting): _renew_subs 2023-09-30T02:12:23.952 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.949+0000 7fe93c86c700 10 monclient(hunting): authenticate will time out at 2189.252311s 2023-09-30T02:12:23.952 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.949+0000 7fe936d9d700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fe938142290 0x7fe93813af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:12:23.952 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.950+0000 7fe93659c700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7fe9380a5b70 0x7fe9380b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:12:23.952 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.950+0000 7fe936d9d700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7fe938142290 0x7fe93813af30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:40056/0 (socket says 172.21.15.59:40056) 2023-09-30T02:12:23.952 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.950+0000 7fe936d9d700 1 -- 172.21.15.59:0/1766556243 learned_addr learned my addr 172.21.15.59:0/1766556243 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:12:23.952 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.950+0000 7fe936d9d700 10 monclient(hunting): get_auth_request con 0x7fe938142290 auth_method 0 2023-09-30T02:12:23.952 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.950+0000 7fe936d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:12:23.953 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.950+0000 7fe936d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:12:23.953 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.950+0000 7fe936d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:12:23.953 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.950+0000 7fe93659c700 10 monclient(hunting): get_auth_request con 0x7fe9380a5b70 auth_method 0 2023-09-30T02:12:23.953 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.950+0000 7fe93659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:12:23.953 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.950+0000 7fe93659c700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:12:23.953 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.950+0000 7fe93659c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:12:23.954 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.951+0000 7fe936d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:12:23.954 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.951+0000 7fe936d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:12:23.954 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.951+0000 7fe936d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:12:23.954 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.951+0000 7fe93659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:12:23.954 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.951+0000 7fe93659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:12:23.954 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.951+0000 7fe93659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:12:23.954 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.953+0000 7fe936d9d700 10 monclient(hunting): handle_auth_done global_id 4941 payload 274 2023-09-30T02:12:23.955 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.953+0000 7fe936d9d700 1 -- 172.21.15.59:0/1766556243 >> v2:172.21.15.110:3300/0 conn(0x7fe9380a5b70 msgr2=0x7fe9380b1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:12:23.955 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.953+0000 7fe936d9d700 1 --2- 172.21.15.59:0/1766556243 >> v2:172.21.15.110:3300/0 conn(0x7fe9380a5b70 0x7fe9380b1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:12:23.955 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.953+0000 7fe936d9d700 1 -- 172.21.15.59:0/1766556243 >> v1:172.21.15.110:6789/0 conn(0x7fe9380a5f40 legacy=0x7fe938141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:12:23.955 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.953+0000 7fe936d9d700 10 monclient: _finish_hunting 0 2023-09-30T02:12:23.955 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.953+0000 7fe936d9d700 1 monclient: found mon.noname-a 2023-09-30T02:12:23.955 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.953+0000 7fe936d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:12:23.955 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.953+0000 7fe936d9d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.59:3300/0 2023-09-30T02:12:23.955 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.953+0000 7fe936d9d700 1 -- 172.21.15.59:0/1766556243 --> v2:172.21.15.59:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9380af3f0 con 0x7fe938142290 2023-09-30T02:12:23.956 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.956+0000 7fe936d9d700 1 --2- 172.21.15.59:0/1766556243 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe938142290 0x7fe93813af30 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fe9380b49e0 tx=0x7fe928009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=714e7d836aec4efb server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:12:23.961 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.960+0000 7fe93559a700 1 -- 172.21.15.59:0/1766556243 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe92800e070 con 0x7fe938142290 2023-09-30T02:12:23.961 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.960+0000 7fe93559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:12:23.961 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.960+0000 7fe93559a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T02:12:23.961 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.960+0000 7fe93559a700 10 monclient: dump: 2023-09-30T02:12:23.961 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:12:23.961 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:12:23.961 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:12:23.961 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:12:23.961 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:12:23.962 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:12:23.962 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:12:23.962 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:12:23.962 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:12:23.962 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:12:23.962 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.960+0000 7fe93559a700 10 monclient: _finish_auth 0 2023-09-30T02:12:23.962 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.960+0000 7fe93559a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:12:23.962 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.960+0000 7fe93559a700 1 -- 172.21.15.59:0/1766556243 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe92800b990 con 0x7fe938142290 2023-09-30T02:12:23.962 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.960+0000 7fe93559a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:12:23.962 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.960+0000 7fe93559a700 1 -- 172.21.15.59:0/1766556243 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9280092e0 con 0x7fe938142290 2023-09-30T02:12:23.962 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.960+0000 7fe93c86c700 5 monclient: authenticate success, global_id 4941 2023-09-30T02:12:23.962 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.960+0000 7fe93c86c700 10 monclient: get_monmap_and_config success 2023-09-30T02:12:23.962 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.960+0000 7fe93559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:12:23.963 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.960+0000 7fe93c86c700 4 set_mon_vals no callback set 2023-09-30T02:12:23.963 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.960+0000 7fe93559a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T02:12:23.963 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.960+0000 7fe93559a700 10 monclient: dump: 2023-09-30T02:12:23.963 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:12:23.963 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:12:23.963 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:12:23.963 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:12:23.963 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:12:23.963 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:12:23.963 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:12:23.963 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:12:23.963 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:12:23.963 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:12:23.966 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.961+0000 7fe93c86c700 10 monclient: shutdown 2023-09-30T02:12:23.966 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.961+0000 7fe93c86c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:12:23.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.961+0000 7fe93c86c700 1 -- 172.21.15.59:0/1766556243 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe938142290 msgr2=0x7fe93813af30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:12:23.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.961+0000 7fe93c86c700 1 --2- 172.21.15.59:0/1766556243 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe938142290 0x7fe93813af30 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fe9380b49e0 tx=0x7fe928009840 comp rx=0 tx=0).stop 2023-09-30T02:12:23.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.961+0000 7fe93c86c700 1 -- 172.21.15.59:0/1766556243 shutdown_connections 2023-09-30T02:12:23.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.961+0000 7fe93c86c700 1 --2- 172.21.15.59:0/1766556243 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe938142290 0x7fe93813af30 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:12:23.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.961+0000 7fe93c86c700 1 --2- 172.21.15.59:0/1766556243 >> v2:172.21.15.110:3300/0 conn(0x7fe9380a5b70 0x7fe9380b1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:12:23.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.961+0000 7fe93c86c700 1 -- 172.21.15.59:0/1766556243 >> 172.21.15.59:0/1766556243 conn(0x7fe9380631f0 msgr2=0x7fe9380635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:12:23.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.961+0000 7fe93c86c700 1 -- 172.21.15.59:0/1766556243 shutdown_connections 2023-09-30T02:12:23.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.961+0000 7fe93c86c700 1 -- 172.21.15.59:0/1766556243 wait complete. 2023-09-30T02:12:23.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.962+0000 7fe93c86c700 10 monclient: build_initial_monmap 2023-09-30T02:12:23.967 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.962+0000 7fe93c86c700 10 monclient: monmap: 2023-09-30T02:12:23.967 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:12:23.967 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:12:23.968 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:12:23.968 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:12:23.968 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:12:23.968 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:12:23.968 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:12:23.968 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:12:23.968 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:12:23.968 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:12:23.968 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.962+0000 7fe93c86c700 1 librados: starting msgr at 2023-09-30T02:12:23.968 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.962+0000 7fe93c86c700 1 librados: starting objecter 2023-09-30T02:12:23.968 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.962+0000 7fe93c86c700 1 Processor -- start 2023-09-30T02:12:23.968 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.962+0000 7fe93c86c700 1 -- start start 2023-09-30T02:12:23.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.962+0000 7fe93c86c700 1 librados: setting wanted keys 2023-09-30T02:12:23.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.962+0000 7fe93c86c700 1 librados: calling monclient init 2023-09-30T02:12:23.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.962+0000 7fe93c86c700 10 monclient: init 2023-09-30T02:12:23.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.963+0000 7fe93c86c700 10 monclient: _reopen_session rank -1 2023-09-30T02:12:23.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.963+0000 7fe93c86c700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T02:12:23.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.963+0000 7fe93c86c700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe9380a5b70 0x7fe93813e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:12:23.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.963+0000 7fe93c86c700 10 monclient(hunting): picked mon.noname-c con 0x7fe9380a5b70 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:12:23.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.963+0000 7fe93c86c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9380a5f40 0x7fe93814dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:12:23.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.963+0000 7fe93c86c700 10 monclient(hunting): picked mon.noname-a con 0x7fe9380a5f40 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:12:23.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.963+0000 7fe93c86c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe938142290 0x7fe93814e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:12:23.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.963+0000 7fe93c86c700 10 monclient(hunting): picked mon.noname-b con 0x7fe938142290 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:12:23.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.963+0000 7fe93c86c700 10 monclient(hunting): start opening mon connection 2023-09-30T02:12:23.969 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.963+0000 7fe93c86c700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9380aa660 con 0x7fe9380a5f40 2023-09-30T02:12:23.970 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.963+0000 7fe93c86c700 10 monclient(hunting): start opening mon connection 2023-09-30T02:12:23.970 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.963+0000 7fe93c86c700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9380aa2d0 con 0x7fe938142290 2023-09-30T02:12:23.970 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.963+0000 7fe93c86c700 10 monclient(hunting): start opening mon connection 2023-09-30T02:12:23.970 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.963+0000 7fe93c86c700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9380aeb10 con 0x7fe9380a5b70 2023-09-30T02:12:23.970 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.963+0000 7fe93c86c700 10 monclient(hunting): _renew_subs 2023-09-30T02:12:23.970 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.963+0000 7fe93c86c700 10 monclient(hunting): authenticate will time out at 2189.266451s 2023-09-30T02:12:23.970 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.964+0000 7fe936d9d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe938142290 0x7fe93814e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:12:23.970 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.964+0000 7fe93659c700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe9380a5b70 0x7fe93813e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:12:23.970 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.964+0000 7fe936d9d700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe938142290 0x7fe93814e270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:51152/0 (socket says 172.21.15.59:51152) 2023-09-30T02:12:23.970 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.964+0000 7fe936d9d700 1 -- 172.21.15.59:0/1790104966 learned_addr learned my addr 172.21.15.59:0/1790104966 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:12:23.970 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.964+0000 7fe936d9d700 10 monclient(hunting): get_auth_request con 0x7fe938142290 auth_method 0 2023-09-30T02:12:23.970 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.964+0000 7fe936d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:12:23.971 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.964+0000 7fe936d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:12:23.971 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.964+0000 7fe936d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:12:23.971 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.964+0000 7fe93659c700 10 monclient(hunting): get_auth_request con 0x7fe9380a5b70 auth_method 0 2023-09-30T02:12:23.971 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.964+0000 7fe93659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:12:23.971 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.964+0000 7fe93659c700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:12:23.971 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.964+0000 7fe93659c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:12:23.971 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.964+0000 7fe936d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:12:23.971 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.964+0000 7fe936d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:12:23.971 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.964+0000 7fe936d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:12:23.971 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.965+0000 7fe935d9b700 1 --2- 172.21.15.59:0/1790104966 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9380a5f40 0x7fe93814dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:12:23.971 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.965+0000 7fe935d9b700 10 monclient(hunting): get_auth_request con 0x7fe9380a5f40 auth_method 0 2023-09-30T02:12:23.971 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.965+0000 7fe935d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:12:23.971 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.965+0000 7fe935d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:12:23.972 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.965+0000 7fe935d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:12:23.972 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.965+0000 7fe93659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:12:23.972 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.965+0000 7fe93659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:12:23.972 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.965+0000 7fe93659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:12:23.972 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.966+0000 7fe935d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:12:23.972 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.966+0000 7fe935d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:12:23.972 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.966+0000 7fe935d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:12:23.973 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.968+0000 7fe936d9d700 10 monclient(hunting): handle_auth_done global_id 4984 payload 824 2023-09-30T02:12:23.973 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.968+0000 7fe936d9d700 1 -- 172.21.15.59:0/1790104966 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe9380a5b70 msgr2=0x7fe93813e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:12:23.973 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.968+0000 7fe936d9d700 1 --2- 172.21.15.59:0/1790104966 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe9380a5b70 0x7fe93813e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:12:23.973 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.968+0000 7fe936d9d700 1 -- 172.21.15.59:0/1790104966 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9380a5f40 msgr2=0x7fe93814dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:12:23.973 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.968+0000 7fe936d9d700 1 --2- 172.21.15.59:0/1790104966 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9380a5f40 0x7fe93814dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:12:23.973 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.968+0000 7fe936d9d700 10 monclient: _finish_hunting 0 2023-09-30T02:12:23.973 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.968+0000 7fe936d9d700 1 monclient: found mon.noname-b 2023-09-30T02:12:23.974 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.968+0000 7fe936d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:12:23.974 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.968+0000 7fe936d9d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:12:23.974 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.968+0000 7fe936d9d700 1 -- 172.21.15.59:0/1790104966 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe928009ae0 con 0x7fe938142290 2023-09-30T02:12:23.974 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.971+0000 7fe936d9d700 1 --2- 172.21.15.59:0/1790104966 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe938142290 0x7fe93814e270 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fe9380b1ff0 tx=0x7fe928018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:12:23.974 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.971+0000 7fe92e7fc700 10 client.?.objecter ms_handle_connect 0x7fe938142290 2023-09-30T02:12:23.974 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.971+0000 7fe92e7fc700 10 client.?.objecter resend_mon_ops 2023-09-30T02:12:23.977 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.975+0000 7fe92e7fc700 1 -- 172.21.15.59:0/1790104966 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9280032f0 con 0x7fe938142290 2023-09-30T02:12:23.977 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.976+0000 7fe92e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:12:23.977 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.976+0000 7fe92e7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:12:23.977 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.976+0000 7fe92e7fc700 10 monclient: dump: 2023-09-30T02:12:23.977 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:12:23.977 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:12:23.977 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:12:23.977 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:12:23.978 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:12:23.978 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:12:23.978 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:12:23.978 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:12:23.978 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:12:23.978 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:12:23.978 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.976+0000 7fe92e7fc700 10 monclient: _finish_auth 0 2023-09-30T02:12:23.978 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.976+0000 7fe92e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:12:23.978 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.976+0000 7fe93c86c700 5 monclient: authenticate success, global_id 4984 2023-09-30T02:12:23.978 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.976+0000 7fe92e7fc700 1 -- 172.21.15.59:0/1790104966 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe928008ad0 con 0x7fe938142290 2023-09-30T02:12:23.978 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.976+0000 7fe93c86c700 10 monclient: _renew_subs 2023-09-30T02:12:23.978 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.976+0000 7fe93c86c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:12:23.979 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.976+0000 7fe93c86c700 1 -- 172.21.15.59:0/1790104966 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9380a8b80 con 0x7fe938142290 2023-09-30T02:12:23.979 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.976+0000 7fe92e7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:12:23.979 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.976+0000 7fe92e7fc700 1 -- 172.21.15.59:0/1790104966 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe928008e90 con 0x7fe938142290 2023-09-30T02:12:23.979 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.976+0000 7fe92e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:12:23.979 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.976+0000 7fe92e7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:12:23.979 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.976+0000 7fe92e7fc700 10 monclient: dump: 2023-09-30T02:12:23.979 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:12:23.979 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:12:23.979 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:12:23.979 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:12:23.979 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:12:23.979 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:12:23.979 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:12:23.980 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:12:23.980 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:12:23.980 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:12:23.980 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.976+0000 7fe934d99700 4 set_mon_vals no callback set 2023-09-30T02:12:23.980 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.976+0000 7fe93c86c700 10 client.4984.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:12:23.980 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.976+0000 7fe93c86c700 10 monclient: _renew_subs 2023-09-30T02:12:23.980 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.976+0000 7fe93c86c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:12:23.980 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.976+0000 7fe93c86c700 1 -- 172.21.15.59:0/1790104966 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9380aacb0 con 0x7fe938142290 2023-09-30T02:12:23.980 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.976+0000 7fe93c86c700 1 librados: init done 2023-09-30T02:12:23.980 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.977+0000 7fe93c86c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:12:23.980 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.977+0000 7fe93c86c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:12:23.980 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.977+0000 7fe93c86c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:12:23.981 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:23.977+0000 7fe93c86c700 1 -- 172.21.15.59:0/1790104966 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9380b1800 con 0x7fe938142290 2023-09-30T02:12:24.232 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.231+0000 7fe92e7fc700 1 -- 172.21.15.59:0/1790104966 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7fe928003040 con 0x7fe938142290 2023-09-30T02:12:24.233 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.232+0000 7fe92e7fc700 1 --2- 172.21.15.59:0/1790104966 >> v2:172.21.15.110:6800/115870 conn(0x7fe9180689e0 0x7fe91806ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:12:24.233 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.232+0000 7fe92e7fc700 1 -- 172.21.15.59:0/1790104966 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7fe928085c50 con 0x7fe938142290 2023-09-30T02:12:24.233 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.232+0000 7fe92e7fc700 10 client.4984.objecter ms_dispatch 0x7fe938001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:12:24.233 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.232+0000 7fe92e7fc700 3 client.4984.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:12:24.233 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.232+0000 7fe92e7fc700 3 client.4984.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:12:24.233 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.232+0000 7fe92e7fc700 20 client.4984.objecter dump_active .. 0 homeless 2023-09-30T02:12:24.233 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.232+0000 7fe93659c700 1 --2- 172.21.15.59:0/1790104966 >> v2:172.21.15.110:6800/115870 conn(0x7fe9180689e0 0x7fe91806ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:12:24.233 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.232+0000 7fe93659c700 10 monclient: get_auth_request con 0x7fe9180689e0 auth_method 0 2023-09-30T02:12:24.234 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.233+0000 7fe93659c700 1 --2- 172.21.15.59:0/1790104966 >> v2:172.21.15.110:6800/115870 conn(0x7fe9180689e0 0x7fe91806ae90 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe930009db0 tx=0x7fe9300091a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:12:24.234 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.233+0000 7fe92e7fc700 10 client.4984.objecter ms_handle_connect 0x7fe9180689e0 2023-09-30T02:12:24.240 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.240+0000 7fe92e7fc700 1 -- 172.21.15.59:0/1790104966 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe928085e60 con 0x7fe938142290 2023-09-30T02:12:24.241 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.240+0000 7fe92e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:12:24.241 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.240+0000 7fe92e7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:12:24.444 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.444+0000 7fe93c86c700 1 -- 172.21.15.59:0/1790104966 --> v2:172.21.15.110:6800/115870 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe93813b1b0 con 0x7fe9180689e0 2023-09-30T02:12:24.448 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.447+0000 7fe92e7fc700 1 -- 172.21.15.59:0/1790104966 <== mgr.4108 v2:172.21.15.110:6800/115870 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+70270 (secure 0 0 0) 0x7fe93813b1b0 con 0x7fe9180689e0 2023-09-30T02:12:24.449 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-09-30T02:12:24.451 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.450+0000 7fe93c86c700 10 librados: watch_flush enter 2023-09-30T02:12:24.451 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.450+0000 7fe93c86c700 10 librados: watch_flush exit 2023-09-30T02:12:24.451 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.450+0000 7fe93c86c700 20 client.4984.objecter shutdown clearing up homeless session... 2023-09-30T02:12:24.451 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.450+0000 7fe93c86c700 10 client.4984.objecter successfully canceled tick 2023-09-30T02:12:24.451 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.450+0000 7fe93c86c700 1 -- 172.21.15.59:0/1790104966 >> v2:172.21.15.110:6800/115870 conn(0x7fe9180689e0 msgr2=0x7fe91806ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:12:24.451 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.450+0000 7fe93c86c700 1 --2- 172.21.15.59:0/1790104966 >> v2:172.21.15.110:6800/115870 conn(0x7fe9180689e0 0x7fe91806ae90 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe930009db0 tx=0x7fe9300091a0 comp rx=0 tx=0).stop 2023-09-30T02:12:24.451 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.450+0000 7fe93c86c700 10 monclient: shutdown 2023-09-30T02:12:24.451 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.450+0000 7fe93c86c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:12:24.451 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.450+0000 7fe93c86c700 1 -- 172.21.15.59:0/1790104966 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe938142290 msgr2=0x7fe93814e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:12:24.451 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.450+0000 7fe93c86c700 1 --2- 172.21.15.59:0/1790104966 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe938142290 0x7fe93814e270 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fe9380b1ff0 tx=0x7fe928018000 comp rx=0 tx=0).stop 2023-09-30T02:12:24.452 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.450+0000 7fe93c86c700 1 -- 172.21.15.59:0/1790104966 shutdown_connections 2023-09-30T02:12:24.452 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.450+0000 7fe93c86c700 1 --2- 172.21.15.59:0/1790104966 >> v2:172.21.15.110:6800/115870 conn(0x7fe9180689e0 0x7fe91806ae90 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:12:24.452 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.450+0000 7fe93c86c700 1 --2- 172.21.15.59:0/1790104966 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7fe938142290 0x7fe93814e270 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:12:24.452 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.450+0000 7fe93c86c700 1 --2- 172.21.15.59:0/1790104966 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7fe9380a5b70 0x7fe93813e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:12:24.452 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.450+0000 7fe93c86c700 1 --2- 172.21.15.59:0/1790104966 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9380a5f40 0x7fe93814dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:12:24.452 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.450+0000 7fe93c86c700 1 -- 172.21.15.59:0/1790104966 >> 172.21.15.59:0/1790104966 conn(0x7fe9380631f0 msgr2=0x7fe9381380f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:12:24.452 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.450+0000 7fe93c86c700 1 -- 172.21.15.59:0/1790104966 shutdown_connections 2023-09-30T02:12:24.452 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.450+0000 7fe93c86c700 1 -- 172.21.15.59:0/1790104966 wait complete. 2023-09-30T02:12:24.452 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.450+0000 7fe93c86c700 1 librados: shutdown 2023-09-30T02:12:24.453 INFO:teuthology.orchestra.run.smithi059.stderr:dumped all 2023-09-30T02:12:24.463 INFO:teuthology.orchestra.run.smithi059.stdout:{"pg_ready":true,"pg_map":{"version":220,"stamp":"2023-09-30T02:12:23.658560+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":43088,"kb_used_data":2448,"kb_used_omap":0,"kb_used_meta":40576,"kb_avail":754931632,"statfs":{"total":773094113280,"available":773049991168,"internally_reserved":0,"allocated":2506752,"data_stored":1396000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":41549824},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002695"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-09-30T02:12:03.527118+0000","last_change":"2023-09-30T02:12:03.520743+0000","last_active":"2023-09-30T02:12:03.527118+0000","last_peered":"2023-09-30T02:12:03.527118+0000","last_clean":"2023-09-30T02:12:03.527118+0000","last_became_active":"2023-09-30T02:06:18.198153+0000","last_became_peered":"2023-09-30T02:06:18.198153+0000","last_unstale":"2023-09-30T02:12:03.527118+0000","last_undegraded":"2023-09-30T02:12:03.527118+0000","last_fullsized":"2023-09-30T02:12:03.527118+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:12:03.518230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:12:03.518230+0000","last_clean_scrub_stamp":"2023-09-30T02:12:03.518230+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T12:56:53.258283+0000","scrub_duration":0.047108065999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-09-30T02:11:42.499140+0000","last_change":"2023-09-30T02:11:42.483742+0000","last_active":"2023-09-30T02:11:42.499140+0000","last_peered":"2023-09-30T02:11:42.499140+0000","last_clean":"2023-09-30T02:11:42.499140+0000","last_became_active":"2023-09-30T02:06:18.224464+0000","last_became_peered":"2023-09-30T02:06:18.224464+0000","last_unstale":"2023-09-30T02:11:42.499140+0000","last_undegraded":"2023-09-30T02:11:42.499140+0000","last_fullsized":"2023-09-30T02:11:42.499140+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:11:42.476968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:11:42.476968+0000","last_clean_scrub_stamp":"2023-09-30T02:11:42.476968+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:41:19.177320+0000","scrub_duration":0.50626253899999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-09-30T02:11:57.649362+0000","last_change":"2023-09-30T02:11:57.648560+0000","last_active":"2023-09-30T02:11:57.649362+0000","last_peered":"2023-09-30T02:11:57.649362+0000","last_clean":"2023-09-30T02:11:57.649362+0000","last_became_active":"2023-09-30T02:06:17.855689+0000","last_became_peered":"2023-09-30T02:06:17.855689+0000","last_unstale":"2023-09-30T02:11:57.649362+0000","last_undegraded":"2023-09-30T02:11:57.649362+0000","last_fullsized":"2023-09-30T02:11:57.649362+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:11:57.645782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:11:57.645782+0000","last_clean_scrub_stamp":"2023-09-30T02:11:57.645782+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:31:54.616937+0000","scrub_duration":0.091109921999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-09-30T02:11:45.008944+0000","last_change":"2023-09-30T02:11:44.999892+0000","last_active":"2023-09-30T02:11:45.008944+0000","last_peered":"2023-09-30T02:11:45.008944+0000","last_clean":"2023-09-30T02:11:45.008944+0000","last_became_active":"2023-09-30T02:06:17.706615+0000","last_became_peered":"2023-09-30T02:06:17.706615+0000","last_unstale":"2023-09-30T02:11:45.008944+0000","last_undegraded":"2023-09-30T02:11:45.008944+0000","last_fullsized":"2023-09-30T02:11:45.008944+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:11:44.982640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:11:44.982640+0000","last_clean_scrub_stamp":"2023-09-30T02:11:44.982640+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:07:32.712429+0000","scrub_duration":0.32725594099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"25'78","reported_seq":130,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-09-30T02:11:38.810874+0000","last_change":"2023-09-30T02:11:38.802202+0000","last_active":"2023-09-30T02:11:38.810874+0000","last_peered":"2023-09-30T02:11:38.810874+0000","last_clean":"2023-09-30T02:11:38.810874+0000","last_became_active":"2023-09-30T02:06:11.592631+0000","last_became_peered":"2023-09-30T02:06:11.592631+0000","last_unstale":"2023-09-30T02:11:38.810874+0000","last_undegraded":"2023-09-30T02:11:38.810874+0000","last_fullsized":"2023-09-30T02:11:38.810874+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"25'78","last_scrub_stamp":"2023-09-30T02:11:38.789901+0000","last_deep_scrub":"25'78","last_deep_scrub_stamp":"2023-09-30T02:11:38.789901+0000","last_clean_scrub_stamp":"2023-09-30T02:11:38.789901+0000","objects_scrubbed":2,"log_size":78,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T13:15:01.381693+0000","scrub_duration":0.75921421200000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-09-30T02:11:55.862695+0000","last_change":"2023-09-30T02:11:55.861465+0000","last_active":"2023-09-30T02:11:55.862695+0000","last_peered":"2023-09-30T02:11:55.862695+0000","last_clean":"2023-09-30T02:11:55.862695+0000","last_became_active":"2023-09-30T02:06:18.091407+0000","last_became_peered":"2023-09-30T02:06:18.091407+0000","last_unstale":"2023-09-30T02:11:55.862695+0000","last_undegraded":"2023-09-30T02:11:55.862695+0000","last_fullsized":"2023-09-30T02:11:55.862695+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:11:55.852135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:11:55.852135+0000","last_clean_scrub_stamp":"2023-09-30T02:11:55.852135+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:50:23.066683+0000","scrub_duration":0.207301711,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-09-30T02:11:43.188136+0000","last_change":"2023-09-30T02:11:43.181007+0000","last_active":"2023-09-30T02:11:43.188136+0000","last_peered":"2023-09-30T02:11:43.188136+0000","last_clean":"2023-09-30T02:11:43.188136+0000","last_became_active":"2023-09-30T02:06:17.420574+0000","last_became_peered":"2023-09-30T02:06:17.420574+0000","last_unstale":"2023-09-30T02:11:43.188136+0000","last_undegraded":"2023-09-30T02:11:43.188136+0000","last_fullsized":"2023-09-30T02:11:43.188136+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:11:43.169695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:11:43.169695+0000","last_clean_scrub_stamp":"2023-09-30T02:11:43.169695+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:10:10.065495+0000","scrub_duration":0.29302578699999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":21,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-09-30T02:11:47.937481+0000","last_change":"2023-09-30T02:11:47.936716+0000","last_active":"2023-09-30T02:11:47.937481+0000","last_peered":"2023-09-30T02:11:47.937481+0000","last_clean":"2023-09-30T02:11:47.937481+0000","last_became_active":"2023-09-30T02:06:17.890282+0000","last_became_peered":"2023-09-30T02:06:17.890282+0000","last_unstale":"2023-09-30T02:11:47.937481+0000","last_undegraded":"2023-09-30T02:11:47.937481+0000","last_fullsized":"2023-09-30T02:11:47.937481+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:11:47.929243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:11:47.929243+0000","last_clean_scrub_stamp":"2023-09-30T02:11:47.929243+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T13:34:17.770214+0000","scrub_duration":0.21181633,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-09-30T02:11:56.282358+0000","last_change":"2023-09-30T02:11:56.270837+0000","last_active":"2023-09-30T02:11:56.282358+0000","last_peered":"2023-09-30T02:11:56.282358+0000","last_clean":"2023-09-30T02:11:56.282358+0000","last_became_active":"2023-09-30T02:06:17.836124+0000","last_became_peered":"2023-09-30T02:06:17.836124+0000","last_unstale":"2023-09-30T02:11:56.282358+0000","last_undegraded":"2023-09-30T02:11:56.282358+0000","last_fullsized":"2023-09-30T02:11:56.282358+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T02:11:56.261674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T02:11:56.261674+0000","last_clean_scrub_stamp":"2023-09-30T02:11:56.261674+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:22:17.480004+0000","scrub_duration":0.22799536000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":19,"seq":81604378702,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5208,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366632,"statfs":{"total":96636764160,"available":96631431168,"internally_reserved":0,"allocated":212992,"data_stored":75040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":12.859999999999999,"5min":16.204000000000001,"15min":26.364000000000001},"min":{"1min":8.8520000000000003,"5min":7.1429999999999998,"15min":7.1429999999999998},"max":{"1min":17.263000000000002,"5min":414.25400000000002,"15min":414.25400000000002},"last":9.8689999999999998},{"interface":"front","average":{"1min":12.984,"5min":17.393999999999998,"15min":29.042999999999999},"min":{"1min":7.1289999999999996,"5min":7.1289999999999996,"15min":7.1289999999999996},"max":{"1min":21.555,"5min":421.178,"15min":421.178},"last":12.888999999999999}]},{"osd":1,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":13.353,"5min":15.817,"15min":24.888999999999999},"min":{"1min":8.9610000000000003,"5min":6.8200000000000003,"15min":6.8200000000000003},"max":{"1min":16.489999999999998,"5min":326.60899999999998,"15min":326.60899999999998},"last":8.9640000000000004},{"interface":"front","average":{"1min":14.826000000000001,"5min":19.149999999999999,"15min":31.222999999999999},"min":{"1min":8.5099999999999998,"5min":8.5079999999999991,"15min":8.5079999999999991},"max":{"1min":24.559999999999999,"5min":418.35500000000002,"15min":418.35500000000002},"last":10.595000000000001}]},{"osd":2,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":16.120000000000001,"5min":21.582000000000001,"15min":33.603999999999999},"min":{"1min":9.2850000000000001,"5min":8.3960000000000008,"15min":8.3960000000000008},"max":{"1min":23.693999999999999,"5min":423.46699999999998,"15min":423.46699999999998},"last":12.599},{"interface":"front","average":{"1min":14.952,"5min":19.960999999999999,"15min":33.359999999999999},"min":{"1min":9.0749999999999993,"5min":8.4000000000000004,"15min":8.4000000000000004},"max":{"1min":23.123000000000001,"5min":438.697,"15min":438.697},"last":15.625999999999999}]},{"osd":4,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":15.225,"5min":19.898,"15min":32.472000000000001},"min":{"1min":9.5370000000000008,"5min":8.4030000000000005,"15min":8.4030000000000005},"max":{"1min":26.390000000000001,"5min":443.36399999999998,"15min":443.36399999999998},"last":25.288},{"interface":"front","average":{"1min":17.562999999999999,"5min":22.780999999999999,"15min":35.773000000000003},"min":{"1min":10.282,"5min":10.282,"15min":10.282},"max":{"1min":23.670000000000002,"5min":453.20699999999999,"15min":453.20699999999999},"last":26.356000000000002}]},{"osd":5,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":14.875,"5min":20.896999999999998,"15min":33.405000000000001},"min":{"1min":8.2430000000000003,"5min":8.141,"15min":8.141},"max":{"1min":22.359000000000002,"5min":443.94600000000003,"15min":443.94600000000003},"last":23.408000000000001},{"interface":"front","average":{"1min":19.251000000000001,"5min":22.654,"15min":32.534999999999997},"min":{"1min":11.301,"5min":10.746,"15min":10.746},"max":{"1min":25.305,"5min":335.11799999999999,"15min":335.11799999999999},"last":27.109000000000002}]},{"osd":6,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":18.507000000000001,"5min":22.931999999999999,"15min":36.216999999999999},"min":{"1min":12.955,"5min":9.4009999999999998,"15min":9.4009999999999998},"max":{"1min":28.02,"5min":432.34699999999998,"15min":432.34699999999998},"last":14.175000000000001},{"interface":"front","average":{"1min":21.754000000000001,"5min":24.712,"15min":34.631999999999998},"min":{"1min":18.292999999999999,"5min":13.018000000000001,"15min":13.018000000000001},"max":{"1min":27.356999999999999,"5min":334.178,"15min":334.178},"last":14.856999999999999}]},{"osd":7,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":17.149000000000001,"5min":23.225000000000001,"15min":35.642000000000003},"min":{"1min":10.933999999999999,"5min":10.928000000000001,"15min":10.928000000000001},"max":{"1min":23.986999999999998,"5min":461.42000000000002,"15min":461.42000000000002},"last":24.093},{"interface":"front","average":{"1min":22.254000000000001,"5min":26.744,"15min":39.024000000000001},"min":{"1min":17.257000000000001,"5min":15.686,"15min":15.686},"max":{"1min":25.774000000000001,"5min":431.27800000000002,"15min":431.27800000000002},"last":28.260000000000002}]}]},{"osd":1,"up_from":18,"seq":77309411405,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5856,"kb_used_data":600,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":94365984,"statfs":{"total":96636764160,"available":96630767616,"internally_reserved":0,"allocated":614400,"data_stored":472880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":13.101000000000001,"5min":13.603999999999999,"15min":13.84},"min":{"1min":8.7560000000000002,"5min":7.1639999999999997,"15min":7.1639999999999997},"max":{"1min":19.596,"5min":24.626999999999999,"15min":24.626999999999999},"last":11.356999999999999},{"interface":"front","average":{"1min":11.407,"5min":12.699,"15min":14.554},"min":{"1min":6.4859999999999998,"5min":6.4859999999999998,"15min":6.4859999999999998},"max":{"1min":17.785,"5min":38.213999999999999,"15min":38.213999999999999},"last":12.035}]},{"osd":2,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":12.917999999999999,"5min":13.146000000000001,"15min":13.359999999999999},"min":{"1min":8.4949999999999992,"5min":6.9509999999999996,"15min":6.9509999999999996},"max":{"1min":20.673999999999999,"5min":39.070999999999998,"15min":39.070999999999998},"last":11.68},{"interface":"front","average":{"1min":14.161,"5min":15.342000000000001,"15min":16.507999999999999},"min":{"1min":8.6050000000000004,"5min":8.6050000000000004,"15min":8.6050000000000004},"max":{"1min":20.436,"5min":28.981999999999999,"15min":28.981999999999999},"last":13.779999999999999}]},{"osd":3,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":18.742999999999999,"5min":28.812999999999999,"15min":56.773000000000003},"min":{"1min":13.654999999999999,"5min":9.2110000000000003,"15min":9.2110000000000003},"max":{"1min":23.431999999999999,"5min":789.26099999999997,"15min":789.26099999999997},"last":15.502000000000001},{"interface":"front","average":{"1min":16.210999999999999,"5min":26.545999999999999,"15min":55.965000000000003},"min":{"1min":7.8869999999999996,"5min":7.8869999999999996,"15min":7.8869999999999996},"max":{"1min":24.469000000000001,"5min":827.53899999999999,"15min":827.53899999999999},"last":16.503}]},{"osd":4,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":14.420999999999999,"5min":21.696999999999999,"15min":34.310000000000002},"min":{"1min":11.004,"5min":8.7029999999999994,"15min":8.7029999999999994},"max":{"1min":20.788,"5min":390.31700000000001,"15min":390.31700000000001},"last":18.241},{"interface":"front","average":{"1min":17.638000000000002,"5min":21.739000000000001,"15min":34.366999999999997},"min":{"1min":10.234,"5min":10.234,"15min":10.234},"max":{"1min":24.907,"5min":381.255,"15min":381.255},"last":17.436}]},{"osd":5,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":13.234999999999999,"5min":14.538,"15min":15.808999999999999},"min":{"1min":9.2680000000000007,"5min":7.6849999999999996,"15min":7.6849999999999996},"max":{"1min":21.724,"5min":31.677,"15min":31.677},"last":21.780999999999999},{"interface":"front","average":{"1min":20.417000000000002,"5min":20.757000000000001,"15min":21.004999999999999},"min":{"1min":9.9860000000000007,"5min":9.9860000000000007,"15min":9.9860000000000007},"max":{"1min":26.379000000000001,"5min":31.449999999999999,"15min":31.449999999999999},"last":18.978999999999999}]},{"osd":6,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":18.373999999999999,"5min":19.23,"15min":21.132000000000001},"min":{"1min":12.045,"5min":9.7560000000000002,"15min":9.7560000000000002},"max":{"1min":24.495999999999999,"5min":41.316000000000003,"15min":41.316000000000003},"last":23.524999999999999},{"interface":"front","average":{"1min":16.523,"5min":17.629999999999999,"15min":19.443999999999999},"min":{"1min":10.407,"5min":9.2699999999999996,"15min":9.2699999999999996},"max":{"1min":22.161999999999999,"5min":53.012,"15min":53.012},"last":22.812000000000001}]},{"osd":7,"last update":"Sat Sep 30 02:11:22 2023","interfaces":[{"interface":"back","average":{"1min":19.762,"5min":24.777999999999999,"15min":37.265999999999998},"min":{"1min":14.295999999999999,"5min":9.5310000000000006,"15min":9.5310000000000006},"max":{"1min":23.609000000000002,"5min":332.072,"15min":332.072},"last":25.079000000000001},{"interface":"front","average":{"1min":17.829000000000001,"5min":20.178999999999998,"15min":24.689},"min":{"1min":11.316000000000001,"5min":8.141,"15min":8.141},"max":{"1min":23.631,"5min":108.00700000000001,"15min":108.00700000000001},"last":26.065000000000001}]}]},{"osd":7,"up_from":14,"seq":60129542223,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5272,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":94366568,"statfs":{"total":96636764160,"available":96631365632,"internally_reserved":0,"allocated":212992,"data_stored":75040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:11:21 2023","interfaces":[{"interface":"back","average":{"1min":8.6750000000000007,"5min":8.9369999999999994,"15min":8.9199999999999999},"min":{"1min":6.2140000000000004,"5min":5.3120000000000003,"15min":5.3120000000000003},"max":{"1min":16.916,"5min":19.491,"15min":19.491},"last":6.9950000000000001},{"interface":"front","average":{"1min":10.153,"5min":9.9879999999999995,"15min":9.7560000000000002},"min":{"1min":7.3099999999999996,"5min":6.2850000000000001,"15min":6.2850000000000001},"max":{"1min":19.263000000000002,"5min":20.952000000000002,"15min":20.952000000000002},"last":9.5060000000000002}]},{"osd":1,"last update":"Sat Sep 30 02:11:21 2023","interfaces":[{"interface":"back","average":{"1min":10.753,"5min":11.709,"15min":14.192},"min":{"1min":6.6630000000000003,"5min":6.6630000000000003,"15min":6.6630000000000003},"max":{"1min":18.591000000000001,"5min":89.454999999999998,"15min":89.454999999999998},"last":10.347},{"interface":"front","average":{"1min":11.548999999999999,"5min":12.834,"15min":14.573},"min":{"1min":8.0909999999999993,"5min":7.0700000000000003,"15min":7.0700000000000003},"max":{"1min":19.093,"5min":83.695999999999998,"15min":83.695999999999998},"last":14.532}]},{"osd":2,"last update":"Sat Sep 30 02:11:21 2023","interfaces":[{"interface":"back","average":{"1min":13.856,"5min":13.919,"15min":13.823},"min":{"1min":8.1240000000000006,"5min":7.4669999999999996,"15min":7.4669999999999996},"max":{"1min":20.259,"5min":21.370999999999999,"15min":21.370999999999999},"last":12.955},{"interface":"front","average":{"1min":14.368,"5min":14.385,"15min":15.127000000000001},"min":{"1min":10.176,"5min":7.8920000000000003,"15min":7.8920000000000003},"max":{"1min":19.542999999999999,"5min":21.712,"15min":21.712},"last":11.683}]},{"osd":3,"last update":"Sat Sep 30 02:11:21 2023","interfaces":[{"interface":"back","average":{"1min":15.417999999999999,"5min":15.926,"15min":15.646000000000001},"min":{"1min":11.319000000000001,"5min":7.1710000000000003,"15min":7.1710000000000003},"max":{"1min":21.238,"5min":24.276,"15min":24.276},"last":16.811},{"interface":"front","average":{"1min":16.335999999999999,"5min":16.087,"15min":16.295000000000002},"min":{"1min":12.477,"5min":7.9630000000000001,"15min":7.9630000000000001},"max":{"1min":22.021999999999998,"5min":24.716000000000001,"15min":24.716000000000001},"last":13.942}]},{"osd":4,"last update":"Sat Sep 30 02:12:12 2023","interfaces":[{"interface":"back","average":{"1min":17.257000000000001,"5min":17.029,"15min":18.841999999999999},"min":{"1min":11.545999999999999,"5min":8.5190000000000001,"15min":5.7039999999999997},"max":{"1min":21.815999999999999,"5min":25.271999999999998,"15min":99.298000000000002},"last":16.123999999999999},{"interface":"front","average":{"1min":14.353,"5min":14.266,"15min":17.119},"min":{"1min":10.654999999999999,"5min":7.1509999999999998,"15min":7.1509999999999998},"max":{"1min":20.363,"5min":29.454999999999998,"15min":86.938999999999993},"last":19.969999999999999}]},{"osd":5,"last update":"Sat Sep 30 02:12:12 2023","interfaces":[{"interface":"back","average":{"1min":17.873999999999999,"5min":16.611999999999998,"15min":19.686},"min":{"1min":9.2620000000000005,"5min":7.7240000000000002,"15min":7.6050000000000004},"max":{"1min":22.972999999999999,"5min":24.268000000000001,"15min":118.675},"last":20.707000000000001},{"interface":"front","average":{"1min":19.183,"5min":18.733000000000001,"15min":21.452000000000002},"min":{"1min":14.173,"5min":8.7330000000000005,"15min":8.7330000000000005},"max":{"1min":26.516999999999999,"5min":26.516999999999999,"15min":114.14400000000001},"last":18.989000000000001}]},{"osd":6,"last update":"Sat Sep 30 02:12:12 2023","interfaces":[{"interface":"back","average":{"1min":20.350000000000001,"5min":19.651,"15min":22.760000000000002},"min":{"1min":17.539000000000001,"5min":8.4359999999999999,"15min":8.4359999999999999},"max":{"1min":25.138999999999999,"5min":26.102,"15min":111.048},"last":18.084},{"interface":"front","average":{"1min":18.172999999999998,"5min":18.350000000000001,"15min":21.824000000000002},"min":{"1min":11.686,"5min":9.8140000000000001,"15min":9.8140000000000001},"max":{"1min":23.251999999999999,"5min":25.498000000000001,"15min":115.241},"last":21.899999999999999}]}]},{"osd":0,"up_from":18,"seq":77309411406,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5208,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366632,"statfs":{"total":96636764160,"available":96631431168,"internally_reserved":0,"allocated":212992,"data_stored":75040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 30 02:12:19 2023","interfaces":[{"interface":"back","average":{"1min":12.506,"5min":11.57,"15min":27.613},"min":{"1min":6.6360000000000001,"5min":6.6360000000000001,"15min":6.6360000000000001},"max":{"1min":21.010000000000002,"5min":24.495000000000001,"15min":425.49599999999998},"last":7.4189999999999996},{"interface":"front","average":{"1min":13.44,"5min":13.093999999999999,"15min":37.469999999999999},"min":{"1min":8.9760000000000009,"5min":8.0009999999999994,"15min":7.9269999999999996},"max":{"1min":23.251999999999999,"5min":24.334,"15min":667.81100000000004},"last":12.294}]},{"osd":2,"last update":"Sat Sep 30 02:12:19 2023","interfaces":[{"interface":"back","average":{"1min":13.605,"5min":13.029,"15min":18.427},"min":{"1min":6.8780000000000001,"5min":6.8780000000000001,"15min":6.8780000000000001},"max":{"1min":24.388000000000002,"5min":25.501000000000001,"15min":162.50700000000001},"last":6.8780000000000001},{"interface":"front","average":{"1min":14.896000000000001,"5min":16.478999999999999,"15min":20.463000000000001},"min":{"1min":10.308999999999999,"5min":8.5079999999999991,"15min":8.5079999999999991},"max":{"1min":21.905000000000001,"5min":27.933,"15min":131.863},"last":13.94}]},{"osd":3,"last update":"Sat Sep 30 02:12:19 2023","interfaces":[{"interface":"back","average":{"1min":17.951000000000001,"5min":15.247999999999999,"15min":15.432},"min":{"1min":7.7300000000000004,"5min":7.4859999999999998,"15min":7.4859999999999998},"max":{"1min":70.129999999999995,"5min":70.129999999999995,"15min":70.129999999999995},"last":13.028},{"interface":"front","average":{"1min":20.867000000000001,"5min":18.678999999999998,"15min":18.905999999999999},"min":{"1min":11.114000000000001,"5min":9.8569999999999993,"15min":9.8569999999999993},"max":{"1min":68.533000000000001,"5min":68.533000000000001,"15min":68.533000000000001},"last":17.231999999999999}]},{"osd":4,"last update":"Sat Sep 30 02:12:19 2023","interfaces":[{"interface":"back","average":{"1min":16.994,"5min":15.326000000000001,"15min":19.013000000000002},"min":{"1min":10.551,"5min":9.0700000000000003,"15min":9.0700000000000003},"max":{"1min":22.936,"5min":22.936,"15min":102.456},"last":20.806999999999999},{"interface":"front","average":{"1min":14.218,"5min":15.561999999999999,"15min":20.363},"min":{"1min":9.3130000000000006,"5min":8.7929999999999993,"15min":8.7929999999999993},"max":{"1min":19.337,"5min":24.814,"15min":98.293999999999997},"last":17.966000000000001}]},{"osd":5,"last update":"Sat Sep 30 02:12:19 2023","interfaces":[{"interface":"back","average":{"1min":18.736000000000001,"5min":19.507999999999999,"15min":22.5},"min":{"1min":10.91,"5min":10.726000000000001,"15min":10.726000000000001},"max":{"1min":25.850999999999999,"5min":28.995000000000001,"15min":79.566999999999993},"last":23.338999999999999},{"interface":"front","average":{"1min":16.315000000000001,"5min":17.263999999999999,"15min":21.027999999999999},"min":{"1min":8.8889999999999993,"5min":8.8889999999999993,"15min":8.8889999999999993},"max":{"1min":22.468,"5min":26.265000000000001,"15min":109.80500000000001},"last":16.244}]},{"osd":6,"last update":"Sat Sep 30 02:12:19 2023","interfaces":[{"interface":"back","average":{"1min":18.645,"5min":18.314,"15min":25.378},"min":{"1min":10.394,"5min":9.2829999999999995,"15min":9.2829999999999995},"max":{"1min":23.396000000000001,"5min":24.276,"15min":161.01499999999999},"last":19.971},{"interface":"front","average":{"1min":21.626999999999999,"5min":21.736999999999998,"15min":25.789999999999999},"min":{"1min":17.408999999999999,"5min":13.163,"15min":13.163},"max":{"1min":25.757000000000001,"5min":29.507999999999999,"15min":118.71599999999999},"last":21.524000000000001}]},{"osd":7,"last update":"Sat Sep 30 02:12:19 2023","interfaces":[{"interface":"back","average":{"1min":18.904,"5min":19.167999999999999,"15min":23.074000000000002},"min":{"1min":13.791,"5min":11.201000000000001,"15min":9.8810000000000002},"max":{"1min":24.332999999999998,"5min":27.186,"15min":111.86},"last":14.800000000000001},{"interface":"front","average":{"1min":21.521000000000001,"5min":21.128,"15min":25.591000000000001},"min":{"1min":16.637,"5min":10.237,"15min":10.237},"max":{"1min":25.135000000000002,"5min":29.968,"15min":126.889},"last":19.234000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574926,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5208,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366632,"statfs":{"total":96636764160,"available":96631431168,"internally_reserved":0,"allocated":212992,"data_stored":75040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:11:20 2023","interfaces":[{"interface":"back","average":{"1min":9.6969999999999992,"5min":9.8569999999999993,"15min":9.7870000000000008},"min":{"1min":6.5830000000000002,"5min":5.774,"15min":5.774},"max":{"1min":16.530999999999999,"5min":19.709,"15min":19.709},"last":8.6920000000000002},{"interface":"front","average":{"1min":10.919,"5min":10.390000000000001,"15min":11.138999999999999},"min":{"1min":6.1239999999999997,"5min":5.0419999999999998,"15min":5.0419999999999998},"max":{"1min":19.603000000000002,"5min":23.134,"15min":23.134},"last":7.8209999999999997}]},{"osd":1,"last update":"Sat Sep 30 02:11:20 2023","interfaces":[{"interface":"back","average":{"1min":11.829000000000001,"5min":12.981999999999999,"15min":14.561999999999999},"min":{"1min":7.351,"5min":6.3010000000000002,"15min":6.3010000000000002},"max":{"1min":17.946999999999999,"5min":27.242999999999999,"15min":27.242999999999999},"last":11.523999999999999},{"interface":"front","average":{"1min":11.532999999999999,"5min":11.537000000000001,"15min":12.59},"min":{"1min":7.9770000000000003,"5min":7.4989999999999997,"15min":7.4989999999999997},"max":{"1min":18.957000000000001,"5min":24.388999999999999,"15min":24.388999999999999},"last":11.855}]},{"osd":2,"last update":"Sat Sep 30 02:11:20 2023","interfaces":[{"interface":"back","average":{"1min":14.006,"5min":14.630000000000001,"15min":15.132},"min":{"1min":9.6690000000000005,"5min":7.7039999999999997,"15min":7.7039999999999997},"max":{"1min":19.521000000000001,"5min":24.510999999999999,"15min":24.510999999999999},"last":13.590999999999999},{"interface":"front","average":{"1min":12.595000000000001,"5min":13.534000000000001,"15min":14.086},"min":{"1min":7.9699999999999998,"5min":7.9699999999999998,"15min":7.9699999999999998},"max":{"1min":20.085000000000001,"5min":25.033999999999999,"15min":25.033999999999999},"last":12.869}]},{"osd":3,"last update":"Sat Sep 30 02:11:20 2023","interfaces":[{"interface":"back","average":{"1min":14.676,"5min":19.765999999999998,"15min":31.925000000000001},"min":{"1min":8.3770000000000007,"5min":8.2089999999999996,"15min":8.2089999999999996},"max":{"1min":20.157,"5min":540.52099999999996,"15min":540.52099999999996},"last":17.189},{"interface":"front","average":{"1min":15.218,"5min":21.224,"15min":35.917999999999999},"min":{"1min":9.4489999999999998,"5min":7.9649999999999999,"15min":7.9649999999999999},"max":{"1min":19.452999999999999,"5min":644.31600000000003,"15min":644.31600000000003},"last":14.659000000000001}]},{"osd":4,"last update":"Sat Sep 30 02:12:19 2023","interfaces":[{"interface":"back","average":{"1min":16.222999999999999,"5min":17.009,"15min":22.004999999999999},"min":{"1min":9.577,"5min":9.4860000000000007,"15min":9.0969999999999995},"max":{"1min":23.646000000000001,"5min":25.888000000000002,"15min":179.749},"last":17.478999999999999},{"interface":"front","average":{"1min":13.401,"5min":13.941000000000001,"15min":19.25},"min":{"1min":7.5209999999999999,"5min":7.5209999999999999,"15min":7.5209999999999999},"max":{"1min":19.59,"5min":22.443000000000001,"15min":174.52799999999999},"last":19.59}]},{"osd":5,"last update":"Sat Sep 30 02:12:19 2023","interfaces":[{"interface":"back","average":{"1min":17.376000000000001,"5min":16.928999999999998,"15min":16.992999999999999},"min":{"1min":13.23,"5min":8.2379999999999995,"15min":8.2379999999999995},"max":{"1min":23.315999999999999,"5min":23.315999999999999,"15min":36.709000000000003},"last":18.513000000000002},{"interface":"front","average":{"1min":16.379999999999999,"5min":16.184000000000001,"15min":16.533999999999999},"min":{"1min":9.1769999999999996,"5min":8.7889999999999997,"15min":5.0279999999999996},"max":{"1min":23.228999999999999,"5min":23.228999999999999,"15min":27.364999999999998},"last":22.568999999999999}]},{"osd":7,"last update":"Sat Sep 30 02:12:14 2023","interfaces":[{"interface":"back","average":{"1min":17.745999999999999,"5min":18.387,"15min":21.994},"min":{"1min":8.7899999999999991,"5min":8.7899999999999991,"15min":8.7899999999999991},"max":{"1min":24.399000000000001,"5min":24.399000000000001,"15min":105.801},"last":21.148},{"interface":"front","average":{"1min":18.082999999999998,"5min":18.442,"15min":20.709},"min":{"1min":13.851000000000001,"5min":7.3120000000000003,"15min":7.3120000000000003},"max":{"1min":20.382000000000001,"5min":27.594999999999999,"15min":90.156000000000006},"last":20.849}]}]},{"osd":4,"up_from":15,"seq":64424509519,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5208,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366632,"statfs":{"total":96636764160,"available":96631431168,"internally_reserved":0,"allocated":212992,"data_stored":75040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:12:20 2023","interfaces":[{"interface":"back","average":{"1min":10.965999999999999,"5min":10.984999999999999,"15min":9.5790000000000006},"min":{"1min":6.7789999999999999,"5min":5.7619999999999996,"15min":5.2699999999999996},"max":{"1min":21.155000000000001,"5min":24.532,"15min":24.532},"last":8.5239999999999991},{"interface":"front","average":{"1min":10.455,"5min":10.584,"15min":11.795999999999999},"min":{"1min":5.915,"5min":5.6180000000000003,"15min":5.4530000000000003},"max":{"1min":17.823,"5min":21.841000000000001,"15min":25.225000000000001},"last":14.416}]},{"osd":1,"last update":"Sat Sep 30 02:12:20 2023","interfaces":[{"interface":"back","average":{"1min":12.978999999999999,"5min":13.025,"15min":13.815},"min":{"1min":7.907,"5min":6.2830000000000004,"15min":6.2830000000000004},"max":{"1min":20.052,"5min":20.052,"15min":28.917000000000002},"last":15.273999999999999},{"interface":"front","average":{"1min":11.965,"5min":11.119,"15min":11.781000000000001},"min":{"1min":8.3059999999999992,"5min":7.0250000000000004,"15min":7.0250000000000004},"max":{"1min":18.099,"5min":21.760999999999999,"15min":30.922999999999998},"last":17.969000000000001}]},{"osd":2,"last update":"Sat Sep 30 02:12:20 2023","interfaces":[{"interface":"back","average":{"1min":13.457000000000001,"5min":13.375,"15min":14.319000000000001},"min":{"1min":7.6050000000000004,"5min":7.1609999999999996,"15min":7.1609999999999996},"max":{"1min":20.756,"5min":20.756,"15min":21.175000000000001},"last":10.067},{"interface":"front","average":{"1min":13.403,"5min":13.571,"15min":14.462999999999999},"min":{"1min":9.125,"5min":6.7060000000000004,"15min":6.7060000000000004},"max":{"1min":19.635999999999999,"5min":22.277999999999999,"15min":27.533999999999999},"last":19.635999999999999}]},{"osd":3,"last update":"Sat Sep 30 02:11:23 2023","interfaces":[{"interface":"back","average":{"1min":16.146000000000001,"5min":16.042999999999999,"15min":15.887},"min":{"1min":9.4719999999999995,"5min":9.4719999999999995,"15min":9.4719999999999995},"max":{"1min":23.526,"5min":27.632999999999999,"15min":27.632999999999999},"last":17.149000000000001},{"interface":"front","average":{"1min":15.352,"5min":16.143999999999998,"15min":17.184999999999999},"min":{"1min":8.2599999999999998,"5min":7.9269999999999996,"15min":7.9269999999999996},"max":{"1min":23.113,"5min":30.495000000000001,"15min":30.495000000000001},"last":17.553000000000001}]},{"osd":5,"last update":"Sat Sep 30 02:12:15 2023","interfaces":[{"interface":"back","average":{"1min":18.210000000000001,"5min":17.216999999999999,"15min":20.684999999999999},"min":{"1min":10.670999999999999,"5min":9.1150000000000002,"15min":7.9080000000000004},"max":{"1min":35.441000000000003,"5min":35.441000000000003,"15min":99.784000000000006},"last":10.33},{"interface":"front","average":{"1min":15.345000000000001,"5min":14.353999999999999,"15min":17.359999999999999},"min":{"1min":6.8380000000000001,"5min":6.8380000000000001,"15min":6.8380000000000001},"max":{"1min":34.255000000000003,"5min":34.255000000000003,"15min":98.634},"last":21.065999999999999}]},{"osd":6,"last update":"Sat Sep 30 02:12:15 2023","interfaces":[{"interface":"back","average":{"1min":16.445,"5min":16.413,"15min":16.905999999999999},"min":{"1min":9.6549999999999994,"5min":7.8949999999999996,"15min":7.8949999999999996},"max":{"1min":23.527000000000001,"5min":23.527000000000001,"15min":27.277999999999999},"last":11.698},{"interface":"front","average":{"1min":15.863,"5min":16.41,"15min":16.085000000000001},"min":{"1min":11.712,"5min":8.391,"15min":6.5810000000000004},"max":{"1min":22.213000000000001,"5min":28.640000000000001,"15min":28.640000000000001},"last":22.34}]},{"osd":7,"last update":"Sat Sep 30 02:12:15 2023","interfaces":[{"interface":"back","average":{"1min":18.379000000000001,"5min":18.099,"15min":18.492999999999999},"min":{"1min":10.675000000000001,"5min":7.79,"15min":7.79},"max":{"1min":25.411999999999999,"5min":29.436,"15min":31.143000000000001},"last":13.535},{"interface":"front","average":{"1min":17.527999999999999,"5min":17.739000000000001,"15min":18.489000000000001},"min":{"1min":8.1739999999999995,"5min":8.1739999999999995,"15min":8.1739999999999995},"max":{"1min":23.006,"5min":25.350000000000001,"15min":30.529},"last":12.696}]}]},{"osd":5,"up_from":15,"seq":64424509519,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5920,"kb_used_data":600,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":94365920,"statfs":{"total":96636764160,"available":96630702080,"internally_reserved":0,"allocated":614400,"data_stored":472880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:12:19 2023","interfaces":[{"interface":"back","average":{"1min":13.289,"5min":10.359,"15min":11.827999999999999},"min":{"1min":5.851,"5min":5.7350000000000003,"15min":5.7350000000000003},"max":{"1min":22.815999999999999,"5min":22.815999999999999,"15min":22.815999999999999},"last":5.851},{"interface":"front","average":{"1min":11.586,"5min":10.646000000000001,"15min":11.523},"min":{"1min":7.7060000000000004,"5min":5.8879999999999999,"15min":5.8879999999999999},"max":{"1min":16.501999999999999,"5min":20.34,"15min":22.821999999999999},"last":10.151}]},{"osd":1,"last update":"Sat Sep 30 02:12:19 2023","interfaces":[{"interface":"back","average":{"1min":16.177,"5min":13.888999999999999,"15min":15.214},"min":{"1min":7.8220000000000001,"5min":7.6710000000000003,"15min":7.29},"max":{"1min":24.887,"5min":24.887,"15min":47.451999999999998},"last":9.7159999999999993},{"interface":"front","average":{"1min":13.401,"5min":11.612,"15min":12.58},"min":{"1min":7.718,"5min":7.1399999999999997,"15min":6.6130000000000004},"max":{"1min":22.173999999999999,"5min":25.352,"15min":48.841999999999999},"last":15.616}]},{"osd":2,"last update":"Sat Sep 30 02:12:19 2023","interfaces":[{"interface":"back","average":{"1min":16.187999999999999,"5min":14.494,"15min":12.179},"min":{"1min":11.064,"5min":6.8250000000000002,"15min":6.8250000000000002},"max":{"1min":21.376000000000001,"5min":27.501000000000001,"15min":27.501000000000001},"last":12.042999999999999},{"interface":"front","average":{"1min":12.847,"5min":14.08,"15min":15.308},"min":{"1min":8.6630000000000003,"5min":7.5369999999999999,"15min":7.5369999999999999},"max":{"1min":18.440000000000001,"5min":25.702999999999999,"15min":26.576000000000001},"last":12.316000000000001}]},{"osd":3,"last update":"Sat Sep 30 02:11:30 2023","interfaces":[{"interface":"back","average":{"1min":14.194000000000001,"5min":15.137,"15min":15.063000000000001},"min":{"1min":7.6980000000000004,"5min":7.2619999999999996,"15min":7.2619999999999996},"max":{"1min":21.891999999999999,"5min":26.895,"15min":26.895},"last":18.074999999999999},{"interface":"front","average":{"1min":15.656000000000001,"5min":17.158999999999999,"15min":17.841999999999999},"min":{"1min":11.603,"5min":11.140000000000001,"15min":11.140000000000001},"max":{"1min":19.952000000000002,"5min":24.702999999999999,"15min":24.702999999999999},"last":18.914999999999999}]},{"osd":4,"last update":"Sat Sep 30 02:12:19 2023","interfaces":[{"interface":"back","average":{"1min":18.436,"5min":17.344000000000001,"15min":18.109999999999999},"min":{"1min":12.776,"5min":8.8160000000000007,"15min":8.8160000000000007},"max":{"1min":23.954000000000001,"5min":26.710000000000001,"15min":26.710000000000001},"last":17.114000000000001},{"interface":"front","average":{"1min":16.672999999999998,"5min":16.448,"15min":15.57},"min":{"1min":12.374000000000001,"5min":8.8149999999999995,"15min":6.0270000000000001},"max":{"1min":20.652000000000001,"5min":28.344999999999999,"15min":28.344999999999999},"last":20.652000000000001}]},{"osd":6,"last update":"Sat Sep 30 02:12:19 2023","interfaces":[{"interface":"back","average":{"1min":18.030999999999999,"5min":16.91,"15min":15.222},"min":{"1min":13.343999999999999,"5min":8.3729999999999993,"15min":7.0679999999999996},"max":{"1min":26.884,"5min":26.884,"15min":26.884},"last":19.763999999999999},{"interface":"front","average":{"1min":15.470000000000001,"5min":15.948,"15min":16.527000000000001},"min":{"1min":8.8140000000000001,"5min":7.5540000000000003,"15min":7.5540000000000003},"max":{"1min":22.797000000000001,"5min":23.09,"15min":27.763999999999999},"last":14.231}]},{"osd":7,"last update":"Sat Sep 30 02:12:19 2023","interfaces":[{"interface":"back","average":{"1min":19.975999999999999,"5min":18.206,"15min":19.585999999999999},"min":{"1min":13.6,"5min":10.584,"15min":10.584},"max":{"1min":25.545999999999999,"5min":27.739999999999998,"15min":28.951000000000001},"last":13.6},{"interface":"front","average":{"1min":17.044,"5min":17.120999999999999,"15min":16.146999999999998},"min":{"1min":9.0079999999999991,"5min":7.0490000000000004,"15min":7.0490000000000004},"max":{"1min":24.77,"5min":25.151,"15min":25.151},"last":14.890000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411405,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5208,"kb_used_data":208,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366632,"statfs":{"total":96636764160,"available":96631431168,"internally_reserved":0,"allocated":212992,"data_stored":75040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 02:11:21 2023","interfaces":[{"interface":"back","average":{"1min":12.927,"5min":13.202999999999999,"15min":13.711},"min":{"1min":7.1929999999999996,"5min":6.665,"15min":6.665},"max":{"1min":20.434000000000001,"5min":69.003,"15min":69.003},"last":10.054},{"interface":"front","average":{"1min":13.492000000000001,"5min":14.566000000000001,"15min":16.558},"min":{"1min":8.6880000000000006,"5min":8.1329999999999991,"15min":8.1329999999999991},"max":{"1min":25.254000000000001,"5min":91.331999999999994,"15min":91.331999999999994},"last":11.542}]},{"osd":1,"last update":"Sat Sep 30 02:11:21 2023","interfaces":[{"interface":"back","average":{"1min":14.648999999999999,"5min":16.484999999999999,"15min":20.882000000000001},"min":{"1min":9.4800000000000004,"5min":7.9630000000000001,"15min":7.9630000000000001},"max":{"1min":21.640999999999998,"5min":145.952,"15min":145.952},"last":13.590999999999999},{"interface":"front","average":{"1min":15.581,"5min":17.215,"15min":21.452000000000002},"min":{"1min":9.6679999999999993,"5min":8.7469999999999999,"15min":8.7469999999999999},"max":{"1min":25.678000000000001,"5min":155.72900000000001,"15min":155.72900000000001},"last":16.552}]},{"osd":3,"last update":"Sat Sep 30 02:11:21 2023","interfaces":[{"interface":"back","average":{"1min":16.968,"5min":16.724,"15min":17.524000000000001},"min":{"1min":10.601000000000001,"5min":10.124000000000001,"15min":10.124000000000001},"max":{"1min":32.027000000000001,"5min":39.378999999999998,"15min":39.378999999999998},"last":22.370999999999999},{"interface":"front","average":{"1min":17.798999999999999,"5min":18.437000000000001,"15min":18.405000000000001},"min":{"1min":8.9250000000000007,"5min":8.9250000000000007,"15min":8.9250000000000007},"max":{"1min":24.704999999999998,"5min":36.649000000000001,"15min":36.649000000000001},"last":19.388000000000002}]},{"osd":4,"last update":"Sat Sep 30 02:11:21 2023","interfaces":[{"interface":"back","average":{"1min":18.395,"5min":18.151,"15min":18.896999999999998},"min":{"1min":11.792999999999999,"5min":8.9359999999999999,"15min":8.9359999999999999},"max":{"1min":29.699000000000002,"5min":39.186999999999998,"15min":39.186999999999998},"last":17.277000000000001},{"interface":"front","average":{"1min":11.427,"5min":11.253,"15min":12.537000000000001},"min":{"1min":6.7329999999999997,"5min":6.7329999999999997,"15min":6.7329999999999997},"max":{"1min":23.036999999999999,"5min":31.32,"15min":31.32},"last":8.9949999999999992}]},{"osd":5,"last update":"Sat Sep 30 02:11:21 2023","interfaces":[{"interface":"back","average":{"1min":19.495000000000001,"5min":18.518999999999998,"15min":19.271000000000001},"min":{"1min":12.682,"5min":8.3499999999999996,"15min":8.3499999999999996},"max":{"1min":26.695,"5min":43.409999999999997,"15min":43.409999999999997},"last":18.638000000000002},{"interface":"front","average":{"1min":20.788,"5min":21.66,"15min":22.609999999999999},"min":{"1min":11.974,"5min":11.974,"15min":11.974},"max":{"1min":33.695999999999998,"5min":40.646000000000001,"15min":40.646000000000001},"last":21.637}]},{"osd":6,"last update":"Sat Sep 30 02:11:21 2023","interfaces":[{"interface":"back","average":{"1min":13.474,"5min":14.634,"15min":15.917999999999999},"min":{"1min":9.8520000000000003,"5min":7.4829999999999997,"15min":7.4829999999999997},"max":{"1min":21.18,"5min":34.045000000000002,"15min":34.045000000000002},"last":12.395},{"interface":"front","average":{"1min":19.893999999999998,"5min":19.949000000000002,"15min":21.067},"min":{"1min":12.353999999999999,"5min":9.1600000000000001,"15min":9.1600000000000001},"max":{"1min":24.425000000000001,"5min":47.997999999999998,"15min":47.997999999999998},"last":15.058999999999999}]},{"osd":7,"last update":"Sat Sep 30 02:11:21 2023","interfaces":[{"interface":"back","average":{"1min":21.699999999999999,"5min":22.599,"15min":24.295000000000002},"min":{"1min":15.781000000000001,"5min":13.542999999999999,"15min":13.542999999999999},"max":{"1min":31.061,"5min":44.585000000000001,"15min":44.585000000000001},"last":20.190999999999999},{"interface":"front","average":{"1min":15.755000000000001,"5min":16.690999999999999,"15min":17.437000000000001},"min":{"1min":8.8230000000000004,"5min":7.6600000000000001,"15min":7.6600000000000001},"max":{"1min":26.234000000000002,"5min":38.295000000000002,"15min":38.295000000000002},"last":14.27}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T02:12:24.466 DEBUG:teuthology.orchestra.run.smithi059:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-09-30T02:12:24.604 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.603+0000 7f2bb225c700 10 monclient: get_monmap_and_config 2023-09-30T02:12:24.604 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.603+0000 7f2bb225c700 10 monclient: build_initial_monmap 2023-09-30T02:12:24.604 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.603+0000 7f2bb225c700 10 monclient: monmap: 2023-09-30T02:12:24.604 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:12:24.604 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:12:24.604 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:12:24.605058+0000 2023-09-30T02:12:24.604 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:12:24.605058+0000 2023-09-30T02:12:24.605 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:12:24.605 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:12:24.605 INFO:teuthology.orchestra.run.smithi059.stderr:0: v1:172.21.15.59:6789/0 mon.noname-a-legacy 2023-09-30T02:12:24.605 INFO:teuthology.orchestra.run.smithi059.stderr:1: v1:172.21.15.110:6789/0 mon.noname-c-legacy 2023-09-30T02:12:24.605 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-b 2023-09-30T02:12:24.605 INFO:teuthology.orchestra.run.smithi059.stderr:3: v2:172.21.15.59:3300/0 mon.noname-a 2023-09-30T02:12:24.605 INFO:teuthology.orchestra.run.smithi059.stderr:4: v2:172.21.15.110:3300/0 mon.noname-c 2023-09-30T02:12:24.605 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:12:24.607 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.604+0000 7f2bb225c700 1 Processor -- start 2023-09-30T02:12:24.607 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.604+0000 7f2bb225c700 1 -- start start 2023-09-30T02:12:24.607 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.604+0000 7f2bb225c700 10 monclient: init 2023-09-30T02:12:24.607 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.605+0000 7f2bb225c700 10 monclient: _reopen_session rank -1 2023-09-30T02:12:24.607 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.605+0000 7f2bb225c700 10 monclient: _add_conns ranks=[4,1,3,0,2] 2023-09-30T02:12:24.607 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.605+0000 7f2bb225c700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f2bac0a5b70 0x7f2bac0b1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:12:24.607 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.605+0000 7f2bb225c700 10 monclient(hunting): picked mon.noname-c con 0x7f2bac0a5b70 addr v2:172.21.15.110:3300/0 2023-09-30T02:12:24.607 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.605+0000 7f2bb225c700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2bac0a5f40 addr v1:172.21.15.110:6789/0 2023-09-30T02:12:24.607 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.605+0000 7f2bb225c700 1 --2- >> v2:172.21.15.59:3300/0 conn(0x7f2bac142290 0x7f2bac13af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:12:24.607 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.605+0000 7f2bb225c700 10 monclient(hunting): picked mon.noname-a con 0x7f2bac142290 addr v2:172.21.15.59:3300/0 2023-09-30T02:12:24.608 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.605+0000 7f2bb225c700 1 -- --> v1:172.21.15.110:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bac0a75a0 con 0x7f2bac0a5f40 2023-09-30T02:12:24.608 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.605+0000 7f2bb225c700 10 monclient(hunting): start opening mon connection 2023-09-30T02:12:24.608 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.605+0000 7f2bb225c700 1 -- --> v2:172.21.15.59:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bac0a63e0 con 0x7f2bac142290 2023-09-30T02:12:24.608 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.605+0000 7f2bb225c700 10 monclient(hunting): start opening mon connection 2023-09-30T02:12:24.608 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.605+0000 7f2bb225c700 1 -- --> v2:172.21.15.110:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bac1426b0 con 0x7f2bac0a5b70 2023-09-30T02:12:24.608 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.605+0000 7f2bb225c700 10 monclient(hunting): _renew_subs 2023-09-30T02:12:24.608 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.605+0000 7f2bb225c700 10 monclient(hunting): authenticate will time out at 2189.908355s 2023-09-30T02:12:24.608 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.605+0000 7f2bab7fe700 1 --2- >> v2:172.21.15.110:3300/0 conn(0x7f2bac0a5b70 0x7f2bac0b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:12:24.608 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.606+0000 7f2baaffd700 1 --1- >> v1:172.21.15.110:6789/0 conn(0x7f2bac0a5f40 0x7f2bac141b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.110:6789/0 says I am v1:172.21.15.59:58628/0 (socket says 172.21.15.59:58628) 2023-09-30T02:12:24.608 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.606+0000 7f2baaffd700 1 -- 172.21.15.59:0/1954022137 learned_addr learned my addr 172.21.15.59:0/1954022137 (peer_addr_for_me v1:172.21.15.59:0/0) 2023-09-30T02:12:24.608 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.606+0000 7f2bab7fe700 10 monclient(hunting): get_auth_request con 0x7f2bac0a5b70 auth_method 0 2023-09-30T02:12:24.608 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.606+0000 7f2bab7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:12:24.609 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.606+0000 7f2bab7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:12:24.609 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.606+0000 7f2bab7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:12:24.609 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.607+0000 7f2bab7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:12:24.609 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.607+0000 7f2bab7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:12:24.609 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.607+0000 7f2bab7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:12:24.609 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.609+0000 7f2bab7fe700 10 monclient(hunting): handle_auth_done global_id 4987 payload 274 2023-09-30T02:12:24.610 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.609+0000 7f2bab7fe700 1 -- 172.21.15.59:0/1954022137 >> v2:172.21.15.59:3300/0 conn(0x7f2bac142290 msgr2=0x7f2bac13af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T02:12:24.610 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.609+0000 7f2bab7fe700 1 --2- 172.21.15.59:0/1954022137 >> v2:172.21.15.59:3300/0 conn(0x7f2bac142290 0x7f2bac13af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:12:24.610 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.609+0000 7f2bab7fe700 1 -- 172.21.15.59:0/1954022137 >> v1:172.21.15.110:6789/0 conn(0x7f2bac0a5f40 legacy=0x7f2bac141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:12:24.610 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.609+0000 7f2bab7fe700 10 monclient: _finish_hunting 0 2023-09-30T02:12:24.610 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.609+0000 7f2bab7fe700 1 monclient: found mon.noname-c 2023-09-30T02:12:24.610 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.609+0000 7f2bab7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:12:24.610 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.609+0000 7f2bab7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.110:3300/0 2023-09-30T02:12:24.610 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.609+0000 7f2bab7fe700 1 -- 172.21.15.59:0/1954022137 --> v2:172.21.15.110:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bac0af3f0 con 0x7f2bac0a5b70 2023-09-30T02:12:24.612 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.612+0000 7f2bab7fe700 1 --2- 172.21.15.59:0/1954022137 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2bac0a5b70 0x7f2bac0b1500 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f2bac0b49e0 tx=0x7f2b9c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1947319dbcf88d41 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:12:24.617 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.617+0000 7f2baa7fc700 1 -- 172.21.15.59:0/1954022137 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b9c00b6b0 con 0x7f2bac0a5b70 2023-09-30T02:12:24.617 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.617+0000 7f2baa7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:12:24.618 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.617+0000 7f2baa7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T02:12:24.618 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.617+0000 7f2baa7fc700 10 monclient: dump: 2023-09-30T02:12:24.618 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:12:24.618 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:12:24.618 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:12:24.618 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:12:24.618 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:12:24.618 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:12:24.618 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:12:24.618 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:12:24.618 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:12:24.619 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:12:24.619 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.617+0000 7f2baa7fc700 10 monclient: _finish_auth 0 2023-09-30T02:12:24.619 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.617+0000 7f2baa7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:12:24.619 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.617+0000 7f2baa7fc700 1 -- 172.21.15.59:0/1954022137 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b9c00b810 con 0x7f2bac0a5b70 2023-09-30T02:12:24.619 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.617+0000 7f2bb225c700 5 monclient: authenticate success, global_id 4987 2023-09-30T02:12:24.619 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.617+0000 7f2bb225c700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T02:12:24.619 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.617+0000 7f2baa7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:12:24.619 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.617+0000 7f2baa7fc700 1 -- 172.21.15.59:0/1954022137 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b9c00bbb0 con 0x7f2bac0a5b70 2023-09-30T02:12:24.619 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.617+0000 7f2baa7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:12:24.619 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.617+0000 7f2baa7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:12:24.619 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.617+0000 7f2baa7fc700 10 monclient: dump: 2023-09-30T02:12:24.619 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:12:24.620 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:12:24.620 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:12:24.620 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:12:24.620 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:12:24.620 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:12:24.620 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:12:24.620 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:12:24.620 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:12:24.620 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:12:24.620 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.617+0000 7f2bb225c700 10 monclient: get_monmap_and_config success 2023-09-30T02:12:24.620 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.617+0000 7f2bb225c700 4 set_mon_vals no callback set 2023-09-30T02:12:24.623 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.618+0000 7f2bb225c700 10 monclient: shutdown 2023-09-30T02:12:24.623 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.618+0000 7f2bb225c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:12:24.623 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.618+0000 7f2bb225c700 1 -- 172.21.15.59:0/1954022137 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2bac0a5b70 msgr2=0x7f2bac0b1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:12:24.623 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.618+0000 7f2bb225c700 1 --2- 172.21.15.59:0/1954022137 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2bac0a5b70 0x7f2bac0b1500 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f2bac0b49e0 tx=0x7f2b9c0091a0 comp rx=0 tx=0).stop 2023-09-30T02:12:24.623 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.618+0000 7f2bb225c700 1 -- 172.21.15.59:0/1954022137 shutdown_connections 2023-09-30T02:12:24.623 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.618+0000 7f2bb225c700 1 --2- 172.21.15.59:0/1954022137 >> v2:172.21.15.59:3300/0 conn(0x7f2bac142290 0x7f2bac13af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:12:24.623 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.618+0000 7f2bb225c700 1 --2- 172.21.15.59:0/1954022137 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2bac0a5b70 0x7f2bac0b1500 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:12:24.623 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.618+0000 7f2bb225c700 1 -- 172.21.15.59:0/1954022137 >> 172.21.15.59:0/1954022137 conn(0x7f2bac0631f0 msgr2=0x7f2bac0635f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:12:24.623 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.618+0000 7f2bb225c700 1 -- 172.21.15.59:0/1954022137 shutdown_connections 2023-09-30T02:12:24.623 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.618+0000 7f2bb225c700 1 -- 172.21.15.59:0/1954022137 wait complete. 2023-09-30T02:12:24.624 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.619+0000 7f2bb225c700 10 monclient: build_initial_monmap 2023-09-30T02:12:24.624 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.619+0000 7f2bb225c700 10 monclient: monmap: 2023-09-30T02:12:24.624 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 0 2023-09-30T02:12:24.624 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T02:12:24.624 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 0.000000 2023-09-30T02:12:24.624 INFO:teuthology.orchestra.run.smithi059.stderr:created 0.000000 2023-09-30T02:12:24.624 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 0 (unknown) 2023-09-30T02:12:24.624 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 1 2023-09-30T02:12:24.624 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.noname-a 2023-09-30T02:12:24.624 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.noname-b 2023-09-30T02:12:24.624 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.noname-c 2023-09-30T02:12:24.624 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:12:24.625 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.619+0000 7f2bb225c700 1 librados: starting msgr at 2023-09-30T02:12:24.625 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.619+0000 7f2bb225c700 1 librados: starting objecter 2023-09-30T02:12:24.625 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.620+0000 7f2bb225c700 1 Processor -- start 2023-09-30T02:12:24.625 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.620+0000 7f2bb225c700 1 -- start start 2023-09-30T02:12:24.625 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.620+0000 7f2bb225c700 1 librados: setting wanted keys 2023-09-30T02:12:24.625 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.620+0000 7f2bb225c700 1 librados: calling monclient init 2023-09-30T02:12:24.625 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.620+0000 7f2bb225c700 10 monclient: init 2023-09-30T02:12:24.625 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.620+0000 7f2bb225c700 10 monclient: _reopen_session rank -1 2023-09-30T02:12:24.625 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.620+0000 7f2bb225c700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T02:12:24.625 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.620+0000 7f2bb225c700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2bac0a5b70 0x7f2bac134f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:12:24.625 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.620+0000 7f2bb225c700 10 monclient(hunting): picked mon.noname-c con 0x7f2bac0a5b70 addr [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] 2023-09-30T02:12:24.625 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.620+0000 7f2bb225c700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2bac0a5f40 0x7f2bac133c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:12:24.626 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.620+0000 7f2bb225c700 10 monclient(hunting): picked mon.noname-b con 0x7f2bac0a5f40 addr [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] 2023-09-30T02:12:24.626 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.620+0000 7f2bb225c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2bac142290 0x7f2bac134150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:12:24.626 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.620+0000 7f2bb225c700 10 monclient(hunting): picked mon.noname-a con 0x7f2bac142290 addr [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] 2023-09-30T02:12:24.626 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.620+0000 7f2bb225c700 10 monclient(hunting): start opening mon connection 2023-09-30T02:12:24.626 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.620+0000 7f2bb225c700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bac13ed20 con 0x7f2bac142290 2023-09-30T02:12:24.626 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.620+0000 7f2bb225c700 10 monclient(hunting): start opening mon connection 2023-09-30T02:12:24.626 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.620+0000 7f2bb225c700 1 -- --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bac13f000 con 0x7f2bac0a5f40 2023-09-30T02:12:24.626 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.620+0000 7f2bb225c700 10 monclient(hunting): start opening mon connection 2023-09-30T02:12:24.626 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.620+0000 7f2bb225c700 1 -- --> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bac134690 con 0x7f2bac0a5b70 2023-09-30T02:12:24.626 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.620+0000 7f2bb225c700 10 monclient(hunting): _renew_subs 2023-09-30T02:12:24.626 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.620+0000 7f2bb225c700 10 monclient(hunting): authenticate will time out at 2189.923722s 2023-09-30T02:12:24.626 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.621+0000 7f2baaffd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2bac0a5f40 0x7f2bac133c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:12:24.627 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.621+0000 7f2bab7fe700 1 --2- >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2bac0a5b70 0x7f2bac134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:12:24.627 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.621+0000 7f2baaffd700 1 --2- >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2bac0a5f40 0x7f2bac133c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.110:3300/0 says I am v2:172.21.15.59:51180/0 (socket says 172.21.15.59:51180) 2023-09-30T02:12:24.627 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.621+0000 7f2baaffd700 1 -- 172.21.15.59:0/3968676778 learned_addr learned my addr 172.21.15.59:0/3968676778 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-09-30T02:12:24.627 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.621+0000 7f2baaffd700 10 monclient(hunting): get_auth_request con 0x7f2bac0a5f40 auth_method 0 2023-09-30T02:12:24.627 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.621+0000 7f2baaffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:12:24.627 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.621+0000 7f2baaffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:12:24.627 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.621+0000 7f2baaffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:12:24.627 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.622+0000 7f2bab7fe700 10 monclient(hunting): get_auth_request con 0x7f2bac0a5b70 auth_method 0 2023-09-30T02:12:24.627 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.622+0000 7f2bab7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:12:24.627 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.622+0000 7f2bab7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:12:24.627 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.622+0000 7f2bab7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:12:24.627 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.622+0000 7f2baaffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:12:24.628 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.622+0000 7f2baaffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:12:24.628 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.622+0000 7f2baaffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:12:24.628 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.622+0000 7f2bab7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:12:24.628 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.622+0000 7f2bab7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:12:24.628 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.622+0000 7f2bab7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:12:24.629 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.624+0000 7f2babfff700 1 --2- 172.21.15.59:0/3968676778 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2bac142290 0x7f2bac134150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:12:24.629 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.625+0000 7f2babfff700 10 monclient(hunting): get_auth_request con 0x7f2bac142290 auth_method 0 2023-09-30T02:12:24.629 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.625+0000 7f2babfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T02:12:24.629 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.625+0000 7f2babfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T02:12:24.630 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.625+0000 7f2babfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T02:12:24.630 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.625+0000 7f2babfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T02:12:24.630 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.625+0000 7f2babfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T02:12:24.630 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.625+0000 7f2babfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T02:12:24.630 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.626+0000 7f2baaffd700 10 monclient(hunting): handle_auth_done global_id 4990 payload 824 2023-09-30T02:12:24.630 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.626+0000 7f2baaffd700 1 -- 172.21.15.59:0/3968676778 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2bac0a5b70 msgr2=0x7f2bac134f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:12:24.630 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.626+0000 7f2baaffd700 1 --2- 172.21.15.59:0/3968676778 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2bac0a5b70 0x7f2bac134f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:12:24.630 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.626+0000 7f2baaffd700 1 -- 172.21.15.59:0/3968676778 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2bac142290 msgr2=0x7f2bac134150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:12:24.630 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.626+0000 7f2baaffd700 1 --2- 172.21.15.59:0/3968676778 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2bac142290 0x7f2bac134150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:12:24.630 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.626+0000 7f2baaffd700 10 monclient: _finish_hunting 0 2023-09-30T02:12:24.630 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.626+0000 7f2baaffd700 1 monclient: found mon.noname-b 2023-09-30T02:12:24.630 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.626+0000 7f2baaffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T02:12:24.631 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.626+0000 7f2baaffd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.110:3300/0 2023-09-30T02:12:24.631 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.626+0000 7f2baaffd700 1 -- 172.21.15.59:0/3968676778 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b9c009420 con 0x7f2bac0a5f40 2023-09-30T02:12:24.631 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.629+0000 7f2baaffd700 1 --2- 172.21.15.59:0/3968676778 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2bac0a5f40 0x7f2bac133c10 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f2b9800c7c0 tx=0x7f2b98007f70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:12:24.631 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.629+0000 7f2ba37fe700 10 client.?.objecter ms_handle_connect 0x7f2bac0a5f40 2023-09-30T02:12:24.631 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.629+0000 7f2ba37fe700 10 client.?.objecter resend_mon_ops 2023-09-30T02:12:24.637 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.635+0000 7f2ba37fe700 1 -- 172.21.15.59:0/3968676778 <== mon.1 v2:172.21.15.110:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b98005b30 con 0x7f2bac0a5f40 2023-09-30T02:12:24.637 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.636+0000 7f2ba37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:12:24.637 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.636+0000 7f2ba37fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T02:12:24.637 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.636+0000 7f2ba37fe700 10 monclient: dump: 2023-09-30T02:12:24.638 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:12:24.638 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:12:24.638 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:12:24.638 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:12:24.638 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:12:24.638 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:12:24.638 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:12:24.638 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:12:24.638 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:12:24.638 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:12:24.638 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.636+0000 7f2ba37fe700 10 monclient: _finish_auth 0 2023-09-30T02:12:24.638 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.636+0000 7f2ba37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T02:12:24.639 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.636+0000 7f2ba37fe700 1 -- 172.21.15.59:0/3968676778 <== mon.1 v2:172.21.15.110:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b98005c90 con 0x7f2bac0a5f40 2023-09-30T02:12:24.639 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.636+0000 7f2bb225c700 5 monclient: authenticate success, global_id 4990 2023-09-30T02:12:24.639 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.636+0000 7f2bb225c700 10 monclient: _renew_subs 2023-09-30T02:12:24.639 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.636+0000 7f2bb225c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:12:24.639 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.636+0000 7f2bb225c700 1 -- 172.21.15.59:0/3968676778 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bac1407b0 con 0x7f2bac0a5f40 2023-09-30T02:12:24.639 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.636+0000 7f2ba37fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T02:12:24.639 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.636+0000 7f2ba37fe700 1 -- 172.21.15.59:0/3968676778 <== mon.1 v2:172.21.15.110:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b98004a80 con 0x7f2bac0a5f40 2023-09-30T02:12:24.639 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.636+0000 7f2ba37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T02:12:24.639 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.636+0000 7f2ba37fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T02:12:24.639 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.636+0000 7f2ba37fe700 10 monclient: dump: 2023-09-30T02:12:24.639 INFO:teuthology.orchestra.run.smithi059.stderr:epoch 1 2023-09-30T02:12:24.639 INFO:teuthology.orchestra.run.smithi059.stderr:fsid 623052cb-4f82-4b2b-b72b-3ed5be614613 2023-09-30T02:12:24.640 INFO:teuthology.orchestra.run.smithi059.stderr:last_changed 2023-09-30T02:04:19.017744+0000 2023-09-30T02:12:24.640 INFO:teuthology.orchestra.run.smithi059.stderr:created 2023-09-30T02:04:19.017744+0000 2023-09-30T02:12:24.640 INFO:teuthology.orchestra.run.smithi059.stderr:min_mon_release 17 (quincy) 2023-09-30T02:12:24.640 INFO:teuthology.orchestra.run.smithi059.stderr:election_strategy: 3 2023-09-30T02:12:24.640 INFO:teuthology.orchestra.run.smithi059.stderr:0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.a 2023-09-30T02:12:24.640 INFO:teuthology.orchestra.run.smithi059.stderr:1: [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] mon.b 2023-09-30T02:12:24.640 INFO:teuthology.orchestra.run.smithi059.stderr:2: [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] mon.c 2023-09-30T02:12:24.640 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-09-30T02:12:24.640 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.636+0000 7f2ba9ffb700 4 set_mon_vals no callback set 2023-09-30T02:12:24.640 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.636+0000 7f2bb225c700 10 client.4990.objecter _maybe_request_map subscribing (onetime) to next osd map 2023-09-30T02:12:24.640 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.636+0000 7f2bb225c700 10 monclient: _renew_subs 2023-09-30T02:12:24.640 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.636+0000 7f2bb225c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:12:24.641 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.636+0000 7f2bb225c700 1 -- 172.21.15.59:0/3968676778 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bac1413d0 con 0x7f2bac0a5f40 2023-09-30T02:12:24.641 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.636+0000 7f2bb225c700 1 librados: init done 2023-09-30T02:12:24.641 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.637+0000 7f2bb225c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T02:12:24.641 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.637+0000 7f2bb225c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:12:24.641 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.637+0000 7f2bb225c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:12:24.641 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.637+0000 7f2bb225c700 1 -- 172.21.15.59:0/3968676778 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bac141650 con 0x7f2bac0a5f40 2023-09-30T02:12:24.655 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.655+0000 7f2ba37fe700 1 -- 172.21.15.59:0/3968676778 <== mon.1 v2:172.21.15.110:3300/0 4 ==== mgrmap(e 5) v1 ==== 84705+0+0 (secure 0 0 0) 0x7f2b980070d0 con 0x7f2bac0a5f40 2023-09-30T02:12:24.656 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.656+0000 7f2ba37fe700 1 --2- 172.21.15.59:0/3968676778 >> v2:172.21.15.110:6800/115870 conn(0x7f2b94068890 0x7f2b9406ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T02:12:24.656 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.656+0000 7f2bab7fe700 1 --2- 172.21.15.59:0/3968676778 >> v2:172.21.15.110:6800/115870 conn(0x7f2b94068890 0x7f2b9406ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T02:12:24.656 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.656+0000 7f2bab7fe700 10 monclient: get_auth_request con 0x7f2b94068890 auth_method 0 2023-09-30T02:12:24.656 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.656+0000 7f2bab7fe700 1 --2- 172.21.15.59:0/3968676778 >> v2:172.21.15.110:6800/115870 conn(0x7f2b94068890 0x7f2b9406ad40 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2b9c0091e0 tx=0x7f2b9c017000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T02:12:24.657 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.656+0000 7f2ba37fe700 10 client.4990.objecter ms_handle_connect 0x7f2b94068890 2023-09-30T02:12:24.663 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.663+0000 7f2ba37fe700 1 -- 172.21.15.59:0/3968676778 <== mon.1 v2:172.21.15.110:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4086+0+0 (secure 0 0 0) 0x7f2b98055ac0 con 0x7f2bac0a5f40 2023-09-30T02:12:24.663 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.663+0000 7f2ba37fe700 10 client.4990.objecter ms_dispatch 0x7f2bac001550 osd_map(25..25 src has 1..25) v4 2023-09-30T02:12:24.663 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.663+0000 7f2ba37fe700 3 client.4990.objecter handle_osd_map got epochs [25,25] > 0 2023-09-30T02:12:24.663 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.663+0000 7f2ba37fe700 3 client.4990.objecter handle_osd_map decoding full epoch 25 2023-09-30T02:12:24.663 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.663+0000 7f2ba37fe700 20 client.4990.objecter dump_active .. 0 homeless 2023-09-30T02:12:24.942 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.942+0000 7f2ba37fe700 1 -- 172.21.15.59:0/3968676778 <== mon.1 v2:172.21.15.110:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2b98051700 con 0x7f2bac0a5f40 2023-09-30T02:12:24.942 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.942+0000 7f2ba37fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T02:12:24.942 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:24.942+0000 7f2ba37fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T02:12:25.164 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:25.163+0000 7f2bb225c700 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2023-09-30T02:12:25.164 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:25.163+0000 7f2bb225c700 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2023-09-30T02:12:25.164 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:25.163+0000 7f2bb225c700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.110:3300/0 2023-09-30T02:12:25.164 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:25.163+0000 7f2bb225c700 1 -- 172.21.15.59:0/3968676778 --> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7f2bac064a00 con 0x7f2bac0a5f40 2023-09-30T02:12:25.475 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:25.475+0000 7f2ba37fe700 1 -- 172.21.15.59:0/3968676778 <== mon.1 v2:172.21.15.110:3300/0 7 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f2b98051700 con 0x7f2bac0a5f40 2023-09-30T02:12:25.476 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:25.475+0000 7f2ba37fe700 10 monclient: handle_config config(1 keys) v1 2023-09-30T02:12:25.476 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:25.475+0000 7f2ba9ffb700 4 set_mon_vals no callback set 2023-09-30T02:12:25.476 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:25.475+0000 7f2ba37fe700 1 -- 172.21.15.59:0/3968676778 <== mon.1 v2:172.21.15.110:3300/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v2)=0 v2) v1 ==== 123+0+0 (secure 0 0 0) 0x7f2b98021020 con 0x7f2bac0a5f40 2023-09-30T02:12:25.476 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:25.475+0000 7f2ba37fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2023-09-30T02:12:25.476 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:25.475+0000 7f2ba37fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T02:12:25.477 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:25.475+0000 7f2ba9ffb700 10 set_mon_vals mon_health_to_clog = false 2023-09-30T02:12:25.478 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:25.477+0000 7f2bb225c700 10 librados: watch_flush enter 2023-09-30T02:12:25.478 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:25.477+0000 7f2bb225c700 10 librados: watch_flush exit 2023-09-30T02:12:25.478 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:25.477+0000 7f2bb225c700 20 client.4990.objecter shutdown clearing up homeless session... 2023-09-30T02:12:25.479 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:25.477+0000 7f2bb225c700 10 client.4990.objecter successfully canceled tick 2023-09-30T02:12:25.479 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:25.477+0000 7f2bb225c700 1 -- 172.21.15.59:0/3968676778 >> v2:172.21.15.110:6800/115870 conn(0x7f2b94068890 msgr2=0x7f2b9406ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:12:25.479 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:25.477+0000 7f2bb225c700 1 --2- 172.21.15.59:0/3968676778 >> v2:172.21.15.110:6800/115870 conn(0x7f2b94068890 0x7f2b9406ad40 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2b9c0091e0 tx=0x7f2b9c017000 comp rx=0 tx=0).stop 2023-09-30T02:12:25.479 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:25.477+0000 7f2bb225c700 10 monclient: shutdown 2023-09-30T02:12:25.479 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:25.477+0000 7f2bb225c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T02:12:25.479 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:25.477+0000 7f2bb225c700 1 -- 172.21.15.59:0/3968676778 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2bac0a5f40 msgr2=0x7f2bac133c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T02:12:25.479 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:25.477+0000 7f2bb225c700 1 --2- 172.21.15.59:0/3968676778 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2bac0a5f40 0x7f2bac133c10 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f2b9800c7c0 tx=0x7f2b98007f70 comp rx=0 tx=0).stop 2023-09-30T02:12:25.479 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:25.477+0000 7f2bb225c700 1 -- 172.21.15.59:0/3968676778 shutdown_connections 2023-09-30T02:12:25.479 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:25.477+0000 7f2bb225c700 1 --2- 172.21.15.59:0/3968676778 >> v2:172.21.15.110:6800/115870 conn(0x7f2b94068890 0x7f2b9406ad40 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:12:25.480 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:25.477+0000 7f2bb225c700 1 --2- 172.21.15.59:0/3968676778 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2bac142290 0x7f2bac134150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:12:25.480 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:25.477+0000 7f2bb225c700 1 --2- 172.21.15.59:0/3968676778 >> [v2:172.21.15.59:3301/0,v1:172.21.15.59:6790/0] conn(0x7f2bac0a5b70 0x7f2bac134f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:12:25.480 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:25.477+0000 7f2bb225c700 1 --2- 172.21.15.59:0/3968676778 >> [v2:172.21.15.110:3300/0,v1:172.21.15.110:6789/0] conn(0x7f2bac0a5f40 0x7f2bac133c10 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T02:12:25.480 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:25.477+0000 7f2bb225c700 1 -- 172.21.15.59:0/3968676778 >> 172.21.15.59:0/3968676778 conn(0x7f2bac0631f0 msgr2=0x7f2bac0635c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T02:12:25.480 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:25.477+0000 7f2bb225c700 1 -- 172.21.15.59:0/3968676778 shutdown_connections 2023-09-30T02:12:25.480 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:25.477+0000 7f2bb225c700 1 -- 172.21.15.59:0/3968676778 wait complete. 2023-09-30T02:12:25.480 INFO:teuthology.orchestra.run.smithi059.stderr:2023-09-30T02:12:25.478+0000 7f2bb225c700 1 librados: shutdown 2023-09-30T02:12:25.492 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1895, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1472, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph_manager.py", line 3189, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2023-09-30T02:12:25.494 INFO:teuthology.misc:Shutting down mds daemons... 2023-09-30T02:12:25.494 INFO:teuthology.misc:Shutting down osd daemons... 2023-09-30T02:12:25.494 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-09-30T02:12:25.494 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T02:12:25.508 INFO:tasks.ceph.osd.0.smithi059.stderr:2023-09-30T02:12:25.499+0000 ee8a700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 116643) UID: 0 2023-09-30T02:12:25.511 INFO:tasks.ceph.osd.0.smithi059.stderr:2023-09-30T02:12:25.501+0000 ee8a700 -1 osd.0 25 *** Got signal Terminated *** 2023-09-30T02:12:31.596 INFO:tasks.ceph.osd.0:Stopped 2023-09-30T02:12:31.597 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-09-30T02:12:31.597 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T02:12:31.614 INFO:tasks.ceph.osd.1.smithi059.stderr:2023-09-30T02:12:31.603+0000 ee8a700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 116644) UID: 0 2023-09-30T02:12:31.614 INFO:tasks.ceph.osd.1.smithi059.stderr:2023-09-30T02:12:31.605+0000 ee8a700 -1 osd.1 27 *** Got signal Terminated *** 2023-09-30T02:12:31.837 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2023-09-30T02:12:34.680 INFO:tasks.ceph.mon.a.smithi059.stderr:==00:00:07:44.545 116256== Warning: unimplemented fcntl command: 1036 2023-09-30T02:12:34.745 INFO:tasks.ceph.mon.a.smithi059.stderr:==00:00:07:44.610 116256== Warning: unimplemented fcntl command: 1036 2023-09-30T02:12:34.774 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:07:44.639 116257== Warning: unimplemented fcntl command: 1036 2023-09-30T02:12:34.780 INFO:tasks.ceph.mon.a.smithi059.stderr:==00:00:07:44.645 116256== Warning: unimplemented fcntl command: 1036 2023-09-30T02:12:34.790 INFO:tasks.ceph.mon.b.smithi110.stderr:==00:00:07:44.637 115869== Warning: unimplemented fcntl command: 1036 2023-09-30T02:12:34.795 INFO:tasks.ceph.mon.b.smithi110.stderr:==00:00:07:44.642 115869== Warning: unimplemented fcntl command: 1036 2023-09-30T02:12:34.820 INFO:tasks.ceph.mon.b.smithi110.stderr:==00:00:07:44.667 115869== Warning: unimplemented fcntl command: 1036 2023-09-30T02:12:34.830 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:07:44.695 116257== Warning: unimplemented fcntl command: 1036 2023-09-30T02:12:34.862 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:07:44.726 116257== Warning: unimplemented fcntl command: 1036 2023-09-30T02:12:37.699 INFO:tasks.ceph.osd.1:Stopped 2023-09-30T02:12:37.700 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-09-30T02:12:37.700 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T02:12:37.715 INFO:tasks.ceph.osd.2.smithi059.stderr:2023-09-30T02:12:37.705+0000 ee8a700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 116649) UID: 0 2023-09-30T02:12:37.718 INFO:tasks.ceph.osd.2.smithi059.stderr:2023-09-30T02:12:37.708+0000 ee8a700 -1 osd.2 29 *** Got signal Terminated *** 2023-09-30T02:12:37.946 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2023-09-30T02:12:37.947 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2023-09-30T02:12:43.801 INFO:tasks.ceph.osd.2:Stopped 2023-09-30T02:12:43.801 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-09-30T02:12:43.801 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T02:12:43.819 INFO:tasks.ceph.osd.3.smithi059.stderr:2023-09-30T02:12:43.807+0000 ee8a700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 116656) UID: 0 2023-09-30T02:12:43.820 INFO:tasks.ceph.osd.3.smithi059.stderr:2023-09-30T02:12:43.809+0000 ee8a700 -1 osd.3 31 *** Got signal Terminated *** 2023-09-30T02:12:43.955 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2023-09-30T02:12:43.955 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2023-09-30T02:12:49.862 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2023-09-30T02:12:49.863 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2023-09-30T02:12:49.903 INFO:tasks.ceph.osd.3:Stopped 2023-09-30T02:12:49.904 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-09-30T02:12:49.904 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T02:12:49.917 INFO:tasks.ceph.osd.4.smithi110.stderr:2023-09-30T02:12:49.909+0000 ee8a700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 116550) UID: 0 2023-09-30T02:12:49.919 INFO:tasks.ceph.osd.4.smithi110.stderr:2023-09-30T02:12:49.911+0000 ee8a700 -1 osd.4 34 *** Got signal Terminated *** 2023-09-30T02:12:54.834 INFO:tasks.ceph.mon.a.smithi059.stderr:==00:00:08:04.699 116256== Warning: unimplemented fcntl command: 1036 2023-09-30T02:12:54.839 INFO:tasks.ceph.mon.a.smithi059.stderr:==00:00:08:04.704 116256== Warning: unimplemented fcntl command: 1036 2023-09-30T02:12:54.846 INFO:tasks.ceph.mon.b.smithi110.stderr:==00:00:08:04.694 115869== Warning: unimplemented fcntl command: 1036 2023-09-30T02:12:54.852 INFO:tasks.ceph.mon.b.smithi110.stderr:==00:00:08:04.699 115869== Warning: unimplemented fcntl command: 1036 2023-09-30T02:12:54.865 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:08:04.731 116257== Warning: unimplemented fcntl command: 1036 2023-09-30T02:12:54.869 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:08:04.735 116257== Warning: unimplemented fcntl command: 1036 2023-09-30T02:12:54.881 INFO:tasks.ceph.mon.a.smithi059.stderr:==00:00:08:04.746 116256== Warning: unimplemented fcntl command: 1036 2023-09-30T02:12:54.884 INFO:tasks.ceph.mon.b.smithi110.stderr:==00:00:08:04.731 115869== Warning: unimplemented fcntl command: 1036 2023-09-30T02:12:54.903 INFO:tasks.ceph.mon.c.smithi059.stderr:==00:00:08:04.769 116257== Warning: unimplemented fcntl command: 1036 2023-09-30T02:12:55.669 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~0s 2023-09-30T02:12:55.669 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2023-09-30T02:12:56.006 INFO:tasks.ceph.osd.4:Stopped 2023-09-30T02:12:56.006 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-09-30T02:12:56.006 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T02:12:56.019 INFO:tasks.ceph.osd.5.smithi110.stderr:2023-09-30T02:12:56.011+0000 ee8a700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 116553) UID: 0 2023-09-30T02:12:56.021 INFO:tasks.ceph.osd.5.smithi110.stderr:2023-09-30T02:12:56.013+0000 ee8a700 -1 osd.5 35 *** Got signal Terminated *** 2023-09-30T02:13:01.376 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2023-09-30T02:13:01.376 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 has been restored 2023-09-30T02:13:02.107 INFO:tasks.ceph.osd.5:Stopped 2023-09-30T02:13:02.108 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2023-09-30T02:13:02.108 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T02:13:02.123 INFO:tasks.ceph.osd.6.smithi110.stderr:2023-09-30T02:13:02.114+0000 ee8a700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 116555) UID: 0 2023-09-30T02:13:02.125 INFO:tasks.ceph.osd.6.smithi110.stderr:2023-09-30T02:13:02.116+0000 ee8a700 -1 osd.6 37 *** Got signal Terminated *** 2023-09-30T02:13:06.981 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~0s 2023-09-30T02:13:06.982 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 has been restored 2023-09-30T02:13:08.209 INFO:tasks.ceph.osd.6:Stopped 2023-09-30T02:13:08.209 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2023-09-30T02:13:08.210 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T02:13:08.224 INFO:tasks.ceph.osd.7.smithi110.stderr:2023-09-30T02:13:08.215+0000 ee8a700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 116558) UID: 0 2023-09-30T02:13:08.226 INFO:tasks.ceph.osd.7.smithi110.stderr:2023-09-30T02:13:08.217+0000 ee8a700 -1 osd.7 37 *** Got signal Terminated *** 2023-09-30T02:13:12.486 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2023-09-30T02:13:12.486 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 has been restored 2023-09-30T02:13:14.311 INFO:tasks.ceph.osd.7:Stopped 2023-09-30T02:13:14.312 INFO:teuthology.misc:Shutting down mgr daemons... 2023-09-30T02:13:14.312 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-09-30T02:13:14.312 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T02:13:14.335 INFO:tasks.ceph.mgr.y:Stopped 2023-09-30T02:13:14.336 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-09-30T02:13:14.336 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T02:13:14.336 INFO:tasks.ceph.mgr.x.smithi110.stderr:2023-09-30T02:13:14.337+0000 7fae0a6eb700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-mgr -f --cluster ceph -i x (PID: 115867) UID: 0 2023-09-30T02:13:14.337 INFO:tasks.ceph.mgr.x.smithi110.stderr:2023-09-30T02:13:14.337+0000 7fae0a6eb700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2023-09-30T02:13:14.362 INFO:tasks.ceph.mgr.x:Stopped 2023-09-30T02:13:14.363 INFO:teuthology.misc:Shutting down mon daemons... 2023-09-30T02:13:14.363 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-09-30T02:13:14.363 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T02:13:14.403 INFO:tasks.ceph.mon.a.smithi059.stderr:2023-09-30T02:13:14.386+0000 dc10700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 116203) UID: 0 2023-09-30T02:13:14.404 INFO:tasks.ceph.mon.a.smithi059.stderr:2023-09-30T02:13:14.388+0000 dc10700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-09-30T02:13:17.689 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2023-09-30T02:13:17.689 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 has been restored 2023-09-30T02:13:20.464 INFO:tasks.ceph.mon.a:Stopped 2023-09-30T02:13:20.465 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-09-30T02:13:20.465 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T02:13:20.490 INFO:tasks.ceph.mon.c.smithi059.stderr:2023-09-30T02:13:20.471+0000 dc10700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 116205) UID: 0 2023-09-30T02:13:20.490 INFO:tasks.ceph.mon.c.smithi059.stderr:2023-09-30T02:13:20.473+0000 dc10700 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2023-09-30T02:13:22.791 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2023-09-30T02:13:22.791 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a has been restored 2023-09-30T02:13:26.566 INFO:tasks.ceph.mon.c:Stopped 2023-09-30T02:13:26.567 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-09-30T02:13:26.567 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T02:13:26.591 INFO:tasks.ceph.mon.b.smithi110.stderr:2023-09-30T02:13:26.572+0000 dc10700 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 115864) UID: 0 2023-09-30T02:13:26.592 INFO:tasks.ceph.mon.b.smithi110.stderr:2023-09-30T02:13:26.574+0000 dc10700 -1 mon.b@1(probing) e1 *** Got Signal Terminated *** 2023-09-30T02:13:27.792 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2023-09-30T02:13:27.792 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2023-09-30T02:13:32.669 INFO:tasks.ceph.mon.b:Stopped 2023-09-30T02:13:32.669 INFO:tasks.ceph:Checking cluster log for badness... 2023-09-30T02:13:32.669 DEBUG:teuthology.orchestra.run.smithi059:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2023-09-30T02:13:32.716 INFO:teuthology.orchestra.run.smithi059.stdout:1696039520.0927126 mgr.x (mgr.4108) 1 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: 'PgAutoscaler' object has no attribute 'noautoscale' 2023-09-30T02:13:32.716 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2023-09-30T02:13:32.716 DEBUG:teuthology.orchestra.run.smithi059:> sudo egrep '\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2023-09-30T02:13:32.760 DEBUG:teuthology.orchestra.run.smithi059:> sudo egrep '\[ERR\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2023-09-30T02:13:32.793 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b has been restored 2023-09-30T02:13:32.847 INFO:teuthology.orchestra.run.smithi059.stdout:1696039520.0927126 mgr.x (mgr.4108) 1 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: 'PgAutoscaler' object has no attribute 'noautoscale' 2023-09-30T02:13:32.847 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi059.front.sepia.ceph.com 2023-09-30T02:13:32.847 DEBUG:teuthology.orchestra.run.smithi059:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-09-30T02:13:32.997 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi059.front.sepia.ceph.com 2023-09-30T02:13:32.998 DEBUG:teuthology.orchestra.run.smithi059:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-09-30T02:13:33.093 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi059.front.sepia.ceph.com 2023-09-30T02:13:33.093 DEBUG:teuthology.orchestra.run.smithi059:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-09-30T02:13:33.189 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi059.front.sepia.ceph.com 2023-09-30T02:13:33.190 DEBUG:teuthology.orchestra.run.smithi059:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-09-30T02:13:33.274 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi110.front.sepia.ceph.com 2023-09-30T02:13:33.275 DEBUG:teuthology.orchestra.run.smithi110:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-09-30T02:13:33.446 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi110.front.sepia.ceph.com 2023-09-30T02:13:33.446 DEBUG:teuthology.orchestra.run.smithi110:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-09-30T02:13:33.525 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi110.front.sepia.ceph.com 2023-09-30T02:13:33.525 DEBUG:teuthology.orchestra.run.smithi110:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2023-09-30T02:13:33.598 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi110.front.sepia.ceph.com 2023-09-30T02:13:33.598 DEBUG:teuthology.orchestra.run.smithi110:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2023-09-30T02:13:33.656 INFO:tasks.ceph:Archiving mon data... 2023-09-30T02:13:33.657 DEBUG:teuthology.misc:Transferring archived files from smithi059:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7407087/data/mon.a.tgz 2023-09-30T02:13:33.657 DEBUG:teuthology.orchestra.run.smithi059:> mktemp 2023-09-30T02:13:33.680 INFO:teuthology.orchestra.run.smithi059.stdout:/tmp/tmp.unNdJYId3s 2023-09-30T02:13:33.680 DEBUG:teuthology.orchestra.run.smithi059:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.unNdJYId3s 2023-09-30T02:13:33.826 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 0666 /tmp/tmp.unNdJYId3s 2023-09-30T02:13:33.924 DEBUG:teuthology.orchestra.remote:smithi059:/tmp/tmp.unNdJYId3s is 290KB 2023-09-30T02:13:34.019 DEBUG:teuthology.orchestra.run.smithi059:> rm -fr /tmp/tmp.unNdJYId3s 2023-09-30T02:13:34.042 DEBUG:teuthology.misc:Transferring archived files from smithi059:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7407087/data/mon.c.tgz 2023-09-30T02:13:34.043 DEBUG:teuthology.orchestra.run.smithi059:> mktemp 2023-09-30T02:13:34.104 INFO:teuthology.orchestra.run.smithi059.stdout:/tmp/tmp.RGm9tpWYxW 2023-09-30T02:13:34.104 DEBUG:teuthology.orchestra.run.smithi059:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.RGm9tpWYxW 2023-09-30T02:13:34.232 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 0666 /tmp/tmp.RGm9tpWYxW 2023-09-30T02:13:34.328 DEBUG:teuthology.orchestra.remote:smithi059:/tmp/tmp.RGm9tpWYxW is 291KB 2023-09-30T02:13:34.401 DEBUG:teuthology.orchestra.run.smithi059:> rm -fr /tmp/tmp.RGm9tpWYxW 2023-09-30T02:13:34.424 DEBUG:teuthology.misc:Transferring archived files from smithi110:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7407087/data/mon.b.tgz 2023-09-30T02:13:34.425 DEBUG:teuthology.orchestra.run.smithi110:> mktemp 2023-09-30T02:13:34.446 INFO:teuthology.orchestra.run.smithi110.stdout:/tmp/tmp.hrA8B220M3 2023-09-30T02:13:34.447 DEBUG:teuthology.orchestra.run.smithi110:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.hrA8B220M3 2023-09-30T02:13:34.573 DEBUG:teuthology.orchestra.run.smithi110:> sudo chmod 0666 /tmp/tmp.hrA8B220M3 2023-09-30T02:13:34.663 DEBUG:teuthology.orchestra.remote:smithi110:/tmp/tmp.hrA8B220M3 is 292KB 2023-09-30T02:13:34.758 DEBUG:teuthology.orchestra.run.smithi110:> rm -fr /tmp/tmp.hrA8B220M3 2023-09-30T02:13:34.777 INFO:tasks.ceph:Cleaning ceph cluster... 2023-09-30T02:13:34.777 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-09-30T02:13:34.779 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-09-30T02:13:34.843 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-09-30T02:13:34.843 DEBUG:teuthology.orchestra.run.smithi059:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-09-30T02:13:34.854 DEBUG:teuthology.orchestra.run.smithi110:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-09-30T02:13:34.929 INFO:tasks.ceph:Archiving crash dumps... 2023-09-30T02:13:34.931 DEBUG:teuthology.misc:Transferring archived files from smithi059:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7407087/remote/smithi059/crash 2023-09-30T02:13:34.932 DEBUG:teuthology.orchestra.run.smithi059:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-09-30T02:13:34.966 DEBUG:teuthology.misc:Transferring archived files from smithi110:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7407087/remote/smithi110/crash 2023-09-30T02:13:34.967 DEBUG:teuthology.orchestra.run.smithi110:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-09-30T02:13:34.997 INFO:tasks.ceph:Compressing logs... 2023-09-30T02:13:34.997 DEBUG:teuthology.orchestra.run.smithi059:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-30T02:13:35.012 DEBUG:teuthology.orchestra.run.smithi110:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-30T02:13:35.947 INFO:tasks.ceph:Archiving logs... 2023-09-30T02:13:35.948 DEBUG:teuthology.misc:Transferring archived files from smithi059:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7407087/remote/smithi059/log 2023-09-30T02:13:35.949 DEBUG:teuthology.orchestra.run.smithi059:> sudo tar c -f - -C /var/log/ceph -- . 2023-09-30T02:13:36.227 DEBUG:teuthology.misc:Transferring archived files from smithi110:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7407087/remote/smithi110/log 2023-09-30T02:13:36.227 DEBUG:teuthology.orchestra.run.smithi110:> sudo tar c -f - -C /var/log/ceph -- . 2023-09-30T02:13:36.361 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1895, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1472, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph_manager.py", line 3189, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2023-09-30T02:13:36.428 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=4940495653ec4eaf8a4ca6ac8d69ef3a Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1895, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1472, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph_manager.py", line 3189, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2023-09-30T02:13:36.430 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-09-30T02:13:36.444 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-09-30T02:13:36.458 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-09-30T02:13:36.459 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-09-30T02:13:36.471 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-09-30T02:13:36.555 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-09-30T02:13:36.564 INFO:teuthology.task.clock:Checking final clock skew... 2023-09-30T02:13:36.565 DEBUG:teuthology.orchestra.run.smithi059:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-30T02:13:36.568 DEBUG:teuthology.orchestra.run.smithi110:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-30T02:13:36.587 INFO:teuthology.orchestra.run.smithi110.stderr:bash: ntpq: command not found 2023-09-30T02:13:36.588 INFO:teuthology.orchestra.run.smithi059.stderr:bash: ntpq: command not found 2023-09-30T02:13:36.591 INFO:teuthology.orchestra.run.smithi110.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-09-30T02:13:36.591 INFO:teuthology.orchestra.run.smithi110.stdout:=============================================================================== 2023-09-30T02:13:36.592 INFO:teuthology.orchestra.run.smithi110.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 26 -231us[ -223us] +/- 82ms 2023-09-30T02:13:36.592 INFO:teuthology.orchestra.run.smithi110.stdout:^* hv02.front.sepia.ceph.com 3 6 377 24 -198us[ -189us] +/- 51ms 2023-09-30T02:13:36.592 INFO:teuthology.orchestra.run.smithi110.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 26 +481us[ +490us] +/- 73ms 2023-09-30T02:13:36.592 INFO:teuthology.orchestra.run.smithi110.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T02:13:36.594 INFO:teuthology.orchestra.run.smithi059.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-09-30T02:13:36.594 INFO:teuthology.orchestra.run.smithi059.stdout:=============================================================================== 2023-09-30T02:13:36.594 INFO:teuthology.orchestra.run.smithi059.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 25 -242us[ -242us] +/- 82ms 2023-09-30T02:13:36.594 INFO:teuthology.orchestra.run.smithi059.stdout:^* hv02.front.sepia.ceph.com 3 7 377 91 -182us[ -184us] +/- 50ms 2023-09-30T02:13:36.594 INFO:teuthology.orchestra.run.smithi059.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 24 +454us[ +454us] +/- 73ms 2023-09-30T02:13:36.594 INFO:teuthology.orchestra.run.smithi059.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T02:13:36.595 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-09-30T02:13:36.606 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-09-30T02:13:36.606 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-09-30T02:13:36.621 DEBUG:teuthology.orchestra.run.smithi059:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-09-30T02:13:36.743 DEBUG:teuthology.orchestra.run.smithi110:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-09-30T02:13:36.837 DEBUG:teuthology.orchestra.run.smithi059:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-09-30T02:13:36.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T02:13:36.877 DEBUG:teuthology.orchestra.run.smithi110:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-09-30T02:13:36.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T02:13:36.910 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-09-30T02:13:36.922 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-09-30T02:13:36.936 INFO:teuthology.task.internal:Duration was 1937.500388 seconds 2023-09-30T02:13:36.936 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-09-30T02:13:36.951 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-09-30T02:13:36.952 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-09-30T02:13:36.954 DEBUG:teuthology.orchestra.run.smithi110:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-09-30T02:13:37.002 INFO:teuthology.orchestra.run.smithi110.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-09-30T02:13:37.011 INFO:teuthology.orchestra.run.smithi059.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-09-30T02:13:37.326 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-09-30T02:13:37.327 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi059.front.sepia.ceph.com 2023-09-30T02:13:37.327 DEBUG:teuthology.orchestra.run.smithi059:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-09-30T02:13:37.356 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi110.front.sepia.ceph.com 2023-09-30T02:13:37.356 DEBUG:teuthology.orchestra.run.smithi110:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-09-30T02:13:37.380 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-09-30T02:13:37.380 DEBUG:teuthology.orchestra.run.smithi059:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-30T02:13:37.399 DEBUG:teuthology.orchestra.run.smithi110:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-30T02:13:37.460 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-09-30T02:13:37.460 DEBUG:teuthology.orchestra.run.smithi059:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-09-30T02:13:37.463 DEBUG:teuthology.orchestra.run.smithi110:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-09-30T02:13:37.571 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-09-30T02:13:37.580 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-09-30T02:13:37.581 DEBUG:teuthology.orchestra.run.smithi059:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-09-30T02:13:37.614 DEBUG:teuthology.orchestra.run.smithi110:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-09-30T02:13:37.642 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-09-30T02:13:37.651 DEBUG:teuthology.orchestra.run.smithi059:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-09-30T02:13:37.656 DEBUG:teuthology.orchestra.run.smithi110:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-09-30T02:13:37.689 INFO:teuthology.orchestra.run.smithi059.stdout:kernel.core_pattern = core 2023-09-30T02:13:37.709 INFO:teuthology.orchestra.run.smithi110.stdout:kernel.core_pattern = core 2023-09-30T02:13:37.746 DEBUG:teuthology.orchestra.run.smithi059:> test -e /home/ubuntu/cephtest/archive/coredump 2023-09-30T02:13:37.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T02:13:37.795 DEBUG:teuthology.orchestra.run.smithi110:> test -e /home/ubuntu/cephtest/archive/coredump 2023-09-30T02:13:37.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T02:13:37.812 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-09-30T02:13:37.823 INFO:teuthology.task.internal:Transferring archived files... 2023-09-30T02:13:37.824 DEBUG:teuthology.misc:Transferring archived files from smithi059:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7407087/remote/smithi059 2023-09-30T02:13:37.824 DEBUG:teuthology.orchestra.run.smithi059:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-09-30T02:13:37.882 DEBUG:teuthology.misc:Transferring archived files from smithi110:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7407087/remote/smithi110 2023-09-30T02:13:37.882 DEBUG:teuthology.orchestra.run.smithi110:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-09-30T02:13:37.920 INFO:teuthology.task.internal:Removing archive directory... 2023-09-30T02:13:37.920 DEBUG:teuthology.orchestra.run.smithi059:> rm -rf -- /home/ubuntu/cephtest/archive 2023-09-30T02:13:37.922 DEBUG:teuthology.orchestra.run.smithi110:> rm -rf -- /home/ubuntu/cephtest/archive 2023-09-30T02:13:37.972 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-09-30T02:13:37.983 INFO:teuthology.task.internal:Not uploading archives. 2023-09-30T02:13:37.983 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-09-30T02:13:37.993 INFO:teuthology.task.internal:Tidying up after the test... 2023-09-30T02:13:37.993 DEBUG:teuthology.orchestra.run.smithi059:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-09-30T02:13:37.996 DEBUG:teuthology.orchestra.run.smithi110:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-09-30T02:13:38.017 INFO:teuthology.orchestra.run.smithi059.stdout: 525947 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 30 02:13 /home/ubuntu/cephtest 2023-09-30T02:13:38.030 INFO:teuthology.orchestra.run.smithi110.stdout: 525357 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 30 02:13 /home/ubuntu/cephtest 2023-09-30T02:13:38.031 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-09-30T02:13:38.041 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-09-30T02:13:38.064 INFO:teuthology.nuke:Checking targets against current locks 2023-09-30T02:13:38.092 DEBUG:teuthology.nuke:shortname: smithi059 2023-09-30T02:13:38.092 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-30T02:13:38.115 DEBUG:teuthology.nuke:shortname: smithi110 2023-09-30T02:13:38.116 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-30T02:13:38.119 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi059.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7407087', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-30 01:33:11.163621', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEIBRHJ0E9yId33pAQ6zaKUIboMamQ5eavNMIuT0xJxpdZT67asfI4+/iMT8fjQz+Bty8+qBIkRMYPZtedOmIbM='} 2023-09-30T02:13:38.143 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi110.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7407087', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-30 01:33:11.165031', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEoQZBS3c/f0LpnsOM7gYX1QUT6fO3qGKPBNFX4nLXfv7pJqFWKHqO/grsNaWh8iLnuAcpGMUTMEOtrac/4nX/Y='} 2023-09-30T02:13:38.174 INFO:teuthology.orchestra.console.smithi059:Power off 2023-09-30T02:13:38.174 DEBUG:teuthology.orchestra.console.smithi059:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-30T02:13:38.197 DEBUG:teuthology.orchestra.console.smithi059:power off output: Chassis Power Control: Down/Off 2023-09-30T02:13:38.197 DEBUG:teuthology.orchestra.console.smithi059:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T02:13:38.213 INFO:teuthology.orchestra.console.smithi110:Power off 2023-09-30T02:13:38.214 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-30T02:13:38.225 DEBUG:teuthology.orchestra.console.smithi059:check power output: Chassis Power is on 2023-09-30T02:13:38.238 DEBUG:teuthology.orchestra.console.smithi110:power off output: Chassis Power Control: Down/Off 2023-09-30T02:13:38.238 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T02:13:38.261 DEBUG:teuthology.orchestra.console.smithi110:check power output: Chassis Power is on 2023-09-30T02:13:42.226 DEBUG:teuthology.orchestra.console.smithi059:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T02:13:42.262 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T02:13:42.337 DEBUG:teuthology.orchestra.console.smithi059:check power output: Chassis Power is on 2023-09-30T02:13:42.372 DEBUG:teuthology.orchestra.console.smithi110:check power output: Chassis Power is on 2023-09-30T02:13:46.339 DEBUG:teuthology.orchestra.console.smithi059:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T02:13:46.374 DEBUG:teuthology.orchestra.console.smithi110:pexpect command: ipmitool -H smithi110.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T02:13:46.449 DEBUG:teuthology.orchestra.console.smithi059:check power output: Chassis Power is off 2023-09-30T02:13:46.485 DEBUG:teuthology.orchestra.console.smithi110:check power output: Chassis Power is off 2023-09-30T02:13:46.551 INFO:teuthology.orchestra.console.smithi059:Power off completed 2023-09-30T02:13:46.586 INFO:teuthology.orchestra.console.smithi110:Power off completed 2023-09-30T02:13:46.692 INFO:teuthology.run:Summary data: description: rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v2only objectstore/bluestore-low-osd-mem-target rados tasks/rados_api_tests validater/valgrind} duration: 1937.5003879070282 failure_reason: '"1696039520.0927126 mgr.x (mgr.4108) 1 : cluster [ERR] Unhandled exception from module ''pg_autoscaler'' while running on mgr.x: ''PgAutoscaler'' object has no attribute ''noautoscale''" in cluster log' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=4940495653ec4eaf8a4ca6ac8d69ef3a status: fail success: false 2023-09-30T02:13:46.692 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-09-30T02:13:46.771 INFO:teuthology.run:FAIL